The Pentagon rejected the idea of a rescue mission for Sgt. Bowe Bergdahl because he was being moved so often by his Taliban captors that U.S. special operators would have had to hit up to a dozen possible hideouts inside Pakistan at once in order to have a chance at rescuing him.

That’s according to U.S. officials, who also say the Obama administration did not want to risk the political fallout in Pakistan from another unilateral U.S. raid, like the Navy SEAL raid that killed al Qaeda leader Osama bin Laden in 2011.

Bergdahl had also twice tried to escape, so the militants guarding him had stepped up their numbers, further complicating any potential rescue attempt.

“A rescue mission would have been fraught politically as well as tactically,” according to a senior defense official briefed on the Bergdahl case.

The lack of information about Bergdahl’s whereabouts shows how few choices the administration had, and why officials felt negotiations with the Taliban were their best option. His repeated attempts to escape also call into question those who call him a deserter who did not intend to return to the U.S. army’s ranks.

The White House released five high-ranking Taliban members from Guantanamo Bay prison over the weekend in return for Bergdahl’s freedom, sparking outrage from lawmakers who were kept in the dark until the trade was done. Law requires Congress to be given 30 days notice before a prisoner is released from Guantanamo, but White House officials say Bergdahl’s deteriorating health necessitated the rapid action.

Senators shown the Taliban’s proof of life video Wednesday that was pivotal in the White House’s decision making process say he did not look well, but argue that should have been shown to them before the release was negotiated.

At the same time, many soldiers who served with Bergdahl have spoken out against him—blaming Bergdahl for wandering off his post, and for diverting needed intelligence and surveillance resources to hunt for him. Some soldiers even blame Bergdahl for the deaths of a half-dozen troops, although those claims have been disputed.

Bergdahl was turned over to U.S. special operations forces by Taliban fighters in eastern Afghanistan last Saturday, an event the fighters filmed and turned into a propaganda video released on Jihadi websites Wednesday.

Two more U.S. officials and a former Afghan official said Bergdahl escaped his Taliban captors twice during his five years of captivity, once in the fall of 2011 as then reported The Daily Beast, and a second time, believed to be sometime in 2012. They spoke on condition of anonymity because they were not authorized to speak publicly.

In his first escape, Afghan sources said he avoided capture for three days and two nights before searches finally found him, exhausted and hiding in a shallow trench he had dug with his own hands and covered with leaves.

In his second bid for freedom, which has not been previously reported, Bergdahl made it to a remote village in the mountainous part of Pakistan, the former Afghan official said. The villagers simply returned him to his captors in the Haqqani Network. The U.S. officials were not familiar with details of the second escape attempt, though they knew Bergdahl had briefly slipped away from his captors.

Three special operations officials say rescue missions to bring him back were contemplated multiple times over the years.

When Gen. Stanley McChrystal was in charge in Afghanistan, the U.S. had a better idea of his general location, and a mission was mapped out and briefed to senior officials.

They rejected it, the officials say, because the mission planners warned of a high probability that Bergdahl and at least two to three special operations troops would be killed in the operation, so well-guarded was he by Haqqani fighters in a hard-to-reach mountain hideout on the Pakistani side of the border.

But a former senior U.S. Official said the U.S. government never pinpointed his location in a way that enabled them to plan a Bin-Laden-style raid—and therefore was never able to present to the president with a plan to go get him. The official spoke on condition of anonymity because he was not authorised to speak publicly.

Subsequent commanders decided it was better to keep tabs on him via spies and satellites as best they could until he was moved to an easier-to-reach location, or negotiations with the Taliban freed him.

The situation was even worse for Pentagon planners considering rescue options in 2014. After Bergdahl’s escapes, the Taliban stepped up security around him, and with the rise in CIA drone strikes in Pakistan’s ungoverned tribal region, constantly moved him among roughly a dozen safe houses; successfully rescuing him would’ve meant launching as many as a dozen raids simultaneously—dramatically increasing the risk.

The Pentagon had put Assistant Secretary of Defense for Special Operations Michael Lumpkin in charge of the negotiations last year, after previous attempts to broker Bergdahl’s release had broken down.

Other special operations officials also maintained back channel communications with his captors through former Taliban officials, to keep tabs on his health and explore options for getting him back.

The senior defense official said the Pentagon stepped up negotiations with the Taliban via the government of Qatar after seeing Bergdahl’s proof-of-life video last December.

“We could see he looked rough, from the way he held his body and slurred some of his words,” the official said. “We got other accounts as well that his health was deteriorating,” he added.

U.S. Army spokesman Col. Steve Warren declined to comment Wednesday on the reports of Bergdahl’s attempted escapes or debates over whether to rescue him.

Eli Lake contributed to this report.