In his State of the Union address Tuesday night, Obama proposed bringing greater transparency to the war on al Qaeda and creating a new group to research alternatives to fossil fuels. One thing he didn’t mention was a pledge he made four years ago: the closing of Guantánamo Bay. Today, there are still 166 detainees languishing in America's most notorious prison—and most of them won’t be leaving any time soon.

Last month, Obama quietly transferred Dan Fried, whose job was to relocate detainees off the island, to a new job overseeing U.S. sanctions. The fact that he won’t be replaced suggests relocations aren’t a big priority these days. Already, Congress has blocked detainees from being moved to U.S. prisons while imposing tough conditions on those who have been cleared to go to other countries. Dozens of prisoners are deemed too dangerous to be released altogether.

The Obama administration insists it’s doing everything possible to fulfill the president’s pledge. “We are absolutely still committed to closing Gitmo,” National Security Council spokesman, Tommy Vietor, said in an interview. He put the blame elsewhere, saying, “The unfortunate reality is that Congress has gone out of its way to prevent us from doing so, but we still believe closing the facility is in our national security interest."

Yet experts say the chances of Gitmo closing, at least before Obama’s out of office in 2016, are exceptionally slim.

“Guantanamo is not going to close any time soon,” said Thomas Joscelyn, a senior fellow at the Foundation for the Defense of Democracies, a right-leaning think tank. “There are too many problems to solve. There are still Yemenis who can’t be repatriated to their home country, there are detainees too dangerous to transfer anywhere and quite a few prisoners who the administration says they cannot try in an open court.”

Take the 46 detainees deemed too dangerous. The Obama administration has never released a list of these prisoners. It’s possible that their status could change in the future. In 2011, Obama signed an executive order to create an appeal process for these men, similar to a parole board. That panel, however, has yet to meet. One State Department official said he expects what is known as the “periodic review board” to begin meeting this year and reviewing cases.

There are an additional 56 detainees from Yemen. Obama has not sent a Guantánamo detainee to Yemen since that country’s al Qaeda affiliate nearly blew up a flight from Amsterdam to Detroit at the end of 2009. Senior Obama administration officials say 26 of those Yemeni detainees have been approved for transfer. In the last two years, the State Department has quietly explored whether third countries could take them in.

One big obstacle is the increasingly stringent requirements Congress has imposed on transferring detainees to a third country. Since 2009, Congress has required the secretary of Defense to confirm that there is little to no risk that a detainee sent to a third country will return to terrorism. That appears to be a tough thing to guarantee.

So perhaps it’s no surprise that the State Department transferred Fried, leaving two lawyers who used to work for him in charge of relocating prisoners. They will now do their work out of the State Department’s Office of the Legal Adviser.

In four years, Fried and his team helped relocate 71 detainees out of Guantánamo, 40 of whom went to third countries. But most of those transfers occurred in 2009, before the Obama administration proposed a moratorium on transfers to Yemen and before Congress imposed the certification requirements for the secretary of Defense. In 2012, Fried transferred only four detainees, two of whom were Uighers, the Chinese Muslim minority, and part of a group determined in 2008 to no longer be enemy combatants. Those Uighers were sent to El Salvador.

Another one of those detainees was Omar Khadr, a Canadian citizen who was captured when he was 15 years old in Afghanistan. He was transferred to a Canadian prison where he is eligible for parole this year.

“It’s incredibly difficult, and each year it gets more difficult,” one senior administration official working on transfer policy told the Daily Beast. This official said the biggest issue was having to certify that a detainee will not reengage in terrorism, but this official also said that new governments in the Middle East brought about by the Arab Spring have made it more difficult to gain assurances that detainees will be monitored after being released from Guantánamo.

Andrea Prasow, a senior counterterrorism counsel for Human Rights Watch, said it was still possible to close the Bush-era prison before Obama leaves office by either prosecuting or transferring the remaining 166 prisoners. But, she said, “the reality is that it requires some political will that seems to be lacking.” Prasow and other civil liberties advocates point out that Obama has signed the Defense authorization bills that include the new congressional restrictions on transfers of detainees. Chris Anders, a senior legislative counsel for the American Civil Liberties Union, said, “For the past three years there has not even been a point person at the White House to coordinate all the different parts of government required to meet the president’s pledge to close Guantánamo.”

When Obama delivered his first State of the Union speech in 2009, he appeared to have that political will to close Guantánamo. He said, “There is no force in the world more powerful than the example of America. That is why I have ordered the closing of the detention center at Guantánamo Bay, and will seek swift and certain justice for captured terrorists—because living our values doesn’t make us weaker, it makes us safer and it makes us stronger.”