U.S. President Barack Obama is a little closer to fulfilling his six-year-old promise to close the U.S. military prison at the Guantanamo Bay Naval Base in Cuba. At least five detainees are expected to be moved from the facility by the end of the month, which would leave 127 remaining prisoners, according to a U.S. official who spoke to CNN anonymously.

Dozens more prisoners who have been held by the United States for years without charges are expected to be moved out in the first half of 2015.

The U.S. is trying to return prisoners to their home countries, but some don’t want them. Earlier this month, Uruguay accepted six prisoners on humanitarian grounds. They currently live as free men in the capital, Montevideo.

But 68 remaining detainees have not yet been ruled eligible for transfer out of the camp, including 20 Yemeni nationals deemed too dangerous to release, and several men accused of direct involvement in the Sept. 11, 2001, attacks that toppled the World Trade Center towers in Manhattan, including the accused architect of the attacks, Khalid Sheikh Mohammed.

Most of the 64 detainees eligible for repatriation are from Yemen, one of al Qaeda’s global strongholds. The U.S. doesn’t want to send the men back to their country out of fear the government – which has been weakened by battles against Sunni al Qaeda militants and Shiite Houthi rebels – can’t ensure they won’t end up back in the ranks of global jihadists.

The Guantanamo Bay military base once held nearly 800 detainees of the U.S. war on terror. They have been held outside of the U.S. in order to deny them legal protections they would have on U.S. soil. Obama pledged in his first presidential campaign in 2008 to shut down the facility, but has been slow to fulfill the promise due to strong opposition to the idea of bringing the prisoners to the U.S. to receive due process.