The suspicious circumstances surrounding the alleged armed robbery of Ryan Lochte and three other members of the U.S. Men's Swimming Team in Rio is threatening to overshadow the cornucopia of medals the team has won in the 2016 Summer Olympics.

On Sunday, multiple news outlets reported Ryan Lochte's story stating that early that morning he and teammates Gunnar Bentz, Jack Conger and Jimmy Feigen were pulled over in a taxi and robbed by gunmen posing as police. However, in the last 24 hours Brazilian authorities have expressed doubt about the swimmers' story. This has apparently culminated in a court's decision Wednesday to seize the passports of Lochte and Feigen in an effort to detain them for further investigation. 

“Local police arrived at the Olympic Village this a.m. and asked to meet with Ryan Lochte and James Feigen and collect their passports in order to secure further testimony from the athletes. The swim team moved out of the village after their competition ended, so we were not able to make the athletes available," said Patrick Sandusky, Chief External Affairs Officer for United States Olympic Committee in a statement released Wednesday. “Additionally, as part of our standard security protocol, we do not make athlete travel plans public and therefore cannot confirm the athletes’ current location. We will continue to cooperate with Brazilian authorities.”

News of the alleged robbery first came from Lochte's mother, Ileana Lochte, whom the swimmer had reportedly called after the incident. After word spread and the media took interest, Lochte, who did not initially go to police with the story, gave his account of what happened to NBC News. 

"We got pulled over, in the taxi, and these guys came out with a badge, a police badge, no lights, no nothing just a police badge and they pulled us over," Lochte said. "They pulled out their guns, they told the other swimmers to get down on the ground — they got down on the ground. I refused, I was like we didn't do anything wrong, so — I'm not getting down on the ground. And then the guy pulled out his gun, he cocked it, put it to my forehead and he said, 'Get down,' and I put my hands up, I was like 'whatever.' He took our money, he took my wallet — he left my cell phone, he left my credentials."

However, an Associated Press report Tuesday revealed that Brazilian authorities had discovered no evidence to back up the story. In fact, a video of the group returning to Olympic Village the night of the alleged incident appears to show Lochte and some of the other swimmers still in possession of some of the items they alleged to have been stolen. Police cannot track down the taxi driver behind the wheel during the robbery, nor any witnesses to the crime and noted that there were inconsistencies between the four swimmers' stories.

Brazilian police say they are still treating Lochte and the others as victims and are simply seeking more information. The swimmers said they had been at a party prior to the incident and were drunk at the time of the robbery, which could explain some of the inconsistencies in their stories. Lochte has said the reason the group did not report the incident was fear of repercussions for being so far away from Olympic Village. It is not clear what motive the group would have for lying about the robbery. 

The incident has been the subject of much controversy in the midst of an Olympics where the threat of local street crime in Rio has been a hot button issue. The International Olympic Committee initially denied the veracity of Lochte's story before walking the accusation back.