The U.S. Secret Service said Monday it revoked the top security clearance of 11 agents and uniformed division personnel over alleged misbehavior in Colombia, and a U.S. official said more than 10 military service members may also have been involved.

George Ogilvie, a Secret Service spokesman, said of the 11 agency personnel: Pending investigation, their top secret clearance has been revoked.

The 11 Secret Service agents were placed on administrative leave following alleged misconduct involving prostitutes, in an incident that marred President Barack Obama's weekend trip to Colombia and brought unwelcome attention to the Secret Service, Reuters reported.

The Secret Service agents and officers involved range in experience from relative newcomers to nearly 20-year veterans, and all have been interviewed at least once, two government officials with knowledge of the probe told CNN.

Before Obama arrived on Thursday, some U.S. agents brought a number of prostitutes back to a beachfront hotel in Cartagena near where the president was due to stay, according to a local police source.

The Secret Service, which is responsible for presidential security, has briefed key congressional leaders on the investigation so far. Rep. Peter King, R-N.Y., chairman of the House Homeland Security Committee, told CNN that he believed 11 prostitutes were brought back to the hotel.

Over the weekend, King told CNN that one of the women refused to leave a hotel room Thursday morning. A hotel manager tried to get in the room, and eventually the woman emerged and said, they owed her money, according to King.

U.S. government sources also said there was a dispute between at least one Secret Service agent and a woman demanding payment. At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.

Initial reports from Colombian police said five U.S. military service members also were involved in the incident.

But Army Col. Scott Malcom, a spokesman for U.S. Southern Command, said an interim investigating officer who began working to collect evidence in the case found information indicating more than five may be involved. One U.S. official, who asked not to be named, told Reuters it was probably more than 10.

Pentagon spokesman George Little declined to say whether the service members involved were members of the Army, Navy, Air Force or Marines. But one Pentagon source said all four services were involved.

Little also declined to describe the nature of the service members' role at the summit Obama was attending, saying only that they were not associated with presidential security and they were performing a support mission to the U.S. Secret Service.

The top U.S. military officer, Gen. Martin Dempsey, said that the service members had let Obama down by distracting from Obama's meeting with Latin leaders in Cartagena, and had embarrassed the Pentagon's top brass.

We let the boss down because nobody's talking about what went on in Colombia other than this incident, Dempsey, the chairman of the Joint Chiefs of Staff, said at a news conference with Defense Secretary Leon Panetta. I can speak for myself and my fellow chiefs, we're embarrassed by what occurred in Colombia.

Panetta said Gen. William Fraser, the head of Southern Command, had begun an investigation to determine the facts of the incident.

On Capitol Hill, lawmakers said they were shocked and upset by the allegations, pointing to the security risks involved.

Well, what we're concerned about is that failure today can lead to blackmail five, 10, 20 years from now.... If you look at how you get somebody to do something wrong, you do it incrementally - something small, something bigger, something bigger, said Rep. Darrell Issa, R-Calif., in a Monday morning appearance on CBS This Morning.

Issa, chairman of the House Oversight and Government Reform Committee, said that he'd been told the prostitution scandal was part of a pattern of behavior that included raucus wheels-up parties for advance personnel on foreign assignments after the departure of principal officials.

Issa's panel will look over the shoulder of the inspector general as he investigates the scandal, the congressman said.

I am having a call this evening with the director of the Secret Service, because I find this to be so appalling, said Sen. Susan Collins of Maine, the top Republican on the Senate Homeland Security and Governmental Affairs Committee, which shares jurisdiction over the Secret Service with the Judiciary Committee.

I can't help but think what if the women involved had been spies, what if they had been members of a drug cartel, what if they had planted equipment or eavesdropping devices?

Sen. Lindsey Graham, R-S.C., said that although he didn't want to prejudge Sullivan, it starts at the top, really.

If the allegations are true, or close to being true, then obviously you've got a leadership problem because things like that don't happen if people are somewhat afraid, he said.

Graham, a former Air Force lawyer, said he believed any members of the military that were involved would be dealt with severely if the allegations are true.

The Secret Service agents were sent home, but the military service members stayed through the summit and finished their work while confined to their rooms, officials said.

The only charge against the service members is that they violated curfew. That could mean they weren't in their rooms when they should have been, or showed up late or had someone in their room who shouldn't have been there, Malcom said.

(Additional reporting by Susan Cornwell. Editing by Warren Strobel and Christopher Wilson)