United States Olympic Committee (USOC) chief executive Scott Blackmun has promised "further action" will be taken about the four swimmers involved in a drunken dispute before claiming they had been victims of an armed robbery.
It comes as Ryan Lochte, the most culpable and high profile of the four involved, issued an apology to Brazilian television and claimed his "mistake was over-exaggerating" what had happened.
"They let down our athletes," Blackmun said.
"They let down Americans."
He did not elaborate on exactly what the punishment would be, but made no effort to hide his frustration.
It also detracted from the success of the Games and of medal table topping performance by the US team, the official said.
Lochte, Gunnar Bentz, Jimmy Feigen and Jack Conger initially claimed they were mugged on the way back to the Olympic Village, before CCTV footage emerged of the men vandalising a petrol station after a night out.
They reportedly then argued with a guard who ordered them to pay the costs of their damage.
Bentz and Conger were prevented from flying out of Brazil until they had been questioned by police while Feigen was briefly detained before agreeing to donate a sum of nearly $11,000 (£8,400/€10.200) to a Brazilian charity.
Lochte had left Brazil before police could take action, but has since been accused by Bentz of having pulled a metal advertisement in a frame to the ground, and then "stood up and began to yell at guards" after they were caught.
The 32-year-old has still not admitted to having lied by has now said he over-exaggerated" the story.
"I just over-exaggerate part of it, the very first part, I was very intoxicated," he told Globo.
"I was a a little frightened, having a gun pointed to you.
"My mistake was over-exaggerating what really happened," he said.
"I wasn't lying to a certain extent. I over-exaggerated what was happened."
Although stopping shot of full contrition, these words are stronger than a previous statement in which he claimed he had been "traumatised by being out in a foreign country with a language barrier and having a stranger point a gun at him and demand money".
The incident has provoked fury here in Brazil, both with the public and within security forces due to the international headlines about security problems.