Rescuers on Thursday recovered four bodies from the rubble of three buildings that collapsed in downtown Rio, highlighting the creaky infrastructure of the city that will host the 2014 soccer World Cup and the 2016 Olympics.

The buildings, one 20 floors high, collapsed on Wednesday night in a cloud of dust behind the city's 100-year-old Belle Epoque-style Municipal Theater.

Engineers said work was being done with no permits from city regulators on two floors of the taller building, which could have caused it to collapse and bring the other buildings down.

Rescue teams pulled six people alive from the rubble, but at least 22 people were missing, said Rio de Janeiro Mayor Eduardo Paes said in a news conference.

We need to keep working with tranquillity so the rescue teams keep up with the search, Paes said.

The buildings housed offices that had mostly closed for the day and few people were on the normally busy street at the time of the disaster.

Paes said the cause of the collapse could have been a structural failure as a result of building work underway inside one of the buildings. He said there was no gas leak that could have caused an explosion.

Building work was underway on the third and ninth floors that was not authorized by the Rio building commission, said Luiz Antonio Cosenza, head of the local engineers association.

The most likely scenario was structural failure, he said.

Rio is struggling to address concerns about its decrepit infrastructure, unreliable power supplies and deficient public transportation as it prepares to host global sporting events.

Construction and renovation of 12 stadiums for the soccer World Cup in 2014 is behind schedule and there is concern that Brazil's overcrowded and inefficient airports will be not able to handle masses of sports fans expected to attend the events.

The building collapses come months after an explosion apparently caused by a gas leak ripped through a restaurant in downtown Rio, killing three people and igniting more concern about the state of the city's infrastructure.

In recent months, Rio's inhabitants have had to deal with exploding sewer lines and landslides in the city's slums caused by heavy rain and deforestation.

The collapsed buildings had a bakery and an Itau Unibanco Holding bank branch on the ground floor and were near the headquarters of state-run companies such as oil giant Petrobras and development bank BNDES.

Witnesses said they heard the structures cracking and saw plaster falling before the buildings collapsed, causing panic in the streets and covering parked cars with dust and debris.

It was like an earthquake. First some pieces of the buildings started to fall down. People started to run. And then it all fell down at once, a witness who identified himself as Gilbert told Reuters.

One man said he was on the 10th floor and ran down the stairs just in time to escape the collapse.

My wife was inside. I spoke with her just before the collapse, another man in tears said on television.

(Additional reporting by Rodrigo Viga Gaier; Writing by Anthony Boadle; editing by Christopher Wilson)