Ahmad Fawzi said the small advance team of UN monitors was unable to stop the bloodshed between opposition forces and government troops.
The news comes after Annan told the Security Council on Tuesday the situation in Syria was bleak, amid reports President Bashar al-Assad's forces had entered the city of Hama on Monday after UN observers left, reportedly killing more than 30 people according to opposition forces.
They are entering areas where there has been conflict like Homs and Hama and when they go [there] the guns are silent, Fawzi told U.N. Television in Geneva, referring to the monitors, according to Reuters.
When they leave, the exchanges start again.
We have credible reports that ... these people who approach the observers may be approached by security forces or Syrian army and harassed or arrested or even worse, perhaps killed.
Annan said the situation in Syria is entirely contrary to the will of the international community and that he is particularly alarmed by reports of government forces firing on protesters in Hama.
If confirmed, this is totally unacceptable and reprehensible, he said.
The reported uptick in violence comes despite the United Nations' pledge to bolster its observation mission within 90 days.
While there are currently only a handful of U.N. observers in Syria, there are plans to authorize up to as many as 300.
Meanwhile, Susan Rice, the US permanent representative to the UN, Susan Rice, said all Security Council members want more UN observers sent to Syria quickly, adding that 100 observers could be deployed there within one month.
But continued bloodshed has led many to doubt the viability of the UN mission, despite a pledge by Secretary General Ban Ki-moon to begin deploying all monitors next week.
Assad's forces have also failed to withdraw heavy weapons from cities as required by a UN brokered peace plan, officials said Tuesday.