A planned speech at a U.S. university by Narendra Modi, the chief minister of India’s Gujarat state, was cancelled Sunday following protests by a section of teachers and students.

Modi, a key leader of the right wing Bharatiya Janata Party (BJP), was set to deliver a keynote address via video conference for the Wharton India Economic Forum (WIEF) of the University of Pennsylvania in Philadelphia during a conference scheduled Mar. 22-23.

The invitation had triggered a controversy with professors and students from various universities registering strong protest to the management.

"This is the same politician who was refused a diplomatic visa by the United States State Department Mar. 18, 2005 on the ground that he, as Chief Minister, did nothing to prevent a series of orchestrated riots that targeted Muslims in Gujarat," a protest letter, signed by 175 teachers and students, said, as reported by the NDTV.

The organizers said they stood by the decision to invite Modi, but adverse reactions from multiple stakeholders prompted them to cancel the address.

"Our team felt that the potential polarizing reactions from sub-segments of the alumni base, student body and our supporters might put Mr Modi in a compromising position, which we would like to avoid at all cost...We do not endorse any political views and do not support any specific ideology. Our goal as a team is only to stimulate valuable dialogue on India's growth story," the Forum said in a statement released Sunday.

The communal riots of 2002 in which more than 1,000 people were killed in Gujarat, most of them Muslims, during Modi's first term as chief minister, has been the primary reason for critics to slam him as a Hindu zealot.

Riots erupted in February 2002, after 60 Hindu pilgrims returning from Ayodhya died as the Sabarmati Express train in which they were traveling caught fire, which Muslim fundamentalists were blamed for igniting.

Modi was denied the U.S. visa in 2005 for the alleged dubious role he played in the 2002 communal riots. Modi has always denied any wrongdoing in connection with the riots, but continues to reject calls to offer an apology.

In December last year, a group of 25 U.S. lawmakers urged then–Secretary of State, Hillary Clinton, to continue Washington’s policy to deny visa to Modi, stating that his government had not adequately pursued justice for the riot victims.

Modi described the denial of a diplomatic visa and revoking of a business visa as insulting and asked the Congress-led central government to take up the issue with the U.S. officials.

He won a record third successive term in the state elections in December, bolstering his prime ministerial ambitions in the parliamentary elections to be held in 2014, with his supporters already kick-starting a "Modi-for-PM" campaign.

Many Indians, however, remain critical of his credentials saying they cannot accept Modi as a prime minister because of his alleged role in the riots.

Modi hasn’t responded to the cancellation of his speech, but the BJP downplayed the incident saying a protest in the U.S. did not matter because Americans cannot vote in India, the NDTV report said.