As a result, AstraZeneca plans no regulatory submissions for zibotentan at this time and a spokesman said on Monday it was discussing the implications of the setback with investigators working on other studies involving the drug.
The failure of zibotentan to improve overall survival in the Phase III study follows similarly unsuccessful trials for two other AstraZeneca pills, Recentin in colon cancer earlier this year and vandetanib in lung cancer in 2009.
Vandetanib has since gone on to show benefits in thyroid cancer, a smaller potential market.
Zibotentan, a once-daily tablet, is being studied in more than 3,000 men with prostate cancer in a program of clinical trials.
Two other studies looking at the medicine in different settings are still ongoing, with one having fully recruited patients and expected to announce results in the second half of 2011 and the second having almost completed recruitment.
Paul Diggle, an analyst at Ambrian Partners, said he had removed modest sales expectations of $200 million a year for the drug from his forecasts.
Zibotentan was designed to help patients who no longer respond to treatments that block the action of testosterone, a hormone driving cancer growth, by blocking another biological pathway that helps tumors thrive.
It was widely viewed as a risky product by analysts and the drug had been forecast to achieve sales of $189 million by 2014, according to consensus forecasts compiled by Thomson Reuters.
Shares in AstraZeneca fell 1 percent by 7:45 a.m. ET, underperforming a 0.2 decline in the European drugs sector.
The Anglo-Swedish drugmaker had better news on Friday when its heart drug Brilinta, or Brilique, was recommended for approval in Europe -- boosting prospects for a medicine that analysts expect to sell some $2 billion by 2014.
AstraZeneca is relying on revenues from new products like Brilinta to offset expiring patents on some of its best-selling medicines, such as heartburn treatment Nexium and Seroquel for schizophrenia.
Within the cancer space, the group has pushed hard to develop convenient tablet-based treatments as an alternative to injections or infusions, although so far the strategy has met with only limited success.
One of its early cancer pills, Iressa for lung cancer, was initially seen as a flop but it has recently seen a revival in its fortunes thanks to tests to identify those patients who will gain from using it.