The ruling, by an arbitration tribunal in Stockholm on Thursday, handed a major victory to BP's billionaire partners in its Russian venture, TNK-BP.
The decision is a blow for the Russian government, which has struck a string of deals this year with global energy majors to tap new oil and gas regions to sustain long-term output growth, and for BP, which is still reeling from the Gulf of Mexico oil spill.
It will especially sting BP Chief Executive Bob Dudley, who was once in charge of TNK-BP before being forced to leave Russia in 2008 due to what he described as a campaign of harassment by the joint venture's co-owners.
BP's long history in Russia demonstrates progress and growth is not always straightforward, Dudley said in an internal memo seen by Reuters, but it has consistently been successful over the long term.
The arbitration panel ruled on the BP-Rosneft case after Alfa-Access-Renova (AAR), which represents the tycoons who own half of TNK-BP, won a temporary injunction last month against the deal.
The tribunal's decision means that BP is prohibited from entering into any future share arrangement with Rosneft that has any kind of strategic component, said AAR, while BP said that the tribunal found the injunction should continue.
Dudley said BP will seek a ruling on whether the share swap -- under which it would exchange 5 percent of its own stock for a 10 percent holding in Rosneft -- may proceed on its own.
BP would also seek to resolve differences with AAR to allow the Arctic exploration pact with Rosneft to go ahead.
Analysts said that the court ruling would not necessarily derail the BP-Rosneft pact, but the cost of any settlement to enable it to go ahead had probably increased.
One issue that could be revisited is the proposal by TNK-BP's Russian shareholders that it effectively supplant BP in the share swap, buying the stake in BP and becoming a minority shareholder in Rosneft.
The price of the case settlement is likely to rise ... and of course, AAR understands it's a blow for BP's prestige, said Valery Nesterov, energy analyst at Troika Dialog in Moscow.
Everyone understands that the outcome is an out-of-court settlement that will allow BP to continue cooperating with Rosneft and for TNK-BP to develop and count on support from the British shareholders.
AAR RELATIONSHIP CONTINUES REGARDLESS
BP said it still intends to honor the TNK-BP shareholders' agreement and would respect the arbitrators' decision, adding that it remains fully committed to investing in Russia.
TNK-BP is Russia's No. 3 oil producer with crude output of 1.4 million barrels per day, and accounts for around 10 percent of BP's earnings.
TNK-BP has been very successful and this is underpinned by the long-standing business relationship between BP and AAR which continues, regardless of this particular issue, Dudley said in the memo.
Having made such a big media splash with the Rosneft deal, Dudley also warned that there may be adverse press comments in the coming days, adding: Please be assured that we are doing all the right things to protect the interests of BP.
The BP-Rosneft agreement, after its announcement in January, quickly met resistance from AAR, which represents Russian billionaires Mikhail Fridman, German Khan, Viktor Vekselberg and Len Blavatnik.
The dispute threatens to raise tensions between the group of oligarchs and Russian Prime Minister Vladimir Putin's powerful energy tsar, Igor Sechin, who is both deputy prime minister and chairman of Rosneft.
Sechin said in a newspaper interview last month that, should the agreement unravel, Rosneft would require compensation from those who would have inflicted ... losses.
Rosneft had no comment on the arbitration ruling. The company has declined to say whether BP would be liable for a break-up fee should the deal collapse.
The Russian government refrained from commenting about the ruling. We need to look into every detail, spokesman Dmitry Peskov told local news agencies.
(Additional reporting by London bureau and Braden Reddall in San Francisco; Editing by Douglas Busvine, Steve Orlofsky and Matthew Lewis)