Soon after paying a record-setting fee to close a now-controversial takeover in the spring of 2010, Japanese camera and endoscope maker Olympus Corp was back out shopping for another deal.

Once again, it turned to a New York-based advisory firm outside the dealmaking mainstream and once again the Japanese company ended up paying an out-sized amount to complete a tangled deal that has come under scrutiny, internal company documents show.

Olympus agreed in December last year to buy the rights to a biotech remedy intended to help regenerate human bone from medical device maker Stryker Corp for $60 million (37.6 million pounds).

But the documents reviewed by Reuters show that the deal may end up costing the Japanese firm around 50 percent more, thanks to payments made to its adviser.

Olympus made a $25 million loan to Viscogliosi Brothers, the firm that advised it on the transaction and expects it will need to write off most of that amount, according to the documents.

The Japanese company also paid a higher-than-normal $5 million transaction fee to Viscogliosi Brothers and agreed to reimburse $4.4 million in expenses, the documents show. The fees for advisers are typically 2 percent to 3 percent for deals of this size.

Richard Torrenzano, a spokesman for Viscogliosi Brothers, said the firm could not discuss the deal under terms imposed by Olympus.

Unfortunately, we are bound by a confidentiality agreement with Olympus and cannot comment on any aspect of the deal, he said.

Olympus declined to comment, as did Stryker spokesman Aaron Kwittke.

A string of troubled acquisitions by Olympus has cost the company more than $1.2 billion in charges and write-offs, led to the resignation of Chairman Tsuyoshi Kikukawa and forced the company to appoint a panel to investigate in the face of a plunging share price.

The 2010 biotech deal is small by comparison to some of Olympus' other deals. But like those acquisitions it was challenged by former Olympus CEO Michael Woodford and could raise questions about the company's governance and management.

Woodford, 51, has said he was fired in a boardroom coup at Olympus for questioning a $687 million payment to advisers in the $2.2 billion takeover of medical equipment maker Gyrus in 2008.

The FBI is investigating the massive advisory fee and Woodford has called on authorities in Japan and Britain to launch investigations of their own. Sources have told Reuters that Japan's securities watchdog was looking into past Olympus takeover deals.

Olympus documents provided to Reuters by Woodford show that the former CEO raised questions about the Stryker acquisition and the role of advisers in that deal as well.

I'm still unclear as to the role played by Viscogliosi Brothers in the transaction and the basis upon which the loan was originally made, Woodford said in a September 23 memo to Olympus Executive Vice President Hisashi Mori.

The documents reviewed by Reuters also include contractual agreements between Olympus and Viscogliosi Brothers signed in April and November of 2010, an April 2010 letter of intent between Stryker and Viscogliosi Brothers, and an August 31, 2011 internal Olympus presentation related to the deal.

COSTLY DIVERSIFICATION

Olympus has described its acquisitions as a way to diversify from its major markets in optical equipment including cameras and the endoscopes that surgeons now rely on for less invasive surgeries.

On December 7, 2010, Olympus announced in a news release that it had formed a new unit, Olympus Biotech. At the same time, it announced that the unit had acquired a line of bone growth therapies from Stryker that would be part of its stepped-up push into biotechnology.

The Stryker assets that Olympus acquired, including a Lebanon, New Hampshire plant, weren't unencumbered.

In August 2010, Kalamazoo, Michigan-based Stryker agreed to pay $1.35 million to settle claims by Massachusetts prosecutors that the company marketed its bone growth products for uses that had not been cleared by federal regulators.

The products known as OP-1 are based on a naturally occurring protein used to stimulate bone growth. Stryker Biotech only had U.S. Food and Drug Administration approval for the product under the humanitarian device exemption, which limits the number of units that can be sold and typically prohibits the company from making a profit.

The Olympus news release did not mention the advisory role that Viscogliosi Brothers had played in the deal or payments to the firm.

Viscogliosi Brothers, a merchant banking and venture capital firm that focuses on the musculoskeletal and orthopaedics sector, was founded by Anthony Viscogliosi, 49, and two of his brothers.

A graduate in economics from the University of Michigan-Dearborn, Viscogliosi has said he was attracted to orthopaedics because of its cutting-edge nature.

Each of his e-mails ends with a quote from American poet and philosopher Ralph Waldo Emerson, Do not go where the path may lead; go instead where there is no path and leave a trail, he told students at his former college in a presentation a year ago just as his firm was wrapping up discussions with Olympus.

Viscogliosi Brothers first signed a letter of intent with Stryker in April 2010 to buy the assets of Stryker Biotech.

The agreement envisaged the Viscogliosi Brothers would form a new company with other investors and buy the assets from Stryker for $50 million. It also gave Stryker an option to take up to a 19.9 percent stake in the new company.

Within days of the agreement, Viscogliosi Brothers signed a confidentiality agreement with Olympus, and subsequently agreed to form a 50-50 joint venture with the Japanese firm to buy Stryker's assets.

This was not the first deal between Olympus and Viscogliosi Brothers. Small Bone Innovations Inc, a company controlled by Viscogliosi Brothers, got a $12 million investment from Olympus in March 2010, according to a company press release.

In May 2010, Small Bone Innovations also announced agreements with Olympus Terumo Biomaterials Corp to distribute Small Bone products in Japan and China.

However, Olympus changed its mind about having Viscogliosi Brothers as a joint venture partner in the bid for the Stryker Biotech assets.

In November 2010, the Japanese company reached a deal with Viscogliosi Brothers to keep them out of the acquisition. In return, it agreed to give the New York firm the $25 million loan and other payments.

In an August 31 presentation, Olympus Biotech cites uncertainty of potential compliance issues as the reason for the abrupt shift in approach.

The Olympus loan to Viscogliosi Brothers carries an eight-year term and 7-percent interest rate, and is collateralized by Viscogliosi Brothers' interests in two companies -- Small Bone Innovations and Paradigm Spine LLC.

Viscogliosi Brothers agreed to use the proceeds from the loan to pay off debt owed to Fortress Investment Group Inc.

Small Bone Innovations had taken out a $30 million debt facility from Fortress in 2009. Fortress spokesman Gordon Runte declined to comment.

Under the terms of the agreement, Olympus also agreed to forgive $2 million of the loan every year, starting January 2012, which would leave a balance of $22.8 million, including accrued interest, by the end of 2018.

The agreement also calls for the forgiveness of the entire loan balance and accrued interest if the acquired products achieve certain FDA approvals by 2015 or 2018.

Olympus Biotech in August expressed concerns about the value of the collateral it has from Viscogliosi Brothers, and Woodford said in his September 23 memo he was told by Mori the company would need to write off the loan.

(Reporting by Paritosh Bansal in New York, Kirstin Ridley and Alex Smith in London; Kevin Krolicki in Detroit; and Isabel Reynolds in Tokyo. Editing by Martin Howell)