Citadel Investment Group, one of the world's most successful hedge fund firms, sued a former top executive in its highly successful quantitative trading unit and two others for setting up their own firm.

Chicago-based Citadel, founded by 40-year-old billionaire Kenneth Griffin, said in a lawsuit filed on Thursday that Mikhail Malyshev, 40, and two other former employees had violated their non-compete clauses by starting their own firm, Teza Technologies LLC. The lawsuit was filed in the circuit court of Cook County, Illinois.

Teza Technologies made headlines this week when it was identified as the firm that had hired Sergey Aleynikov, a former Goldman Sachs Group Inc computer programer whom federal prosecutors had accused of stealing trade secrets from the Wall Street investment bank.

Malyshev, a Russian emigre with a doctorate in astrophysics from Princeton, left Citadel's quantitative trading unit in February after the funds he helped run returned about 40 percent last year. Their performance stood out at a time when most hedge funds lost money and Citadel's flagship portfolios tumbled 50 percent.

Citadel, which manages $11 billion and one of whose flagship hedge funds returned an average 20 percent per year between 1998 and 2006, said it zealously guards the secrecy of its own computer codes. The hedge fund firm said it spent hundreds of millions of dollars to develop strategies, software and hardware, or what is sometimes referred to as the secret sauce of the high frequency business, court papers show.

If the information were obtained by someone else, the company, which has often been compared with Goldman Sachs for its trading prowess, said it would suffer irreparable harm.

Defendants' activities, particularly Teza's decision to hire Aleynikov, an accused software thief, create a substantial risk that they have stolen, or may be planning to steal, Citadel's proprietary code, the hedge fund firm said in court papers.

A spokesman for Teza told Reuters columnist Matthew Goldstein that the firm had voluntarily turned over some of its computers to the Federal Bureau of Investigations.

Like all employees who leave Citadel, Malyshev faced a 9-month non-compete clause and was being paid $30,000 a month to sit out the period to at least November 2009, court papers show. Citadel found out about Malyshev's new firm only this week after Aleynikov was arrested, the firm said.

Former Citadel employees Jace Kohlmeier and Matthew Hinerfeld are also listed on the civil complaint.

A spokesman for Teza called the suit frivolous and said it appears to be timed to harass Teza executives.

We knew nothing about the theft of Goldman's software until it hit the press in connection with the arrest, said Chris Gair, Teza's lawyer. We immediately started working with the FBI. We are not going to compromise anyone's proprietary information.

(Reporting by Svea Herbst-Bayliss; Additional reporting by Steve Eder, Christian Plumb, and Matthew Goldstein in New York; Editing by Lisa Von Ahn, Gerald E. McCormick and Richard Chang)