A U.S. judge on Tuesday rejected a $125 million settlement between Google Inc and authors that would have let the company publish millions of books online to create the world's largest digital library.

New York Judge Denny Chin said the deal gave Google a significant competitive advantage and would simply go too far in giving it the power to digitize all copyrighted works and sell subscriptions to them online without explicit permission.

Google has scanned roughly 12 million books from some of the country's finest libraries, in what it has said was an effort to provide easier access to the world's knowledge.

It was sued in 2005 by the Authors Guild and Association of American Publishers for violating copyright laws, but reached a settlement by agreeing to pay $125 million to people whose copyrighted books have been scanned, and to locate and share revenue with the authors who have yet to come forward.

Still, some critics contended that the deal gave Google an unfair competitive advantage and broke antitrust law. Chin -- a judge on the Second Circuit Court of Appeals who heard the case while he sat in district court -- agreed on both counts.

The ASA (amended settlement agreement) would give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission, while releasing claims well beyond those presented in the case, Chin wrote in rejecting the deal.

He urged Google and authors and publishers to amend their settlement to only include books whose copyright owners have agreed to the arrangement, rather than a blanket deal that would require authors to opt out should they not want their scanned books to be sold online.

Chin rejected the settlement without prejudice, meaning a revised pact could be submitted.

Critics of the proposed settlement also include Amazon.com Inc, which sells the Kindle digital reader that would not be compatible with Google's library, and Microsoft Corp. Sony Corp, which makes an e-reader compatible with Google's software, favors the pact.

The Justice Department is also looking into the deal, and has said it might violate antitrust and copyright law. The government focused on a group of books often described as orphan works because they are still in copyright but the rights holder cannot be found. The settlement gives Google the right to market these works.

The ruling comes as Google Co-founder Larry Page is set to take the CEO reins in April from Eric Schmidt. Page is an early champion of digital books and personally scanned many of the works himself early on in the project.

Analysts said the setback would have little impact on Google's financials, since any revenue would have paled in comparison to Google's main Internet search business.

This is clearly disappointing, but we'll review the court's decision and consider our options, Google's managing counsel, Hillary Ware said in an emailed statement. Regardless of the outcome, we'll continue to work to make more of the world's books discoverable online through Google Books and Google eBooks.

Since the settlement, Google had launched in December an electronic bookstore with three million books, with permission from the relevant publishers.

Other complaints about the settlement -- such as a lack of adequate notice to the writers whose works were affected -- were rejected by the judge.

Under the agreement, authors and publishers would register works and be paid for books and other publications the search giant puts online. The settlement covered books that were out of print but still under copyright.

The case is The Authors Guild et al v. Google, Inc, U.S. District Court, Southern District of New York, No. 05-08136.