Novell said its board had instead initiated a review of alternatives, including a share buyback, cash dividend, joint ventures, recapitalization, alliances or an outright sale.
Elliott Associates announced its $5.75 a share offer on March 2, sending Novell's stock soaring 28 percent as investors hoped it would trigger a bidding war and put the world's No. 2 maker of the open source Linux operating system into play.
It undervalues the company's franchise and growth prospects, Novell said in a statement.
The Blue Harbour Group, which owns 4 percent of the company, agreed with the board.
We support the company's decision today to pursue a formal review process with the assistance of its advisors, the investment firm said in a separate statement.
Analysts say large software makers may consider bidding as they could sell their business programs alongside Novell's version of Linux, used widely by financial institutions and telecommunications carriers.
Companies including HP
Elliott, which already owns 8.5 percent of Novell, had offered at the time to pay a 21 percent premium for a company that has been hurt by competition and struggled to boost sales. In 2009, Novell's revenue declined about 10 percent.
Shares in the company -- which has nearly $1 billion in cash and saw its stock surge above $6 in the wake of the bid -- closed on Friday at $5.64.
Elliott, an investment firm with over $16 billion in assets under management, has gone after undervalued companies before and has targeted other technology firms, including Epicor and Packeteer.
Novell's key rivals include Microsoft
JPMorgan is serving as Novell's financial adviser.
(Reporting by Edwin Chan; Editing by Peter Cooney)