Taiwan's economics ministry said AIG's plan to sell Nan Shan Life to battery maker China Strategic and Hong Kong investment firm Primus did not comply with regulations on mainland investment nor meet criteria on experience in the insurance business and ability to raise funds.
It gave the buyers 30 days to appeal, but noted that no previous such appeals had succeeded.
AIG, which needs to sell assets to pay back the U.S. government for a bailout, first agreed to sell Nan Shan last October, but suspicions in Taiwan about the connections of China Strategic with political foe China held up the deal.
The rejection of AIG's Nan Shan plan was the second collapse of a planned Asian deal in four months.
AIG lost a $35.5 billion sale of Asian insurance unit American International Assurance (AIA) in May.
It must now choose between finding another buyer, who may offer a lower price for Nan Shan, or hold on to it and find other ways to pay back the bailout money.
If lower bids are now forthcoming this will further reduce AIG's ability to repay U.S.-based debts, said Wenli Yuan, senior analyst at Celent, a financial industry research and advisory firm.
AIG may now try to quickly find a willing and acceptable local bidder to complete this process and then re-focus energies on the AIA disposal in Hong Kong, a deal that is likely to contribute far more to the debt-reduction process.
In a statement, AIG said it was disappointed by the decision and was conferring with the buyers on whether to appeal.
It said it believed it had met all criteria and other concessions it had offered demonstrated incontrovertible commitment to the long-term health and prosperity of Nan Shan.
Wu Tang-chieh, vice chairman of the Financial Supervisory Commission (FSC), Taiwan's top financial regulator, told a separate media briefing that he would be happy for AIG to continue running Nan Shan but would want the company to make sure any new buyer met the FSC's criteria.
China Strategic was not available for comment. Trading in its shares were suspended in Hong Kong after the announcement.
WAITING IN THE WINGS
AIG will find Taiwanese bank Chinatrust Financial waiting in the wings for Nan Shan. The bank, Taiwan's top credit card firm, has repeatedly said it wants to buy the insurance unit after coming second to China Strategic in the original bid in October.
Chinatrust President Daniel Wu declined to comment on Tuesday when asked what he would do, saying only that he respected the regulator's decision.
On Monday, a retired Taiwanese diplomat, who said he wanted to save Nan Shan from mainland Chinese hands, claimed he was lining up a bid, backed by unnamed Japanese and Qatari investors.
The rejection of the bid comes just after Taiwan and China signed a landmark trade deal that brings the two political foes closer than they have been in six decades, though the rejection was unlikely to be viewed in political terms.
It's probably just a reflection that there might be better suitors in terms of criteria, such as experience, said Wai Ho Leong, regional economist at Barclays Capital in Singapore.
The two sides have been getting closer for three years, and I see that continuing.
Suspicions still linger in Taiwan over China's intentions toward Taiwan and the trade deal did not touch on the sensitive subject of Chinese investment in the island's financial sector.
Highlighting such wariness, and given that Nan Shan's policyholders make up almost one-sixth of Taiwan's population, regulators repeatedly sought information from China Strategic on its mainland connections, while Taiwan's parliament took the unusual step of discussing the deal.
(Additional reporting by Rachel Lee and Ralph Jennings in Taipei and Michael Flaherty and Denny Thomas in Hong Kong; Writing by Jonathan Standing; Editing by Valerie Lee and David Cowell)