U.S. securities regulators may reduce the number of investment advisers who would have to undergo an annual surprise audit under a proposal to ensure that their clients' assets are safe, two sources familiar with the plan said on Tuesday.

The Securities and Exchange Commission votes on Wednesday on rules to hold investment advisers more accountable for their customers' assets after Bernard Madoff defrauded investors out of billions of dollars.

Madoff, who sent out false statements to his clients, ran a Ponzi scheme over two decades, where earlier investors were paid with money collected from later investors.

In May, the SEC proposed that investment advisers undergo a surprise audit once a year to make sure that their customers' assets are really there.

Madoff was registered as a broker-dealer and subject to oversight by both the SEC and the Financial Industry Regulatory Authority, an industry-funded watchdog. His investment advisory business was also registered with the SEC.

Under the May proposal, the surprise audit would apply to about 9,600 of the 11,000 registered investment advisers including those who have physical custody and those deemed to have custody or the ability to deduct fees from their client's assets.

Now, the SEC is fine tuning the plan and considering not requiring the some 6,000 registered investment advisers who simply deduct fees to comply with such an exam, said the two sources. The sources requested anonymity because the rule may change before Wednesday's meeting.

The potential risk is that an investment adviser can steal all of your money under guise that he or she is deducting your fee, said one of the sources.

But that has not been an issue. The surprise audit does not go to the risk you are trying to address. It won't go to whether the fee that custodian deducted was proper, said the source.

(Reporting by Rachelle Younglai; Editing by Steve Orlofsky)