Investors have hired attorneys to argue that large banks have a conflict of interest as mortgage servicers and holders of second liens, as a government plan to give servicers the power to rework the troubled loans has raised concerns.

Some of the investors, which include insurers and asset management firms have hired Patton Boggs LLP to lobby the government, saying the banks have a conflict of interest.

“There is no prohibition against a servicer acting in his self-interest in modifying these loans,” said Micah Green, an attorney with the firm, according to Reuters. He is a former co-chief executive of the Securities Industry and Financial Markets Association.

The SIFMA, whose members include the banks, says it is working with investors “to find workable solutions to the current financial crisis.”