Convicted and admitted fraudster Bernard Madoff believed federal authorities had figured out his scam on several occasions, according a lawyer who met with him on Tuesday.

“There were several times that I met with the [Securities and Exchange Commission] and thought ‘they got me,’” Madoff told San Francisco lawyer Joseph Cotchett, who had threatened to sue Madoff’s wife, sons and brother an behalf of some victims, according to ABC News.

Cotchett told ABC he and partner Nancy Fineman met with Madoff for four and a half hours at Butner, NC, where he is in federal prison serving a 150 year sentence for cheating investors out of billions of dollars.

Cotchett felt Madoff was candid. He says Madoff explained how he pulled off the scam. The lawyer believed Madoff cared for his wife but not for his sons. The pair, Mark and Andrew, turned in their father to police after Madoff admitted he had committed fraud.

Cotchett added that he was almost certain to name Madoff’s brother Peter in a suit, but did not yet know if he would name his wife or sons in the suit.