Charlie Sheen will not pay his previous sexual partners who threatened to expose that he was HIV-positive, TMZ reported Wednesday. But, the partners will “go after him with a vengeance” if Sheen stopped paying, the report added.

Sheen had allegedly paid $5 million to make sure that his HIV-positive status was not made public. The payments to the partners were carried out in installments over a period of years, the entertainment website reported. Sheen reportedly settled the matter out of court and paid the partners in return for the promise of confidentiality. One settlement is said to have taken place only weeks ago, in late October.

But, the “Two and a Half Men” actor will cease the payment after Tuesday’s revelation about the disease on “Today” show, according to TMZ.

However, the lawyers for the partners told TMZ that they were Sheen’s “victims” and did not know that they were sleeping with an HIV-positive person. Furthermore, the website reported, citing the lawyers that if Sheen missed an installment the entire amount would become due immediately.

Sheen reportedly knew that he was HIV-positive for over a year and decided to keep it to himself, confiding in only a few close friends. He was being threatened with lawsuits from the partners who say they were not told about his diagnosis when they were with him.