U.S. President Donald Trump has been working on a way to get ultra-conservatives on board with legislation to repeal and replace the Affordable Care Act, but now that the legislation is becoming more draconian, support from the Republican center, which wasn’t thrilled to begin with, may be slipping.

Vice President Mike Pence was scheduled to meet with various House GOP factions Tuesday night.

Read: President Lashes Out At Democrats, Far-Right On Defeat

Trump spent Sunday golfing with Sen. Rand Paul, R-Ky., a vocal opponent of the American Health Care Act, which was pulled from the House floor before a vote could be held March 24. House Speaker Paul Ryan said he just didn’t have the votes to pass the measure, largely because the Freedom Caucus refused to go along.

Since then, quiet negotiations have been underway to make the AHCA more palatable to the right. Under consideration now is elimination of two provisions conservatives argue drive up premiums: requiring insurance companies to cover such things as mental health and prescription drugs, and preventing insurance companies from charging higher premiums for pre-existing conditions. There’s also talk of waivers for states to opt out of certain regulations.

That may draw right-wing support but such action is threatening to drive away support from the center, the Hill reported Tuesday.

Read: Obamacare Replacement Bill Rejected By Conservatives

“While we haven't picked up any votes yet, this concept is already showing signs of losing a ton of them,” a source told the Hill.

Paul pitched his own plan to Trump: partial repeal, leaving safety nets in place.

“I think the compromise could be keeping some of the underlying things in Obamacare, some small percentage of them in order to placate the people who want that but not affirmatively putting it in the bill and saying conservatives you have to vote for [things you’re uncomfortable with],” Paul said. “Conservatives want 100 percent repeal, let’s say moderates want 80 percent repeal. Let’s vote for 90 percent repeal and be done with it.”

Centrists weren’t happy with the original legislation, fearing constituents would be impacted negatively. Now they fear the community rating provision of Obamacare, which forces insurers to charge subscribers in a given area the same price, regardless of health status, is on the way out.

“I appreciate the state’s rights argument but recognize that there’s a reason behind community rating and the benefit that it brings to the insurance reforms,” Rep. Tom Reed, R-N.Y., told reporters.

“I want to make sure there is no denial of coverage based upon a pre-existing condition,” Rep. Leonard Lance, R-N.J., said, noting that allowing insurers to jack up premiums for pre-exisiting conditions effectively would price people out of the market.

Ryan said he’s not ready to roll out another bill yet, describing the talks as at the concept stage.