Republican Paul Ryan has come under fire barely two months into his role as speaker of the house for his work alongside Democrats to pass a $1.1 trillion spending bill. But, more bizarrely, some of the attacks are focused on his facial hair.

In what scores of Twitter users have dubbed his “Muslim beard,” Ryan’s newly grown whiskers have attracted speculation from journalists, pundits and even members of his own party, who have wondered what's behind his decision to grow the beard out and whether it signals some sort of political statement. GQ called the beard “the eyeroll-inducing style move equivalent to your father showing up to dinner on a skateboard,” while the National Review suggested Ryan “grow the economy, not facial hair. Cut taxes, as well as whiskers.”

The suggestion that Ryan’s facial hair indicates a conversion to Islam was not restricted to social media posts. Ryan reportedly told President Obama in a phone call that some on the political right have accused him of being Muslim, according to a Bloomberg View column.

The far-right members of Congress grew furious this month with Paul’s cooperation on the omnibus spending bill, and conservative pundits — such as conservative talk radio host Rush Limbaugh — claimed Ryan sold his party “down the river,” the Hill reported. But while members of the Freedom Caucus, who were responsible for overthrowing previous Speaker John Boehner, nearly unanimously voted against the deal, they may be more understanding toward Ryan’s decision than they let on.

“I think most Freedom Caucus members hated the omnibus product but acknowledge that Speaker Ryan could only do so much within the parameters that he had to work with,” an unnamed Freedom Caucus leader told the Hill. “But so many grassroots supporters have been disappointed so many times that they can see no difference.”