Republican presidential candidate Ben Carson has been under fire for anti-Muslim comments. Reuters

In the days after the Sept. 11, 2001, terror attacks, the leader of the GOP had a message that ran contrary to the modern-day position of 2016 Republican presidential candidates. "The face of terror is not the true faith of Islam. That's not what Islam is all about," President George W. Bush said a week after the attacks that killed thousands. "Islam is peace."

Nearly two decades later, the GOP has struggled to win over black, Latino and female voters as the nation's changing demographics have ensured Republicans can no longer count on white men alone to win the White House. But unlike those electoral groups, which have long protested the GOP's conservative positions on low-income and minority communities, Muslim voters were once a solid Republican voting bloc. Their turn away from the GOP in recent years amid growing anti-Muslim rhetoric after Sept. 11, 2001, suggests Republicans are poised to lose more minority voters in the future as the nation becomes more diverse.

"We don't put people at the head of our country whose faith might interfere with them carrying out the duties of the Constitution," retired neurosurgeon Ben Carson, one of the top Republicans in the 2016 race, said Monday. He was doubling down on Sunday statements in which he questioned a Muslim's ability to practice Islam while respecting the Constitution. "If you're a Christian and you're running for president and you want to make this [country] into a theocracy, I'm not going to support you. I'm not going to advocate you being the president."

Carson isn't the only candidate who has been at the center of anti-Muslim talk. Donald Trump, the Republican front-runner, recently listened to a supporter call President Barack Obama a Muslim at a rally. Trump didn't correct the man and has refused to defend the president since.

After 9/11, American Muslims largely switched sides to the Democratic party, noting that Republican policies had made their lives more difficult than they were before the attacks. A 2011 Pew survey found that Muslim support had flipped in just over 10 years after 78 percent of Muslims backed Republicans in the 2000 election. By 2011, 70 percent of Muslims identified as Democrats, and 11 percent leaned toward Republicans. An informal exit poll conducted by the Council on American-Islamic Relations in 2012 determined that 85 percent of Muslim voters in the election broke for President Barack Obama over his Republican challenger, Mitt Romney.

The shift comes amid growing suspicion toward Muslims among Republicans. A 2012 Arab American Institute poll found that 60 percent of Republican voters had a negative image of Muslim Americans. Many Republicans also see President Barack Obama as Muslim, compared with 29 percent for the American population as a whole. In separate polling, 51 percent of all Americans said Muslims want to remain a distinct culture from the rest of the country, and 47 percent said that American and Muslim worldviews are incompatible.

To be sure, Muslims aren't a make-or-break constituency in presidential or federal elections. Their numbers are increasing, however, at the same time that the number of Americans who identify as Christian, the largest religious group in the country, is seeing a sharp decline. Muslims represented 0.4 percent of the population in 2007, according to a Pew survey, and rose to 0.9 percent in 2014. The Christian population dropped from 78.4 percent to 70.6 percent during that time.

While Muslim voters may be a small portion of the American population, they are concentrated in some of the states that could play a decisive role in 2016, including Florida, Ohio, Virginia and Pennsylvania. Those states can be decided by razor-thin margins. At that point, every constituency matters. Whether or not Republicans have pushed Muslim voters away to attract white conservative voters in the primary could have an impact on who gets the White House next.