Massachusetts Gov. Deval Patrick Tuesday said he was unsure if Tea Party opposition to President Barack Obama is based on race, responding to actor Morgan Freeman's comments on racism in the conservative political movement.

The Academy Award winning actor said on CNN's Piers Morgan Tonight on Friday that Tea Party's opposition was a racist thing. When Patrick was asked on a Boston talk radio station about the statement, the Bay State's first black governor balked at attributing racism to Obama's Tea Party critics after the show's co-host pushed the question.

I don't know, Patrick told Jim and Margery co-host Jim Braud. One of the curses of racism in this country is that you're always asking yourself if the stuff that goes wrong is on account of racism. I hope it's not.

Patrick did, however, agree with Freeman that the Tea Party wants Obama to be a one-term president, regardless of how staunch opposition to his policies may harm the country.

I can tell you that it's clear from the evidence that the 'Heck with the interest of the common good and whatever we need to do to derail this presidency' has characterized some if not all of Tea Party behavior in the Congress of the United States, no doubt about it, Patrick said.

The governor also described conservative behavior as seditious, saying that patriots come together in crisis to work on a solution.

The notion that the singular focus of the hard right today is to defeat this president,  even if there's an idea he puts forward to help-that they used to support-is incredibly worrisome to me and a very different political climate, I think, that we've been dealing with for a long time, he said.

Patrick was not the only one to respond to Morgan's comments. GOP presidential contender Herman Cain had told Fox News that Morgan's remarks were short-sighted.

Most of the people that are criticizing the Tea Partiers about having a racist element, they have never been to a Tea Party, Cain said.