The Justice Department doesn’t yet know how wide-ranging the IRS-tea party scandal is, U.S. Attorney General Eric Holder testified Wednesday.

Appearing before the House Judiciary Committee, Holder said he ordered a criminal investigation into the IRS activities on Friday.

When asked by U.S. Rep. Steve Chabot, R-Ohio, whether the scandal involves “just low-level IRS workers,” Holder said it’s too early to tell.

“I simply don’t know at this stage. We’ve only begun our investigation and it will take us time to determine who was involved in these matters,” the attorney general said.

Holder said that while inquiry of groups that seek 501(c)(4) status, which would exempt them from paying taxes, is appropriate, focusing the probes based on political affiliation is out of bounds.

“This [investigation] will not be about parties, this will not be about political persuasions” Holder said.

Chabot also chided the Obama administration for what the congressman said was a lack of accountability.

“I believe there’s been a pattern by this administration in not taking responsibility for failures, avoiding blame, pointing the fingers in somebody else’s direction. Do you agree?” the congressman asked Holder.

“No,” the attorney general responded.

After Chabot strongly recommended that the administration get out front on scandals and be forthcoming, Holder said he agreed.

“It is one of the reasons why I ordered the investigation last Friday, because it seemed to me there was the need for review … that the Justice Department needed to be ahead of this mater.”

While Holder couldn’t specifically comment on the IRS scandal investigation because it’s ongoing, he said potential crimes include civil rights violations and making false statements.

U.S. Rep. Ted Poe, R-Tex., said the Justice Department lacked credibility and asked Holder if it would be best to appoint a special prosecutor to look at teh IRS scandal.

Poe relayed a story about a constitutent who launched an organization with "patriots" in the name and was visited six times by the FBI as well as other federal agencies. He said the IRS has since apologized about her case but has yet to grant her 501 (c)(4) status.

Holder disagreed with Poe's assertion about how his office operates.

"The notion that the Justice Department has credibility problems is belied by the notion that people have of government, of the good [that] government does," he said.