Actor and comedian Bill Cosby in Norristown, Pennsylvania, Feb. 2, 2016. Reuters

A judge in Pennsylvania refused to dismiss a sexual assault case against Bill Cosby on Wednesday, clearing the way for a criminal prosecution of the comedian, who has long denied allegations of assault by dozens of women, Reuters reported.

The ruling was a setback for Cosby, who has starred in films, TV shows and stand-up comedy appearances for decades.

In 2004, Cosby, 78, was accused of drugging and sexually assaulting Andrea Constand, a former Temple University employee, though he has said the instance in question was consensual, USA Today reported. While about five dozen women have said Cosby drugged and sexually assaulted them, Constand’s claim is the only one that has made it to court.

Cosby’s legal team wanted the sexual assault case against him thrown out, arguing that he made a binding commitment with a prior district attorney years ago that said Cosby would not be prosecuted, the Associated Press reported. Bruce Castor, a former district attorney from Montgomery County , Pennsylvania, declined to prosecute Cosby in 2004 because of a lack of evidence, prompting Constand to sue the former television star in civil court.

Cosby’s defense team has said the alleged deal stipulated he wouldn’t be prosecuted if he testified in Constand’s suit. Castor has said there was an oral non-prosecution agreement worked out with Cosby’s former attorney Walter Phillips Jr., who died last year.

The current district attorney, Kevin Steele, has said there is no evidence an agreement exists in writing — something Cosby would need to have his case tossed. Even if there wasn’t a written agreement, a spoken one may be enforceable, according to some legal experts.

"An oral agreement is just as binding as a written agreement, with some exceptions," Kevin Sali, a Portland, Oregon-based defense attorney told USA Today. "If one prosecutor makes an otherwise enforceable deal, it is binding on subsequent prosecutors."

In his deposition made more than 10 years ago, Cosby admitted to giving quaaludes to women with whom he wanted to have sex and said he gave Constand three pills. He said at the time he had affairs with young women.

Constand has said she met Cosby in 2002 and thought of him as a mentor and a friend, CNN reported. A confidential settlement was reached in the civil suit in 2006.