Updated as of 4:06 p.m. EDT: Former Speaker of the House Dennis Hastert was allowed to go free on a $4,500 bail on Tuesday after pleading not guilty to charges that he lied to federal agents and attempted to pay $3.5 million to a victim to hide past sexual abuse allegations. After his attorney delivered his plea to the judge in Chicago, Hastert was allowed to walk free after he agreed to provide DNA samples, surrender his passport, avoid anyone involved in the case and to remove any firearms from his home, according to CBS News.

The federal judge presiding over the case also said that he will give prosecutors until Thursday to say if they want him to recuse himself, according to the Associated Press. The judge gave political contributions to Hastert’s congressional runs in 2002 and 2004, totalling $1,500.

U.S. District Judge Thomas M. Durkin was an attorney at a firm in Chicago at the time of the contributions, and says he has no doubt that he can be impartial in Hastert’s trial. The arraignment on Tuesday, in which Hastert plead not guilty to all counts, gives the attorneys a chance to request that the judge recuse himself, though they have yet to do so.

Elsewhere, a man was taken into custody as Hastert left the courthouse. A New York Times reporter identified the man as a journalist in a tweet.

Original story: Former Speaker of the House Dennis Hastert arrived in a Chicago federal court Tuesday and pleaded not guilty to both charges he is facing after allegations surfaced that he sexually abused former students when he was a high school teacher before his political career and that he later paid a victim as a cover up. Included in the charges are allegations that Hastert tried to pay off individuals related to the case with $3.5 million in hush money.

Hastert also allegedly lied to federal investigators about paying the cash to his victims. There have been reports indicating at least three victims were involved. 

After leaving the House of Representatives, one of his victims allegedly approached Hastert to ask for the bribe. Hastert, in an attempt to cover up banking reporting requirements, allegedly withdrew less than the required disclosure threshold for the series of payments. Between 2010 and 2014, Hastert allegedly withdrew $1.7 million in small sums after he and the victim came to the $3.5 million total payment agreement.