Police arrested Ariel Castro and two of his brothers in connection with the kidnapping and imprisonment of three women in Cleveland.

In an interview with ABC News, Juan Perez, a neighbor of the 52-year-old suspected kidnapper, claimed to have known Castro for years without ever suspecting him of any wrongdoing.

"My heart is feeling rough right now to know that this happened two houses from me and that none of us noticed anything," Perez told ABC News. "I feel ashamed of myself and my community right now and this neighborhood that we didn't see anything."

According to Perez, Castro was known throughout the neighborhood as a “charismatic” person who always wanted to play with local kids, at times offering them rides on his bicycle or four-wheeler, ABC news reports.

Police confirmed to ABC News affiliate WEWS-TV that Castro is in custody and could face charges as early as Tuesday. Local authorities believe that Castro, a former city school bus driver, was holding Amanda Berry, Gina DeJesus and Michelle Knight hostage in the basement of his two-story home, ABC News reports.

Castro had brushes with the law before. In 1993, he was arrested on a domestic violence charge, but a grand jury declined to indict him, USA Today reports. Later that year, Castro pled guilty to a disorderly conduct charge, WEWS reports.

Police have yet to release of the names of the other two Castro brothers arrested in connection to the Cleveland kidnappings. However, investigators believe that the men, between ages 50 and 54, aided Castro in his efforts.

Berry, DeJesus and Knight went missing between 2000 and 2004 in separate kidnappings and had not been seen since. However, the missing women were rescued Monday night after neighbor Charles Ramsey helped Berry escape, ABC News reports. Ramsey and Berry each dialed 911, alerting police to the situation and facilitating the rescue of DeJesus and Knight.

During her call, Berry identified Castro as her kidnapper, ABC News reports. The three women were taken to Metro Health Medical Center, where they are “doing well,” according to police. They were released on Tuesday.

Ramsey, who has become an Internet sensation for his role in the rescue, said Monday that he barbecued with Castro and never suspected that he was involved in criminal activity.

"He just comes out to his backyard, plays with the dogs, tinkering with his cars and motorcycles and goes back in the house," Ramsey said. "Sometimes, you look and then look away, because he's not doing nothing but average stuff. Ain't nothing exciting about him. Well, until today."