Jim Bob Duggar
Jim Bob Duggar, photographed during Rick Santorum's caucus night event on Jan. 3, 2012, in Johnston, Iowa, reportedly rules his family with an iron fist. Sources say he has been controlling and limiting the family's contact with the outside world following the Josh Duggar cheating scandal. Getty Images

Josh Duggar’s dark past might not be the only secret the Duggars were hiding during their more than 10 years on television. The family skyrocketed to fame with the TLC series “17 Kids and Counting,” which evolved as their already large family grew. Fans were drawn in by their loving, fundamentalist Christian ways -- but was it all an act for the cameras?

New reports from People magazine suggest Jim Bob Duggar, father of 19, rules his home with an iron fist. Sources told the publication he acts "like a dictator" and is not afraid to confront people when their lifestyles and ideals don’t align with his. The source said his authority is rarely questioned and when it is, he’s unsure how to react “because no one ever confronts Jim Bob. No one.” Another source added that the Duggar dad is, and always will be, in charge.

“Jim Bob calls the shots. Always has and always will,” the second source said.

Jim Bob, 50, is said to be using his power to control his eldest son, Josh, and daughter-in-law Anna Duggar following the Ashley Madison hack. Anna, 27, reportedly has had little contact with anyone outside of the Duggar family since news of the scandal broke on Aug. 19. Sources say this should be cause for concern. The Duggar family and others like them avoid talking about sex and sexuality, a stance that many have blamed for Josh’s behavior. Because of this, it seems likely that the family may try to sweep the scandal under the rug, as they've done in the past.

Sources told People magazine they had always been a bit leery of Josh, 27. The father of four met his now-wife in 2006 at a Christian convention, according to the magazine. He and Anna, both from fundamentalist families, agreed to enter into a courtship. At the time, neither Anna nor her family knew Josh had been accused of molesting five young girls when he was 14. Sources say he was honest with them and, despite his misconduct, Anna’s parents “still gave their blessing.” The source, however, was not sold on Josh being the godly man he said he was, but says, “he sold it so well.”

It has since been revealed that, in addition to the molestation scandal, Josh had two separate profiles on Ashley Madison, a popular site that helps facilitate extramarital affairs. Gawker released photos showing that, during his time on the site, Josh paid $986.76 for two accounts. He also is said to have paid for an affair guarantee, which the site promises will increase your chances of cheating. The accounts were canceled in May 2015, around the same time police reports regarding Josh’s earlier “inappropriate touching” became public.

Josh has admitted to having been unfaithful, but chalks it up to a pornography addiction. He, Anna and their four children are said to be hiding out in their mountain home in Arkansas waiting for the scandal to die down. The rest of the Duggar family has been holed up in the Duggar household, avoiding the public eye.