Bruce Jenner might be OK with telling the world that he’s a woman trapped in a man’s body, but his family is still struggling to accept his new gender identity. During his interview with Diane Sawyer, Jenner admitted that while he’s been keeping his true self a secret from the public, his daughters actually caught him wearing female clothing.

Kim Kardashian was the first to catch Jenner wearing a dress, and according to the Olympian, she left the house and went for a drive. He tried to talk to her about it, but she wasn’t ready at the time. Meanwhile, his two youngest daughters, Kendall and Kylie Jenner, accidentally videotaped him looking at himself in the mirror wearing one of his dresses. Kendall had been trying to catch Kylie stealing her clothes, but instead she found her dad cross-dressing. Like Kim, the topic got pushed under the rug.

However, a couple months ago Jenner decided that he had to stop living a lie and sat down with his kids in private. To his surprise, Khloé Kardashian took it the hardest. As the most open-minded and fun, Jenner thought that she’d be the most willing to accept his new identity. But that wasn’t the case, and Jenner understands. For Khloé, the situation is emotional because she has had so many losses in her life. She sees Jenner as the man in her life, and he understands why she feels like her “whole life is falling apart.”

“I can’t argue that point,” he said. “But I’m not going anywhere.”

Kourtney was accepting and just concerned about her young kids, but like Khloé, Kim also had a hard time hearing the news from Jenner. However, it took one person to help Kim accept her step dad’s new identity – her husband, Kanye West.

Jenner relayed what Kim told him, that West said that nothing in the world matters – like his beautiful wife and daughter – if he can’t be true to himself. Since then, Jenner said that Kim has been the easiest to talk to.

Kourtney, Khloé and Kim didn’t release a statement to Diane Sawyer, but Kendall and Kylie did. “We love our dad very much,” they stated. “He has the biggest heart and we just want him to be happy.”