Simon Cowell’s career has been on an upswing lately with the reveal that he’ll be returning to American reality television as a judge on the next season of NBC’s “America’s Got Talent” and his recent accolade at the Music Industry Awards. It seems the only thing he could possibly do to halt his own trajectory is upset the entirety of the popular band “One Direction” and their fans.

Unfortunately, that’s just what an admittedly tipsy Cowell did at the Nov. 2 awards show. The four remaining members of the band -- Niall Horan, Liam Payne, Harry Styles and Louis Tomlinson -- took the stage with their former “X Factor” judge and current music mentor to award him with a Music Industry Trust Award, writes The BBC. According to Daily Mail, Cowell took the stage and almost immediately admitted that he’d been drinking and made a joke about the British boy band’s recent loss of its fifth member, Zayn Malik.

As many know, the group originally consisted of the five boys, but Malik dropped out of the band in March of 2015 saying, at the time, that he hoped distancing himself from the spotlight would allow him to lead a less hectic lifestyle. Since then, the remaining four members of the group have continued to tour and make music together without him, but the topic is still a sore subject for the band and fans alike -- which Cowell found out the hard way.

The mentality behind the joke was sound, as the 56-year-old music mogul would later explain. It happened when he and the four singers were up on stage together. He made a comment along the lines of: “who needs Zayn?” The moment was awkward and neither the crowd nor the band seemed to appreciate it. Luckily, it was a quick moment and Cowell was able to move on to a much more heartfelt acceptance speech in which he commented on the music industry and how kind a place it can be. The highlights of his speech can be seen in a video by the Daily Mail. Still, after the show, Zayn fans were upset:

“I just kind of thought, when I was standing on stage with them, that they were back to being five, and I saw myself [being a part of] One Direction for a moment,” he explained to New York’s Fresh 102.7 on Nov. 6 on why the joke didn’t land (via Us Weekly). “That’s when I made the quip, ‘Who needs Zayn?’ which backfired hugely. So I apologize to any Zayn fans out there -- it was mean as a joke.”

To his credit, Cowell is among the last people one would expect to be flippant about Malik leaving the immensely popular band that he helped create. As previously reported, the reality TV personality claims he was among those that tried to convince Zayn to stay in the band just before he finally announced that he was leaving for good. Unfortunately for fans, his efforts didn’t work and One Direction has been down to four members ever since. The band has recently announced it will be taking a temporary break from music at the conclusion of its latest tour.