Both Paula Abdul and Nicole Scherzinger will not return to the X Factor next season, after news of a big shake-up to Simon Cowell's FOX series. The two judges, along with host Steve Jones, will be leaving.

Sources close to the show confirmed all three will not return for the second season of X Factor next fall, according to Entertainment Weekly. FOX has only confirmed the departure of Scherzinger and Jones.

Simon Cowell and record producer L.A. Reid will be the only two returning judges.

Nicole Scherzinger reportedly wants to focus on her music career instead of sitting on the X Factor judging panel. She wants to focus on her music career. She's spoken to Simon [Cowell] and he's given her his blessing. X Factor took a lot more of her time than she ever expected. She didn't have any time really to work on her music, a source told E! News.

However, Scherzinger's performance has been consistently criticized by critics and fans alike.

Jones' departure is not much of a surprise. FOX entertainment producer Kevin Reilly hinted at it during the network's press tour, There will be some changes ... as we know [hosting] is a much harder job than meets the eye, he said. I think everybody has come to see the value of Ryan Seacrest. Whether Steve is the guy or not, it comes under the heading of growth in general. 

I won't be hosting next seasons X Factor which is a shame but I cant complain as I've had a great time, Jones wrote on Twitter. Good luck to everyone on the show.

Speculation has surrounded the judging table over the past few weeks, as word leaked that Scherzinger and Jones could be out. Paula Abdul was expected to return, as she has had a longstanding relationship with Simon Cowell.

Both FOX and Cowell are determined to turn the X Factor into a huge hit, as the UK series dominates the ratings. Though the U.S. version's first season performed well, garnering about 12.6 million viewers and a 4.4 average rating among adults 18-49 for its Wednesday episodes, there is room for growth.

The show's biggest competition is NBC's The Voice, showcasing powerhouse judges like Christina Aguilera and Cee Lo Green.

Who will replace the two X Factor judges?