Meghan Markle and Prince Harry are leaving Kensington Palace. The two announced late last year that they’d move to Frogmore Cottage in Windsor, but now a report claims that Queen Elizabeth II was the one who felt the couple should move out.

Meghan and Harry have the two-bedroom Nottingham Cottage at Kensington Palace, but they’d need to upgrade once starting their family. The Feb. 25 print issue of InTouch claims that they were planning to move into Apartment 1, right next to Prince William and Kate Middleton. However, the feuding made the Queen suggest they leave.

“She had no other choice than to sit Harry and Meghan down to discuss alternate living arrangements,” an anonymous source claimed. “She didn’t point fingers or start making accusations but in her own diplomatic way suggested that they’d be happier living outside the palace, and they’d be better off moving.”

As soon as their departure was announced, rumors started to swirl that Meghan and Harry were leaving due to the Fab Four feuds. There are rumors that Meghan and Kate don’t like each other, but many say that the women simply aren’t close. Instead, other reports indicate that it’s the brothers who are really fighting. William reportedly didn’t give Meghan the warm welcome that Prince Harry felt was necessary, and the two have been at odds ever since.

Of course, there might be some less dramatic reasons behind the departure. London’s Kensington Palace is rather busy between all the royals living there and the tourists admiring the place. The Duke and Duchess of Sussex would just prefer a little privacy as they start their family.

It’s not just a physical split, though. As previously reported, the Sussexes and Cambridges are going to have separate offices and staffs.