Prince Harry and Meghan Markle made waves with their expensive renovations to their home at Frogmore Cottage, but the couple’s residence wasn’t the place where they initially intended to live—but Queen Elizabeth put her foot down on their plans.

According to a recent report from the Sunday Times (via Express UK), the Duke and Duchess of Sussex initially hoped to live at Windsor Castle itself with Queen Elizabeth and Prince Philip after their marriage. However, the Queen quickly put her foot down and made an end to the plans.

“The couple are understood to have set their hearts first on Windsor Castle, and are believed to have asked the Queen if living quarters could be made available after their marriage,” writer Rosa Nikkah stated.

However, she went on to explain that the Queen turned down the request and gifted them Frogmore Cottage instead. According to royal author Hugo Vickers, the Queen likely decided to do it because it wouldn’t really work to have their young family—which now includes son Archie Harrison—living in the same space as the reigning monarch.

“There are empty bedrooms and suites in the private apartments within Windsor Castle which the Sussexes have had their eye on, or perhaps some former living quarters in the castle grounds converted into other things,” he said. “But I can see how it might not be entirely appropriate to have a young family living there.”

However, while the couple did settle at Frogmore, the renovations they made to the property—which included a price tag of £2.4 million in taxpayer money—have been contentious to the public. The couple reportedly paid for specific furnishings themselves, but the replaced ceiling beams, floor joists, heating, gas, plumbing and electrical systems were all paid for by taxpayers.

Prince Harry, Meghan Markle and Son Archie
Prince Harry and Meghan Markle are pictured presenting their newborn son, Archie Harrison Mountbatten-Windsor, during a photocall in St. George’s Hall at Windsor Castle on May 8, 2019 in Windsor, England. Dominic Lipinski-WPA Pool/Getty Images