Queen Elizabeth had a falling out with the Duke and Duchess of Windsor decades ago. And when she was asked to mend her broken relationship with them, the 93-year-old monarch reportedly refused.

The Queen’s former assistant private secretary, Edward Ford, said that he suggested for the Queen to invite Edward VIII and Wallis Simpson to the Royal Ascot, but the monarch gave him a firm “no” as a response.

Journalist Kenneth Rose wrote about Ford’s revelation in his private journals, which will be published this month.

“Edward Ford tells me that he used to suggest to the Queen that she might publicly heal the breach with the Windsors by inviting them for a day or two of Ascot races, where they would be swallowed up among the other guests. But the Queen said ‘no,’” he wrote.

Edward VIII left London after he abdicated from the throne in December 1936. He decided to choose his relationship with Simpson over the crown. His abdication resulted in the Queen’s father, King George VI, to become the King of England. And the abdication also altered the order of succession.

Her Majesty wasn’t supposed to be Queen but since she is her dad’s eldest daughter, she became an heir to the throne when King George VI became King. After Edward VIII chose love over his duty, they were reportedly banned from the United Kingdom.

But in 1967, the Queen decided to lift off the couple’s ban when she asked them to attend the royal ceremony to dedicate a centenary memorial to the Queen’s grandmother, Edward’s mother, Queen Mary.

Edward VIII and Simpson have resided in Paris for years and they only traveled back and forth to the United Kingdom for their short visits. In 1952, Edward VIII was also allowed to return to the United Kingdom alone to attend the funeral of his brother, King George VI. Twenty years later, Simpson went to the UK to attend Edward VIII’s funeral.

Britain's Queen Elizabeth II
Britain's Queen Elizabeth II announced a list of new bills ranging from implementing a yet-to-be finalized EU divorce agreement to criminal sentencing. POOL/Tolga Akmen