They may be happily married now and preparing to one day be King and Queen Consort of England, but that doesn’t mean things were always great between Prince Charles and Camilla Parker Bowles.

While the couple’s history is muddied because of their affair during his marriage to Princess Diana, which led to an ugly divorce, something that happened the first time they dated actually led to even more drama, according to Penny Junor’s book “The Duchess: The Untold Story.”

The pair dated initially in their 20s and was forced to separate when Prince Charles went on a tour with the Royal Navy, and despite the fact they were reportedly in love at the time, Camilla wasn’t seen as a suitable bride for the future King of England and the relationship wasn’t supported by the Royal Family. Because she knew that she wasn’t considered suitable, Camilla reportedly decided to seek out a new suitor instead—and sent a letter to the Prince to inform him when she became engaged to her eventual first husband, Andrew Parker Bowles.

“She wrote to Charles herself to tell him. It broke his heart,” Junor wrote.

She also stated that after it happened, he went off to reveal his anguish to his family, and even referred to the breakup as something cruel.

“He fired off anguished letters to his nearest and dearest,” she recalled. “It seemed to him particularly ‘cruel,’ that after ‘such a blissful, peaceful and mutually happy relationship’ fate had decreed that is should only last a mere six months.”

Still, despite his anguish, he went on to accept what happened, and Camilla went on to marry her husband in 1973. Charles, meanwhile, famously married Princess Diana in 1981. Both had two kids with their spouses before resuming their affair and eventually divorcing, respectively, in 1995 and 1996.

They went on to officially marry in a civil ceremony in 2005.