“Colbert Report” fans are abuzz for the show's former host Stephen Colbert to take over from David Letterman on CBS’ “The Late Show,” but Letterman thinks the network may have missed a big opportunity when choosing his replacement. The veteran television host thinks CBS could have taken steps to introduce some more diversity into the late-night lineup.

In a recent interview with the New York Times, Letterman said Colbert will do a great job with the show, but he also expressed some regret that the network did not go in another direction.

"I always thought Jon Stewart would have been a good choice. And then [Stephen Colbert]. And then I thought, well, maybe this will be a good opportunity to put a black person on, and it would be a good opportunity to put a woman on," Letterman said. "Because there are certainly a lot of very funny women that have television shows everywhere. So that would have made sense to me as well."

Letterman went on to note that he was not consulted in selecting his successor.

There has been dramatic turnover in late night programming over the last two years – Jimmy Fallon replaced Jay Leno on NBC’s “The Tonight Show” and was in turn replaced by Seth Meyers on “Late Night”; James Corden replaced Craig Ferguson on CBS’ “The Late, Late Show.” But no network late night show has promoted anyone other than a white man. Only Comedy Central – which replaced Colbert with Larry Wilmore’s “The Nightly Show” and will replace “Daily Show” host Jon Stewart with South African comedian Noah Trevor – has made efforts to diversify the genre.

Colbert will take over in the fall after Letterman’s last show on May 20, but viewers will be left wondering about what could have been.