Whether it’s been drug-testing regulations, a split of the purse, or the quality of his opponent, Floyd Mayweather has contended that a number of issues have prevented him from facing Manny Pacquiao in the last five years. Many of Mayweather’s detractors, including PacMan’s promoter, believe the boxer is simply afraid to tarnish his perfect record.

Top Rank CEO Bob Arum was in Denver on Thursday, promoting Saturday’s fight between Mike Alvarado and Brandon Rios. He said that the only thing holding up a potential mega-fight is Mayweather’s reluctance to get in the ring with Pacquiao.

"He's afraid to lose." Arum said of Mayweather, according to the Denver Post.

Mayweather is 47-0 in his career, and on his way to retiring with no losses. The boxer has two fights left on his contract with Showtime, and intends to call it quits after his second bout in 2015. Mayweather has taken on the best that the sport has to offer, and none of his opponents have presented him with a serious challenge. A fight with Pacquiao, though, could prove to be his toughest match in years.

No matter the opponent, Mayweather is always an overwhelming favorite to come out on top. Heading into both 2014 fights against Marcos Maidana, Mayweather was a -1000 and -650 favorite, respectively. Canelo Alvarez was expected by some to give Mayweather a hard time, considering the champ was just a -280 favorite. However, Mayweather easily took care of Alvarez, while Pacquiao could present an entirely different set of problems.

"He doesn't want to fight southpaws,” Arum said. “The worst possible opponent for him is Pacquiao.”

Mayweather last fought a southpaw on May 4, 2005, when he easily beat Robert Guerrero in a unanimous decision. He had some trouble with Zab Judah in 2006, though he won the fight in a unanimous decision, and he knocked out Victor Ortiz in 2011 with a controversial fourth-round punch. According to CompuBox, Mayweather has been largely dominant against southpaws.

Arum’s comments came on the heels of statements made by Alex Ariza, Mayweather’s strength coach. While negotiations between Mayweather and Pacquiao’s camps are ongoing, Ariza told the New York Daily News that Arum is the reason why the sport’s top two stars have never met in the ring.

Arum claims that Ariza isn’t involved in the negotiations and is looking to blame him because Pacquiao fired Ariza two years ago. According to the promoter, Pacquiao has signed off on a contract to fight Mayweather, and all that’s needed is Mayweather to agree to the terms.

"Everybody involved wants this fight, just like the fans," Arum said. "We've signed off on every point. We've agreed to everything. But push comes to shove, there's one guy to blame."

Despite the war of words and the inability of both camps to negotiate over the last five years, Arum is confident that a deal will get done. Should the fight happen, it will likely take place at the MGM Grand in Las Vegas on May 2.