A massive fire broke out Tuesday morning at a Bangladeshi sweater factory that supplies Western retailers H&M and J.C. Penney. The conflagration injured at least four, but according to global labor rights advocates, barely escaped inflicting tragedy on a much deeper scale: Most of the factory’s more than 6,000 workers had not yet showed up to work.

“Had the fire started even one hour later ... the risk of death would have been extreme,” read a statement from the Clean Clothes Campaign, the International Labor Rights Forum, the Maquila Solidarity Network and the Worker Rights Consortium.

The incident, at the Matrix Sweater Factory in Gazipur, occurred just days after the labor groups released a report criticizing delays in safety improvements at H&M suppliers in Bangladesh.

A November 2012 fire at the Tazreen Fashion factory in the nation’s capital of Dhaka killed at least 117 people, bringing global attention to workplace safety in Bangladesh, one of the world’s top garment exporters. The Rana Plaza building collapse in April 2013 killed more than 1,100 people, triggering more scrutiny.

Since then, two groups, the U.S.-based Alliance for Bangladesh Worker Safety and the largely Europe-based Accord on Fire and Building Safety in Bangladesh, have overseen inspections and initiatives designed to improve factory safety. Critics say the renovations haven’t taken place quickly enough.

The former has reported that safety renovations at Matrix Sweater are “on track.” The latter has said that 63 percent of renovations at the factory remain incomplete.

“It is astonishing that the alliance has Matrix Sweaters rated as ‘on track’ with safety renovations,” Scott Nova, executive director of the Worker Rights Consortium, said. “This factory has missed dozens of deadlines to eliminate fire hazards and make the structure safe, with 72 different hazards still uncorrected almost two years after inspection. Just how dangerous does a factory have to be to earn criticism from the alliance?”

“We are extremely relieved that this fire hasn’t resulted in another tragedy on the scale of the Tazreen factory fire of 2012,” Sam Maher, a coordinator at the Clean Clothes Campaign, said. “However, this is more [due] to luck than anything else."