Microsoft Security Essentials Fails Antivirus Certification Test For Second Time In A Row; Microsoft Disputes The Results

These results are noteworthy because Microsoft Security Essentials is currently (as of December) the most popular security suite in North America and the world.

Microsoft disputed AV-Test's results in a blog post on Wednesday evening, defending its software and challenging the antivirus lab's findings.

“We conduct a rigorous review of the results whenever test results warrant it,” wrote Joe Blackbird, program manager at Microsoft’s Malware Protection Center. “We take the protection of our customers very seriously, and the investments we make to do these reviews is an example of that commitment.

“Our review showed that 0.0033 percent of our Microsoft Security Essentials and Microsoft Forefront Endpoint Protection customers were impacted by malware samples not detected during the test. In addition, 94 percent of the malware samples not detected during the test didn't impact our customers.”

In his review, Blackbird points out that AV-Test only reports on “samples hit/missed by category,” whereas Microsoft prioritizes its work based on consumer impact. Furthermore, AV-Test’s results indicated that Microsoft Security Essentials detected roughly 72 percent of all “0-day malware,” with a sample size of 100 pieces of malware.

“We know from telemetry from hundreds of millions of systems around the world that 99.997 percent of our customers hit with any 0-day did not encounter the malware samples tested in this test,” Blackbird said.

With missing malware entirely, AV-Test reported Microsoft Security Essentials missing 9 percent of “recent malware,” using a sample size of 216,000 pieces of malware. In rebuke, Blackbird pointed out that 94 percent of missed malware samples “were never encountered by any of our customers.”

“During the test, our products didn't detect 28 of the 0-day malware samples, and 9 percent of the recent malware samples,” Blackbird said. “AV-Test uses a minimum bar in their scoring: Our results for these two areas fell under that bar. The missed samples in both of these sections were where we focused our analysis, as we wanted to ensure we weren't missing anything impactful to our customers. When we did our review, we found that our customer-focused processes had already added signatures that protected against 4 percent of the missed samples. These files affected 0.003 percent of our customers.”

Finally, Microsoft used a retrospective analysis to inspect the remaining “missing files” to see if any customers had encountered them.

“We were looking for files that slipped through because of gaps in our telemetry or file collection process. And we found that 2 percent of these files existed across 0.003 percent of our customers,” Blackbird said. “The other 94 percent of the samples don't represent what our customers encounter. When we explicitly looked for these files, we could not find them on our customers' machines.”

AV-Test’s review looks at three key areas of security software, including protection, reparability, and usability of the whole computer based on the security software’s impact. Across those three areas, Microsoft Security Essentials scored a 1.5 out of 6 on protection against viruses and worms, a 3.0 out of 6 on a reparability scale, and a 5.5 out of 6 on the usability scale, where “lower values indicate better results.”

Out of the 25 programs tested by AV-Test, including security software from companies like McAfee, Norman, Kaspersky and others, Microsoft’s Security Essentials was just one out of three that failed to gain certification.

“We know from feedback from customers that industry testing is valuable, and their tests do help us improve,” Blackbird said. “We're committed to reducing our 0.0033 percent margin to zero.”