Anti-Malware Test Cheats revisited: AMTSO speaks

Here’s more about the companies that have been chastised by AV-Test, AV-Comparatives and Virus Bulletin for cheating in comparative tests.

First, AV-Comparatives announced on its Facebook page that one of the vendors participating in its tests had infringed its testing agreement by submitting a version for testing that ‘had been specifically engineered for the major testing labs.’ Since there wasn’t much in the way of hard information there, my article here on Gaming the tests: who’s being cheated? was equally sketchy on detail, but hopefully highlighted some issues by way of some commentary leavened with reminiscence.

Subsequently, a joint statement by AV-Comparatives, AV-Test and Virus Bulletin here announced that the products submitted by Qihoo for testing had the Bitdefender engine enabled by default and its own QVM engine disabled, whereas ‘all versions made generally available to users in Qihoo’s main market regions had the Bitdefender engine disabled and the QVM engine active…’

Which led me (and hopefully many others) to wonder why Qihoo was ‘apparently going out of its way to provide its customers with a default configuration that – according to the joint statement – not only demonstrates inferior detection performance, but actually impacts on usability by increasing the risk of false positives.’

Qihoo (or Qihu) subsequently attempted to answer some of the criticisms and questions on its Facebook page, claiming that the criticism of its engine was unfair because “many popular software add-ons in China that are flagged as malware by the AV-C definition are in fact performing proper functions and not malicious. Therefore, Qihoo 360 and other domestic vendors’ security products in China treat such add-ons as legitimate and non-threatening.” This may sound similar to an issue I cited in a previous blog:

In general, security products are cautious about detecting PUAs/PUPs/PUS by default, for a number of reasons. That’s problematical, though, where testers insist on using default settings and don’t filter PUAs out of their sample sets.

That’s a scenario that has irritated me for many years – which is why I cited it – but it was just an example – I didn’t know at the time that Qihoo were going to use much the same issue as a defence. In fact, as Simon Edwards quite rightly pointed out, testers (at least, reputable testers) nowadays are pretty careful about filtering correctly, and that certainly includes the three testers we’re concerned with here, so Qihoo’s argument is of doubtful relevance to the testers’ criticism. It also has a blog article offering an explanation for its preference for its own QVM engine in its public versions, and claims that the testing labs were made aware that the version supplied was configured differently. Clearly this differs from the statement made by the labs, and it seems that Qihoo has announced its withdrawal from their tests.

Next, Tencent was criticized by the same testers on somewhat similar grounds, though in this case it seems that the product (not only the version submitted for testing, but apparently all recent publicly available versions) was optimized for fast scanning by bypassing objects that are normally – and quite rightly – routinely scanned by anti-malware scanners. The conclusion, though, is pretty much the same:

These optimizations, which have been found in all recent public versions of the products, provide minimal benefit to normal users and could even degrade the level of protection offered by the products.

According to The Register, Virus Bulletin’s John Hawes comments that:

“Their software has so many feedback systems and each user was pumping the data back to Tencent’s labs.”

The Register also suggests that Baidu is also still being investigated, so perhaps there’s more to come on that. Not to mention a report that Tencent plans to take legal action against one of the labs, apparently in the hope of persuading it ‘to lift its allegations and resume all certifications and awards granted to Tencent.’

And finally, AMTSO, the Anti-Malware Testing Standards Organization, also weighed in: Why we cannot tolerate unethical behavior in the anti-malware industry. This is a big deal: when I was heavily involved with AMTSO, I and other Board members spent a lot of time debating testing issues with people outside the organization, and some of those discussions were pretty heated. AMTSO has seemed subsequently to avoid controversy, and in fact has been pretty quiet altogether, but while it doesn’t name names in its statement, it makes its position quite clear. It doesn’t approve of vendors that try to game tests, and is particularly concerned when vendors seem to be putting test scores ahead of their users’ safety. Can’t argue with that.

Well, I did have a bit more to say than that in an article for ITSecurity.co.uk but I’m pleased to see AMTSO taking a firm stand on inappropriate practice by vendors, who have been known to use the organization as a threatening response to an unfavourable review. But there are still plenty of poor tests out there: it will be interesting to see whether AMTSO will be as ready to comment on genuinely poor practice by testers when appropriate.

David Harley