AV-Comparatives Testing with AMTSO List

The Austrian security product testing lab AV-Comparatives has announced what it describes as ‘the first public industry test using the AMTSO Real Time Threat List (RTTL).’

The Anti-Malware Testing Standards Organizations describes the RTTL as follows:

The Real-Time Threat List (RTTL) is a repository of malware samples collected by experts from around the world. The repository is managed, maintained and secured by the Anti-Malware Testing Standards Organization (AMTSO).

Ah, I hear you say, isn’t that what the WildList Organization does? Well, yes, but RTTL is supposed to do it better, according to the abstract for a 2013 presentation for Virus Bulletin by Righard Zwienenberg, Richard Ford, and Thomas Wegele: The Real Time Threat List.

The abstract states:

In particular, the change in the nature of online threats has left the WildList trailing the ‘real-time’ threat, making it unsuitable for effective ‘in-the-wild’ testing.

And that is undeniably true, as I and others have said many times. For instance here:

It may be that until we see methodologically sound ‘testing of testers’ we will have to accept that there is still a place for ‘best endeavours’ testing based on sample sets that lack freshness but are known to have been validated. Clearly, though, certification based entirely on detection of WildCore samples can no longer be regarded as a sufficient guarantee of a product’s effectiveness. It’s fortunate, then, that the reputable organizations making use of WildList testing are already using or moving towards methodologies that hold onto what is good about the WildList while adding functionality and value by bolstering it with fresher samples and more dynamic technologies.

The VB presentation on RTTL here goes just a little further than the abstract in 2013:

Something new, fast and “accurate” had to be created, which eventually resulted in the conception of the Real Time Threat List…The idea of the Real Time Threat List is to share new threats with additional meta-data incorporated into the system.

AMTSO’s terse FAQ doesn’t really add much in the way of information. There has been plenty of discussion at AMTSO meetings, of course, but AMTSO seems to be clinging to its reluctance to share information outside those meetings, even to AMTSO members, though I keep hearing rumours of an occasional chink of light breaking through at least one of those walls.  We shall see.

AV-Comparatives is, as you’d hope and expect, reasonably forthcoming about its methodology in the test report. It took the ‘Top500’ samples from RTTL as of 9th March 2016, and filtered out the 247 samples that didn’t display malicious behaviour in its sandboxes, in the expectation of removing false positives and ‘possibly unwanted’ samples. It used its own ‘Real World Protection Framework’ to execute each sample simultaneously against security programs configured with default settings and cloud access. All vendors were, in this instance, AMTSO members. And I don’t object to it in principle, since the description should tell any reasonably well-informed reader enough to evaluate the validity of the tests. But what does it tell us about the validity of the RTTL? Well, it tells us that the Top500 isn’t necessarily exhaustively validated or categorized.

And that isn’t necessarily a problem. An advantage of WildCore is that it represents a notionally validated sample set is that it represents, as the WildList Organization FAQ put it,’ a set of replicated virus samples that represents the real threat to computer users.’ The corresponding disadvantage is that by definition the samples aren’t ‘fresh’ because they’ve been around long enough to undergo through some sort of validation process. Which was fine back in the 90s, when the pace of sample release and discovery was infinitely slower. I should also mention that back then, malware was almost entirely viral, whereas most malware samples nowadays aren’t viruses: that is, they don’t self-replicate. So maybe requiring the testing organization to do its own validation is an acceptable trade-off, as long as RTTL access doesn’t get as far as all those amateur (or amateurish) testers who are unfamiliar with the concept of sample validation.

But AV-Comparatives methodology described tells us nothing further about the RTTL, and AMTSO remains uncommunicative. It seems to me that by not expanding on the public details of RTTL, AMTSO is asking us simply to take its word for it that its collection is a Good Thing. And in fact, it has a lot going for it, potentially at least. But there are plenty of testers who are still prepared to say that their test sets are better than anyone else’s without feeling the need to supply evidence.

AMTSO, of course, is no more a testing organization than it is an anti-malware vendor. But if it’s providing samples to testers it might as well be, without information on the conditions under which it is distributed, and it would certainly be nice to know more about the underlying collection methodology.

Don’t get me wrong: while my formal involvement with AMTSO is at this point minimal, I remain sympathetic to its agenda and intentions, and I do believe that testing is in much better shape now than it would be if the organization had not existed. But there are plenty of other people who are less sympathetic – not to say actively hostile. Even if the information above was easily available to outsiders, they will remain suspicious of the apparently close – not to say symbiotic – relationship between vendors and testers within the organization. Sadly, that relationship is in my experience often pretty stressed, inside and outside AMTSO. However, also in my experience, members on both sides of the vendor/tester divide are generally in agreement on the need to raise testing standards and accountability for the benefit of the users of security products, and are committed to working towards achieving higher standards.

AMTSO’s caution as regards sharing information is understandable, given the mauling it has received on occasion from the media (and from vendors and testers who may be less willing to be too accountable for comfort). But in my opinion, it was too ready to give up on transparency.

David Harley