Anti-Malware Testing

[Updated 22nd June 2018]

Here are a few testing-related resources, some of which I have (or have had) something to do with, some not. Caveat: this is a work in progress. Obviously, I can’t guarantee that links on sites I don’t maintain will stay current: let me know if you spot any broken links (I’ve just removed rather a lot that had been fractured since I first posted this page). I may gather together a few more resources in due course in preparation for a publishing project.

This page is maintained independently of AMTSO. However, while I’m not AMTSO’s biggest fan, I’m generally in agreement with its aims, if not always its methodology. And in general, its activities have helped to raise the standard of testing in general to a much higher level than it would have achieved if AMTSO had never existed. So you’ll find quite a few references to AMTSO resources below. (Unfortunately, several of those resources have disappeared or moved: hence the current spring-cleaning.)

Testers

I’m not the biggest fan of the testing industry, either. Some testers, even if their intentions are good, do more harm than good. There are well-intentioned testers, competent testers, testers with hidden agendas that are not necessarily in the best interests of the community at large (being in some instances intended to promote a single vendor), and testers who – however good their intentions – know as much about testing as I do about quantum mechanics. In the words of Vesselin Bontchev – a man who knows more than most about testing – ‘…all anti-virus testing outfits generally fall into two categories – incompetent and incomplete. (Of course, some are both.)’ Harsh, but not unfair.

Still, good testers do their best to fulfil a vital function. Evaluating security software, whether for your own use or for corporate use, is hard. (Trust me on that – it’s pretty much how I got into the security industry in the first place, evaluating and implementing security technology for a medical research organization.) Honest testers who try to provide accurate guidance to their readers perform an essential service. Imagine an anarchic world where there was nowhere to go for an impartial and reasonably well-informed opinion on how products perform in the real world, and you had to check it all out for yourself. (Well, some vendors say that they would like you to do exactly that, but what they really mean is that they’d like you to get your testing tools and samples from sites that favour their products…)

I’ve been writing about this for decades now, but this paper* – “The (Testing) World Turned Upside Down” – is probably my last word on the subject – on the conference circuit at any rate – and I’d recommend that you read it if you’re interested in why I’m cautious about making active recommendations of testing sites. (Let alone of individual products: since I currently provide consultancy to a security company, it would be pretty flaky if I was to offer product recommendations.)

After that long preamble, here’s a (rather short) list of the test sites that I consider reasonably reliable:

• Virus Bulletin: VB Testing – “Virus Bulletin’s testing and certification services” Offers testing reports for malware detection, spam detection, and web filtering, and doesn’t make spurious claims to be the source of all testing wisdom.
• SE Labs – “Intelligence-led testing” Simon Edwards’ scrupulous descriptions of the lab’s testing methodology are an excellent example of what the testing industry in general should be aiming for in terms of transparency and disclosure. The SE Labs blog is well worth following, too.
• AV-Comparatives – “Independent Tests of Anti-Virus Software” Tests and other commentary on a wide range of platforms. I don’t always agree with them, but they’re good guys and do their best to get their testing right.
• AV-Test – covers a wide range of tests and commentary: their current work on IoT (“Internet of Things”) testing is rather interesting. I’m not always convinced by their methodology, but they do have a lot of experience.

All these organizations have a long history of engagement with AMTSO – we’ll get onto that shortly – though that hasn’t always been a peaceful, uncontroversial relationship. There is also a list of AMTSO members   (vendors and testers) here.  You may find this useful in a number of respects, but bear in mind that membership of AMTSO is not in itself a guarantee of probity and transparency.

You may notice that there are no consumer magazine testers represented here. (Virus Bulletin was a paper magazine at one time, but it’s always been more than that, notably the organizer of a very important security conference, as well as a specialist testing lab.) Most consumer magazines don’t have the expertise or resources to do competent product testing: some outsource the actual testing to professional independent sites like the ones listed above, but even then, misunderstandings and distortions of test results can creep in. There are no multi-scanner sites like VirusTotal, either, because they’re not suitable for comparative product testing, though they’ve been misused for that purpose many times. In fact, VirusTotal’s Julio Canto and I wrote about that issue some years ago: Man, Myth, Malware and Multi-Scanning.

*You can also get a PDF version here.

AMTSO

The Anti-Malware Testing Organization has four aims according to its current charter:

• Providing a forum for discussions related to the testing of anti-malware and related products.
• Developing and publicizing objective standards and best practices for testing of anti-malware and related products.
• Promoting education and awareness of issues related to the testing of anti-malware and related products.
• Providing tools and resources to aid standards-based testing methodologies.

Its own guidelines documents are listed here and currently include:

• AMTSO Fundamental Principles of Testing
• AMTSO Best Practices for Dynamic Testing
• AMTSO Suggested Methods for the Validation of Samples
• AMTSO Best Practices for Testing In-the-Cloud Security Products
• AMTSO Guidelines for Testing Network-Based Security Products
• AMTSO Issues Involved in Creation of Samples
• AMTSO Whole Product Testing Guidelines
• AMTSO Guidelines to False Positive Testing
• AMTSO Guidelines on Facilitating Testability
• AMTSO Use and Misuse of Test Files in Anti-Malware Testing
• AMTSO Sample Selection for Testing
• AMTSO Performance Testing Guidelines
• AMTSO Guidelines on Mobile Testing
• AMTSO Guidelines for Testing Protection Against Targeted Attacks
• AMTSO Testing Protocol Standard for the Testing of Anti-Malware Solutions v1.0

If you’re new to this, you should at least read the AMTSO Fundamental Principles of Testing. In some respects it’s the nearest AMTSO has come to date to meeting its object of ‘promoting education and awareness’.

The AMTSO resources page used to include both papers and links from outside the organization. Unfortunately, it seems to have disappeared in recent years.

Here’s an article for Virus Bulletin from 2010 on what AMTSO had achieved so far, and what might lie ahead: ‘AMTSOlutely Fabulous’, January 2010, Virus Bulletin. Would I hold to those views now?  Now also available on this site.

Apple Testing

While OS X threats have become a significant problem over recent years (though tiny compared to the sheer volume of Windows malware), comparative tests have been rarer than hen’s teeth. AV-Comparatives did do a static test in 2012 which was by no means worthless, though it went against the flow of whole product testing. I blogged about it for Infosecurity Magazine at the time, and kind of picked up the theme later in a paper co-written and co-presented with Lysa Myers: Mac Hacking: the way to better testing? (for the Virus Bulletin conference in October 2013.)

CARO

Here’s a whole bunch of presentations from a CARO (Computer Antivirus Research Organization) workshop in Iceland in 2007 that was one of the drivers for the formation of AMTSO: Presentations made at AV Testing Workshop 2007 (wish I could have been there…)

ESET

These are most of the relevant conference papers I’ve written/co-written and presented on behalf of ESET.

• Daze of Whine and Roses
  Paper by David Harley and Larry Bridwell, presented at the Virus Bulletin Conference 2011 in Barcelona
• Real Performance?
  Ján Vrabec and David Harley: “Real Performance?” presented at the EICAR 2010 conference in Paris. (On performance testing rather than detection testing.)
• AMTSO: the Test of Time?
  Article by David Harley for Network Security Magazine. Pre-proof copy here.
• After AMTSO: a funny thing happened on the way to the forum
  Paper by David Harley presented at EICAR 2012
• Antivirus Testing and AMTSO Has anything changed?
  David Harley, presented at the CFET Conference 2010 in Canterbury, UK
• With Andrew Lee. Call of the WildList: Last Orders for WildCore-Based Testing?  
  
• With Peter Košinár, Juraj Malcho, Richard Marko: direct link to paper – AV Testing Exposed
• Unfortunately, of a number of very relevant papers presented at the AVAR (Association of Anti Virus Asia Researchers) conference between 17th and 19th November 2010, I only have access to one at this point due to web site changes at AVAR: Test Files and Product Evaluation: the Case for and against Malware Simulation, by David Harley, Lysa Myers, and Eddy Willems
• At EICAR 2009: Execution Context in Anti-Malware Testing
• With Andrew Lee for Virus Bulletin 2008: Who Will Test The Testers?
• With Andrew Lee for AVAR 2007: Testing, testing: Anti-Malware Evaluation for the Enterprise

WildList Organization

Papers on the WildList organization web site. Although the web site is mostly dormant, this page is still there., but some of the links are now broken. These still seem to be active:

• Joe Wells on How Scientific Naming Works
• Sara Gordon on What is Wild?
• Richard Ford and Sara Gordon on Reviews and Evaluation of Antivirus Software: The Current State of Affairs
• Sara Gordon and Fraser Howard on Antivirus Software Testing for the Next Millenium

Miscellaneous 

This is quite a mixture of old and new, but these papers and articles are still available at time of writing.

Benchmarking Without Weightings: Like a Burger Without a Bun
Blog by Eugene Kaspersky, September 2011

Anti-malware testing discussions
Blog by Simon Edwards, September 2011

Why there’s no one test to rule them all
Article by Lysa Myers for Virus Bulletin, October 2011

The Holy Grail of AV Testing, and Why It Will Never Be Found
Blog by Eugene Kaspersky, October 2011

Real World Reviews: Current State
Sarah Gordon and Richard Ford: “Real World Anti-Virus Product Reviews And Evaluations – The Current State Of Affairs”

A Reader’s Guide to Reviews
Dr. Alan Solomon: “A Reader’s Guide to Reviews” (originally published in “Virus News International”, and credited to Sarah Tanner)

David Harley