Testing Resources

Here are a few testing-related resources, some of which I have (or have had) something to do with, some not. Caveat: this is a work in progress. Obviously, I can’t guarantee that links on sites I don’t maintain will stay current: let me know if you spot any broken links. (Comment here, or for a (normally) quicker response, antimalware[dot]testing[at]gmail[dot]com.

In case I have to say so yet again, this page is maintained independently of AMTSO.


The Anti-Malware Testing Organization has four aims according to its current charter:

  • Providing a forum for discussions related to the testing of anti-malware and related products.
  • Developing and publicizing objective standards and best practices for testing of anti-malware and related products.
  • Promoting education and awareness of issues related to the testing of anti-malware and related products.
  • Providing tools and resources to aid standards-based testing methodologies.

Its own guidelines documents are listed here.

The AMTSO resources page includes both papers and links: . (Some of which are also linked to below, as I can’t guarantee that links on that page will hold.)

  • Virus Bulletin Spotlight article on AMTSO
    An article by David Harley on what AMTSO has achieved so far, and what might lie ahead: ‘AMTSOlutely Fabulous’, January 2010, Virus Bulletin. Would I hold to those views now? I guess that depends on what comes out of the changes currently going on in AMTSO’s executive structure, into which I have absolutely no insight, currently. Now also available on this site.

Apple Testing

While OS X threats have become a significant problem over recent years (though tiny compared to the sheer volume of Windows malware), comparative tests have been rarer than hen’s teeth. AV-Comparatives did do a static test in 2012 which was by no means worthless, though it went against the flow of whole product testing. I blogged about it for Infosecurity Magazine at the time, and I’ll probably pick up the theme again here shortly.


Tests, reviews, reports.


AV-Test.org Archive of Conference Papers


Papers presented at the AVAR (Association of Anti Virus Asia Researchers) conference between 17th and 19th November 2010:


Here’s a whole bunch of presentations from a CARO (Computer Antivirus Research Organization) workshop in Iceland in 2007 that was one of the drivers for the formation of AMTSO: Presentations made at AV Testing Workshop 2007
(wish I could have been there…)


ESET’s white papers page includes a number of relevant articles, white papers and conference papers. Yes, I did write or co-write most of them: writing on behalf of ESET is what I do (some of the time), so of course that’s where a lot of my work gets published. 🙂 Some papers are individually listed below: I’ll do a little more tidying in due course.

WildList Organization

Papers on the WildList organization web site.

Miscellaneous (I’ll get back to these)

This is quite a mixture of old and new, so probably still needs more sorting and checking.

Benchmarking Without Weightings: Like a Burger Without a Bun
Blog by Eugene Kaspersky, September 2011

Anti-malware testing discussions
Blog by Simon Edwards, September 2011

Why there’s no one test to rule them all
Article by Lysa Myers for Virus Bulletin, October 2011

The Holy Grail of AV Testing, and Why It Will Never Be Found
Blog by Eugene Kaspersky, October 2011

Comparing the Comparatives
Igor Muttik: “Comparing the Comparatives”

Testing Methodology for Spyware Removal
Josh Harriman: Proposal for testing anti-spyware removal (Symantec)

Testing Methodology for Rootkit Removal
Josh Harriman: Proposed methodology for testing anti-rootkit removal technologies (Symantec)

A Single Metric for Evaluating Security Products
Igor Muttik: “A Single Metric for Evaluating Security Products” presented at the EICAR 2010 conference in Paris

Bringing Testing into the Cloud
Anthony Arrott, Wei Yan, Geoff Grindrod & Jeffrey Wong, presented at the AVAR2009 conference in Kyoto.

Paradigm Shift – From Static To Realtime, A Progress Report
Matt Garrad, Paul Jones, Lysa Myers and Michael Parsons, presented at the EICAR 2010 Conference in Paris

Let telemetry be your guide, a proposal for security tests
Jimmy Kuo, blog on Technet, 16 July 2009

Update on Telemetry Usage in Tests, Part 1
Jimmy Kuo, blog on Technet, 15 June 2010

Older papers, still worth reading

Pragmatic Anti-virus Testing
Joe Wells: “Pragmatic Anti-Virus Testing”

Real World Reviews: Current State
Sarah Gordon and Richard Ford: “Real World Anti-Virus Product Reviews And Evaluations – The Current State Of Affairs”

A Reader’s Guide to Reviews
Dr. Alan Solomon: “A Reader’s Guide to Reviews” (originally published in “Virus News International”, and credited to Sarah Tanner)

Analysis and Maintenance of a Clean Virus Library
Vesselin Bontchev: “Analysis and Maintenance of a Clean Virus Library”

What is Wild?
Sarah Gordon: “What is Wild?”

Small Blue-Green World/Mac Virus
ESET Senior Research Fellow



  1. Broken link -> Paradigm Shift – From Static To Realtime, A Progress Report.

    • Thanks, but it works for me. Maybe it was a glitch on the AMTSO web site at the time you checked.

  2. http://www.amtso.org/amtso-download-amtso-false-positive-testing-guidelines.html is not available, can you please restore it.?

    • I haven’t been involved with the administration of the AMTSO site for years, and can’t restore that document. However, I did hear that the service provider was recently subjected to a major hacking attack, and that it was taking time to get the issues fixed. I suspect that the failure of that link is connected to those issues.

    • That document is back, by the way, at http://www.amtso.org/documents/ – I assume it has been for some time, but I’ve only just had occasion to look up something on the same page.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: