Here are a few testing-related resources, some of which I have (or have had) something to do with, some not. Caveat: this is a work in progress. Obviously, I can’t guarantee that links on sites I don’t maintain will stay current: let me know if you spot any broken links. (Comment here, or for a (normally) quicker response, antimalware[dot]testing[at]gmail[dot]com.
In case I have to say so yet again, this page is maintained independently of AMTSO.
The Anti-Malware Testing Organization has four aims according to its current charter:
- Providing a forum for discussions related to the testing of anti-malware and related products.
- Developing and publicizing objective standards and best practices for testing of anti-malware and related products.
- Promoting education and awareness of issues related to the testing of anti-malware and related products.
- Providing tools and resources to aid standards-based testing methodologies.
Its own guidelines documents are listed here.
- AMTSO License Agreement
If you click through to downloading the papers listed below, you have to signify that you’re in agreement with the agreement. It’s a lawyer thing.
- AMTSO Fundamental Principles of Testing
AMTSO Fundamental Principles of Testing as approved by the AMTSO meeting held in Oxford 31st October 2008.
- AMTSO Best Practices for Dynamic Testing
AMTSO Best Practices for Dynamic Testing as approved by the AMTSO meeting held in Oxford 31st October 2008
- AMTSO Best Practices for Validation of Samples
AMTSO Best Practices for validation of samples as approved by the AMTSO meeting held in Budapest 7th May 2009
- AMTSO Best Practices for Testing In-the-Cloud Security Products
AMTSO Best Practices for Testing In-the-Cloud Security Products as approved by the AMTSO meeting held in Budapest 7th May 2009
- AMTSO Guidelines for testing Network Based Security Products
AMTSO Guidelines for testing Network Based Security Products as approved by the AMTSO meeting held in Prague 13th October 2009
- AMTSO Issues involved in the “creation” of samples for testing
AMTSO Issues involved in the “creation” of samples for testing as approved by the AMTSO meeting held in Prague 13th October 2009
- AMTSO Whole Product Testing Guidelines
AMTSO Whole Product Testing Guidelines as approved by the AMTSO meeting held in Helsinki 25th May 2010
- AMTSO Performance Testing Guidelines
AMTSO Performance Testing Guidelines as approved by the AMTSO meeting held in Helsinki 25th May 2010
- AMTSO False Positive Testing Guidelines
AMTSO False Positive Testing Guidelines as approved by the AMTSO meeting held in Munich 22nd October 2010.
- AMTSO Testability Guidelines
AMTSO Testabiity Guidelines as approved by the AMTSO meeting held in Prague 4th May 2011.
- AMTSO Use and Misuse of Test Files
Guidelines on the proper use of test files as approved by the AMTSO meeting held in San Mateo 24th February 2012.
- AMTSO Sample Selection for Testing
Guidelines on the collection, validation and classification of samples as approved by the AMTSO meeting held in San Mateo 24th February 2012
The AMTSO resources page includes both papers and links: . (Some of which are also linked to below, as I can’t guarantee that links on that page will hold.)
- Virus Bulletin Spotlight article on AMTSO
An article by David Harley on what AMTSO has achieved so far, and what might lie ahead: ‘AMTSOlutely Fabulous’, January 2010, Virus Bulletin. Would I hold to those views now? I guess that depends on what comes out of the changes currently going on in AMTSO’s executive structure, into which I have absolutely no insight, currently. Now also available on this site.
While OS X threats have become a significant problem over recent years (though tiny compared to the sheer volume of Windows malware), comparative tests have been rarer than hen’s teeth. AV-Comparatives did do a static test in 2012 which was by no means worthless, though it went against the flow of whole product testing. I blogged about it for Infosecurity Magazine at the time, and I’ll probably pick up the theme again here shortly.
Tests, reviews, reports.
Papers presented at the AVAR (Association of Anti Virus Asia Researchers) conference between 17th and 19th November 2010:
- The Difference Between False Positives and False Positives in Testing, by Mark Kennedy. See also the AMTSO False Positive Testing Guidelines document .
- Test Files and Product Evaluation: the Case for and against Malware Simulation, by David Harley, Lysa Myers, Eddy Willems. The full paper is now available here.
- Weightwatching: Why prevalence and persistence matter in Anti-malware testing by Andrew Lee, Lysa Myers, Matt Garrad, Michael Parsons
- The Power of US, by Igor Muttik.
Here’s a whole bunch of presentations from a CARO (Computer Antivirus Research Organization) workshop in Iceland in 2007 that was one of the drivers for the formation of AMTSO: Presentations made at AV Testing Workshop 2007
(wish I could have been there…)
ESET’s white papers page includes a number of relevant articles, white papers and conference papers. Yes, I did write or co-write most of them: writing on behalf of ESET is what I do (some of the time), so of course that’s where a lot of my work gets published. 🙂 Some papers are individually listed below: I’ll do a little more tidying in due course.
- Man, Myth, Malware and Multi-Scanning
Paper by David Harley and Julio Canto for CFET 2011 in Canterbury
- Daze of Whine and Roses
Paper by David Harley and Larry Bridwell, presented at the Virus Bulletin Conference 2011 in Barcelona
- Real Performance?
Ján Vrabec and David Harley: “Real Performance?” presented at the EICAR 2010 conference in Paris. (On performance testing rather than detection testing.)
- AMTSO: the Test of Time?
Article by David Harley for Network Security Magazine. Pre-proof copy here.
- After AMTSO: a funny thing happened on the way to the forum
Paper by David Harley presented at EICAR 2012
- Antivirus Testing and AMTSO Has anything changed?
David Harley, presented at the CFET Conference 2010 in Canterbury, UK
Papers on the WildList organization web site.
Miscellaneous (I’ll get back to these)
This is quite a mixture of old and new, so probably still needs more sorting and checking.
Benchmarking Without Weightings: Like a Burger Without a Bun
Blog by Eugene Kaspersky, September 2011
Anti-malware testing discussions
Blog by Simon Edwards, September 2011
Why there’s no one test to rule them all
Article by Lysa Myers for Virus Bulletin, October 2011
The Holy Grail of AV Testing, and Why It Will Never Be Found
Blog by Eugene Kaspersky, October 2011
Comparing the Comparatives
Igor Muttik: “Comparing the Comparatives”
Testing Methodology for Spyware Removal
Josh Harriman: Proposal for testing anti-spyware removal (Symantec)
Testing Methodology for Rootkit Removal
Josh Harriman: Proposed methodology for testing anti-rootkit removal technologies (Symantec)
A Single Metric for Evaluating Security Products
Igor Muttik: “A Single Metric for Evaluating Security Products” presented at the EICAR 2010 conference in Paris
Bringing Testing into the Cloud
Anthony Arrott, Wei Yan, Geoff Grindrod & Jeffrey Wong, presented at the AVAR2009 conference in Kyoto.
Paradigm Shift – From Static To Realtime, A Progress Report
Matt Garrad, Paul Jones, Lysa Myers and Michael Parsons, presented at the EICAR 2010 Conference in Paris
Let telemetry be your guide, a proposal for security tests
Jimmy Kuo, blog on Technet, 16 July 2009
Update on Telemetry Usage in Tests, Part 1
Jimmy Kuo, blog on Technet, 15 June 2010
Older papers, still worth reading
Pragmatic Anti-virus Testing
Joe Wells: “Pragmatic Anti-Virus Testing”
Real World Reviews: Current State
Sarah Gordon and Richard Ford: “Real World Anti-Virus Product Reviews And Evaluations – The Current State Of Affairs”
A Reader’s Guide to Reviews
Dr. Alan Solomon: “A Reader’s Guide to Reviews” (originally published in “Virus News International”, and credited to Sarah Tanner)
Analysis and Maintenance of a Clean Virus Library
Vesselin Bontchev: “Analysis and Maintenance of a Clean Virus Library”
What is Wild?
Sarah Gordon: “What is Wild?”
David Harley CITP FBCS CISSP
Small Blue-Green World/Mac Virus
ESET Senior Research Fellow