Testing Resources

[Updated 22nd June 2018]

Here are a few testing-related resources, some of which I have (or have had) something to do with, some not. Caveat: this is a work in progress. Obviously, I can’t guarantee that links on sites I don’t maintain will stay current: let me know if you spot any broken links (I’ve just removed rather a lot that had been fractured since I first posted this page). I may gather together a few more resources in due course in preparation for a publishing project.

This page is maintained independently of AMTSO. However, while I’m not AMTSO’s biggest fan, I’m generally in agreement with its aims, if not always its methodology. And in general, its activities have helped to raise the standard of testing in general to a much higher level than it would have achieved if AMTSO had never existed. So you’ll find quite a few references to AMTSO resources below. (Unfortunately, several of those resources have disappeared or moved: hence the current spring-cleaning.)

Testers

I’m not the biggest fan of the testing industry, either. Some testers, even if their intentions are good, do more harm than good. There are well-intentioned testers, competent testers, testers with hidden agendas that are not necessarily in the best interests of the community at large (being in some instances intended to promote a single vendor), and testers who – however good their intentions – know as much about testing as I do about quantum mechanics. In the words of Vesselin Bontchev – a man who knows more than most about testing – ‘…all anti-virus testing outfits generally fall into two categories – incompetent and incomplete. (Of course, some are both.)’ Harsh, but not unfair.

Still, good testers do their best to fulfil a vital function. Evaluating security software, whether for your own use or for corporate use, is hard. (Trust me on that – it’s pretty much how I got into the security industry in the first place, evaluating and implementing security technology for a medical research organization.) Honest testers who try to provide accurate guidance to their readers perform an essential service. Imagine an anarchic world where there was nowhere to go for an impartial and reasonably well-informed opinion on how products perform in the real world, and you had to check it all out for yourself. (Well, some vendors say that they would like you to do exactly that, but what they really mean is that they’d like you to get your testing tools and samples from sites that favour their products…)

I’ve been writing about this for decades now, but this paper* – “The (Testing) World Turned Upside Down” – is probably my last word on the subject – on the conference circuit at any rate – and I’d recommend that you read it if you’re interested in why I’m cautious about making active recommendations of testing sites. (Let alone of individual products: since I currently provide consultancy to a security company, it would be pretty flaky if I was to offer product recommendations.)

After that long preamble, here’s a (rather short) list of the test sites that I consider reasonably reliable:

  • Virus Bulletin: VB Testing – “Virus Bulletin’s testing and certification services” Offers testing reports for malware detection, spam detection, and web filtering, and doesn’t make spurious claims to be the source of all testing wisdom.
  • SE Labs – “Intelligence-led testing” Simon Edwards’ scrupulous descriptions of the lab’s testing methodology are an excellent example of what the testing industry in general should be aiming for in terms of transparency and disclosure. The SE Labs blog is well worth following, too.
  • AV-Comparatives – “Independent Tests of Anti-Virus Software” Tests and other commentary on a wide range of platforms. I don’t always agree with them, but they’re good guys and do their best to get their testing right.
  • AV-Test – covers a wide range of tests and commentary: their current work on IoT (“Internet of Things”) testing is rather interesting. I’m not always convinced by their methodology, but they do have a lot of experience.

All these organizations have a long history of engagement with AMTSO – we’ll get onto that shortly – though that hasn’t always been a peaceful, uncontroversial relationship. There is also a list of AMTSO members   (vendors and testers) here.  You may find this useful in a number of respects, but bear in mind that membership of AMTSO is not in itself a guarantee of probity and transparency.

You may notice that there are no consumer magazine testers represented here. (Virus Bulletin was a paper magazine at one time, but it’s always been more than that, notably the organizer of a very important security conference, as well as a specialist testing lab.) Most consumer magazines don’t have the expertise or resources to do competent product testing: some outsource the actual testing to professional independent sites like the ones listed above, but even then, misunderstandings and distortions of test results can creep in. There are no multi-scanner sites like VirusTotal, either, because they’re not suitable for comparative product testing, though they’ve been misused for that purpose many times. In fact, VirusTotal’s Julio Canto and I wrote about that issue some years ago: Man, Myth, Malware and Multi-Scanning.

*You can also get a PDF version here.

AMTSO

The Anti-Malware Testing Organization has four aims according to its current charter:

  • Providing a forum for discussions related to the testing of anti-malware and related products.
  • Developing and publicizing objective standards and best practices for testing of anti-malware and related products.
  • Promoting education and awareness of issues related to the testing of anti-malware and related products.
  • Providing tools and resources to aid standards-based testing methodologies.

Its own guidelines documents are listed here and currently include:

  • AMTSO Fundamental Principles of Testing
  • AMTSO Best Practices for Dynamic Testing
  • AMTSO Suggested Methods for the Validation of Samples
  • AMTSO Best Practices for Testing In-the-Cloud Security Products
  • AMTSO Guidelines for Testing Network-Based Security Products
  • AMTSO Issues Involved in Creation of Samples
  • AMTSO Whole Product Testing Guidelines
  • AMTSO Guidelines to False Positive Testing
  • AMTSO Guidelines on Facilitating Testability
  • AMTSO Use and Misuse of Test Files in Anti-Malware Testing
  • AMTSO Sample Selection for Testing
  • AMTSO Performance Testing Guidelines
  • AMTSO Guidelines on Mobile Testing
  • AMTSO Guidelines for Testing Protection Against Targeted Attacks
  • AMTSO Testing Protocol Standard for the Testing of Anti-Malware Solutions v1.0

If you’re new to this, you should at least read the AMTSO Fundamental Principles of Testing. In some respects it’s the nearest AMTSO has come to date to meeting its object of ‘promoting education and awareness’.

The AMTSO resources page used to include both papers and links from outside the organization. Unfortunately, it seems to have disappeared in recent years.

Here’s an article for Virus Bulletin from 2010 on what AMTSO had achieved so far, and what might lie ahead: ‘AMTSOlutely Fabulous’, January 2010, Virus Bulletin. Would I hold to those views now?  Now also available on this site.

Apple Testing

While OS X threats have become a significant problem over recent years (though tiny compared to the sheer volume of Windows malware), comparative tests have been rarer than hen’s teeth. AV-Comparatives did do a static test in 2012 which was by no means worthless, though it went against the flow of whole product testing. I blogged about it for Infosecurity Magazine at the time, and kind of picked up the theme later in a paper co-written and co-presented with Lysa Myers: Mac Hacking: the way to better testing? (for the Virus Bulletin conference in October 2013.)

CARO

Here’s a whole bunch of presentations from a CARO (Computer Antivirus Research Organization) workshop in Iceland in 2007 that was one of the drivers for the formation of AMTSO: Presentations made at AV Testing Workshop 2007 (wish I could have been there…)

ESET

These are most of the relevant conference papers I’ve written/co-written and presented on behalf of ESET.

WildList Organization

Papers on the WildList organization web site. Although the web site is mostly dormant, this page is still there., but some of the links are now broken. These still seem to be active:

Miscellaneous 

This is quite a mixture of old and new, but these papers and articles are still available at time of writing.

Benchmarking Without Weightings: Like a Burger Without a Bun
Blog by Eugene Kaspersky, September 2011

Anti-malware testing discussions
Blog by Simon Edwards, September 2011

Why there’s no one test to rule them all
Article by Lysa Myers for Virus Bulletin, October 2011

The Holy Grail of AV Testing, and Why It Will Never Be Found
Blog by Eugene Kaspersky, October 2011

Real World Reviews: Current State
Sarah Gordon and Richard Ford: “Real World Anti-Virus Product Reviews And Evaluations – The Current State Of Affairs”

A Reader’s Guide to Reviews
Dr. Alan Solomon: “A Reader’s Guide to Reviews” (originally published in “Virus News International”, and credited to Sarah Tanner)

David Harley

Advertisements

Responses

  1. Broken link -> Paradigm Shift – From Static To Realtime, A Progress Report.

    • Thanks, but it works for me. Maybe it was a glitch on the AMTSO web site at the time you checked.

  2. http://www.amtso.org/amtso-download-amtso-false-positive-testing-guidelines.html is not available, can you please restore it.?

    • I haven’t been involved with the administration of the AMTSO site for years, and can’t restore that document. However, I did hear that the service provider was recently subjected to a major hacking attack, and that it was taking time to get the issues fixed. I suspect that the failure of that link is connected to those issues.

    • That document is back, by the way, at http://www.amtso.org/documents/ – I assume it has been for some time, but I’ve only just had occasion to look up something on the same page.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: