…is the title of a presentation AVG‘s Larry Bridwell and I did at Virus Bulletin recently. The paper is now up (by kind permission of Virus Bulletin) on the ESET white papers page as Daze of Whine and Neuroses.
Daze of whine and neuroses (but testing is FINE)
According to Aerosmith (not to mention The Italian Job), FINE is an acronym for (in its politer version) Freaked out, Insecure, Neurotic and Emotional. We could (and probably will) offer alternatives, but there’s no doubting that anti-malware testing inspires all those reactions.
Sometimes it seems that AMTSO has become a dumping ground for the rest of the world’s misgivings about the AV industry, even though it originated in a coalition with some of the testers who are monitoring that industry’s performance with the most assiduous professionalism: indeed, that coalition has in itself inspired mistrust. And recently, it has become plain that even within AMTSO both testers and vendors sometimes find the alliance problematical.
AMTSO’s purpose is simple to state, but much harder to achieve. It represents a realization by professional testers and security vendors that the quality of anti-malware testing was so variable that it was at best confusing for people who need guidance on how to select the best product for their needs. Perhaps testing has improved more in the past few years than it would have without AMTSO’s presence, and discussions and generation of material in a single forum has accelerated a much-needed move away from static testing towards dynamic testing.
But it’s time to ask (and attempt to answer) a number of tough but critical questions.
- Looking over the historical evolution of testing before and since AMTSO, is that move towards dynamic testing enough to set the testing world to rights?
- Are the aims of testers and vendors close enough to allow continued cooperation within AMTSO?
- Has AMTSO already outlived its usefulness?
- If not, what should it do next?
- What is the future of comparative detection testing?
David Harley CITP FBCS CISSP
ESET Senior Research Fellow