For several weeks now I’ve been meaning to write about Carl Gottlieb’s site TestMyAV. Well, not about that site so much as the pros and cons of companies setting up their own test labs.
It may surprise you to know that I actually started my career in security working for a medical research organization, and part of the job was, in fact, evaluating anti-malware products. Though in those days malware was nearly all viral. But long before I crossed the Great Divide and starting working with security companies rather than just using their products, I was scaling back on actual virus testing, as with the gradual escalation of the problem, I couldn’t give it the attention it required. Nowadays, since a sizeable proportionof my income comes from providing security companies with consultancy, I couldn’t ethically set myself up as an independent tester, even if I could find the time.
TestMyAV worries me (a lot). It suggests that testing is simple enough that anyone can do it with the help of the resources that TestMyAV provides, including some high-level advice and documentation on setting up a lab, but also offering samples. And it seems to me that if newbie testers are reliant on samples from a site that doesn’t disclose its sources, they have at least two problems. They have to assume that the samples are valid, in the absence of a documented validation process. And they don’t know whether the samples are sourced from one of the companies they plan to test, which is a methodological disaster. As Simon Edwards, one of the most scrupulous testers I know, observed on Twitter:
‘Testing anti-malware with malware provided by tested vendors (or related companies) is about as biased as testing can be. Don’t do it!’
Clearly, he’s referring to the fact that Carl Gottlieb is CTO of Cognition, which is a major Cylance reseller.
Well, I still intend to get back to this topic at greater length, though I’m making no promises about when or where. But in the meantime, it seems that Kevin Townsend has been worried about the site, too. In Anti-malware testing issues he lays stress on links between TestMyAV and Cognition. He emphasizes the number of pages there that offer an antivirus product recommendation. He summarizes the ongoing war of words between the mainstream used-to-be-antivirus industry and those companies that call themselves ‘next generation’. And he suggests a less contentious way of testing products.
I don’t agree with every word Kevin says: I think it’s pretty harsh to suggest baldly that independent testers aren’t independent, for instance, even though I’m not the testing industry’s biggest fan. The symbiotic relationship between testers and the mainstream security industry is complex and in some senses problematical, but both industries have – sometimes, at least – fought hard (in AMTSO and elsewhere) to strike the best possible balance in the interests of fair testing and the best outcome for the consumer. Nevertheless, he makes some very important points.
Carl Gottlieb evidently disagrees vehemently, but has said that he won’t ‘address the points publicly’.
David Harley
(Not speaking for any company or organization.)
Anti-Malware Testing 
Leave a Reply