Myths, Marketing, and Testing: the Antimalware Generation Gap

ESET has just published an article that was originally my contribution to this report for ESET: TRENDS 2017: SECURITY HELD RANSOM. The new(-ish) article is about so-called next-gen’s uneasy relationship with more established players in the anti-malware industry, rather than focusing exclusively on testing, but does nevertheless look at testing issues such as ‘next-gen’ companies’ reluctance to participate in testing, yet pushing ‘an already open door even wider by their own attempts to compare the effectiveness of their own products and those of ‘first-gen’ vendors.’ The blog article is here: Next-gen security software: Myths and marketing.

What it doesn’t really address – well, it was written a while ago – is the recent resurgence of the bad old idea that testing is so easy that anyone can do it, with guidance and off-the-peg samples from (next-gen) vendors or their associates. Remind me to tell you about Grottyscan and Wonderscan sometime, from an article on how to bias testing.

Recently, next-gen companies have stepped up their war with some testers. Ironically, treading much the same ground that longer-established vendors have stumbled on in the past. But I intend to come back to that.

Meanwhile, here are some relevant links.

Crowdstrike versus NSS:

• CrowdStrike Initiates Legal Action Against NSS Labs For Misappropriation of Intellectual Property and Engaging in a Sham Transaction to Illegally Obtain Access To Our Falcon Software
• NSS’s riposte: AEP Public Test Announcement
• Some harsh but entertaining commentary from Vesselin Bontchev: Some thoughts on the CrowdStrike vs NSS Labs debacle

Steve Ragan on Cylance versus AV-Comparatives and MRG Effritas: Cylance accuses AV-Comparatives and MRG Effitas of fraud and software piracy – Is it time for a new testing and certification model in the industry? Interestingly, Cylance seems to have backtracked somewhat on its opposition to Pay-to-Play, and has commissioned a test from AV-Test using methodology Cylance ‘co-created’.

I got a certain amount of sour amusement from Cylance’s own assertion that this is ‘The first time that a 3rd party testing organization created their own malware in order to conduct testing.’ If only that were true… And if only it were a good idea. Apparently it’s also the first time that a tester ‘developed new testing methods specifically designed to target next-generation vendors.’ I wonder if SE Labs has any thoughts about that?As far as I can tell from Ragan’s article the ‘new’ methodology consists of the following:

• The first test, the ‘Holiday Test’ seems to be an old-school ‘frozen update test’.
• The second test used ‘malware’ created by AV-Test ‘to simulate certain types of attack’.
• The third test involves disabling URL filtering.
• The fourth is a small-scale FP test.

AMTSO’s latest press release doesn’t mention specific vendors or tests, but does observe with regard to ‘recent tests’ that

We reject turning off product capabilities while comparing the capabilities of products in real-world use, as we believe that this introduces bias in the results.

Hard to argue against that, though such cherrypicking of functionality has long bedevilled testing: that has a lot to do with AMTSO’s emphasis on ‘whole product testing’. The press release doesn’t mention malware creation or the use of simulated malware, though these are other areas that have been discussed at some length at AMTSO meetings and in documentation (and many other places).

These are interesting times for testers and vendors. And indeed for AMTSO, which includes among its members nearly all these players. Not Cylance, though, but Cylance reseller Cognition is a member, and also seems to have a close relationship with TestMyAV, which advocates DIY testing rather than using independent testers, or as they put it, trusting ‘the experts’.

I’m glad I’m not caught up in that particular cat-herding exercise any more.

On the other hand, I think I may feel a paper coming on.

David Harley