Which?-Hunting

[Update: hat tip to Charles Schloss for the correction in the paragraph below.]

My attention was caught by a post at Security-FAQs: Which? Claims Windows 8 Is More Secure Than Third Party Antivirus. Lee Munson rightly points out that this seems a little bizarre given the recent fuss about Microsoft Security Essentials’ recent failure to achieve AV-Test certification, though obviously there’s a lot more to Windows 8 security than MSE (hat tip to Charles Schloss for pointing out that the anti-virus in Windows 8 is branded as a component of Windows Defender, not as MSE – to the best of my knowledge, it’s the same engine though).

I’m not here to knock MSE, and in any case it won’t come as a surprise that I’m pretty cautious about recommending tests at the best of times, but if I had to pick one test over the other, I’d give more credence to the test by a tester with a long track record in specialized security testing than to a test by non-specialists like Which? or Consumer Reports, especially those that charge for access to reviews that – even on payment – tend not to discuss their methodology in sufficient depth to assess its accuracy. (Which? has a very similar “hands-off approach” to Consumer Reports when it comes to testing: “…our product recommendations are influenced only by our test results. We don’t take advertising and we buy all the products that we test ourselves…” Not necessarily problematical in itself, but does a refusal to discuss results with the companies under test (or even with AMTSO, which does after all include testers as well as vendors) really guarantee good practice? Mainstream testers normally have some sort of appeal process to weed out false positives and such.

Here’s  the Which? methodology, insofar as it’s discussed publicly.

• We expose a collection of new and old viruses and malicious files to each security software package we test, and observe how the software copes with them so you can pick the one that will keep you safest.
• Your computer won’t be safe if the package you are using is confusing and potentially a compromise to security as a result. We perform a raft of ease of use tests to weed out the ones not worth bothering with.
• We check how the software interacts with you, so you are not baffled with jargon and other confusing messages.

The 2nd and 3rd bullets seem worthy enough aims to me, though I’d like to know more about how their scoring system actually works. In fact, the three bullets together sound not unlike whole product testing, which is a Good Thing in principle. And 2 and 3 are aspects of security software that a non-specialist has some chance of testing reasonably – perhaps better than some security companies’ internal testing – though there is no shortage of potential pitfalls. The first bullet also sounds reasonable, but where does that collection come from? I actually have a problem generally with the assumption that testers are better than AV companies at collecting and verifying samples. At least mainstream testers have (apart from their own honeynets and so on) reciprocal arrangements with security companies that allow them to accumulate high volumes of samples, and procedures for verification. It’s unlikely that a non-specialist tester will have those resources. Consumer Reports has, in the past, used a third-party security company to conduct tests on its behalf, but in its eagerness to avoid being tainted by contact with the AV industry, it’s used companies working outside their own specialism.

I haven’t seen the individual product reviews, but the article by Jessica Moreton does suggest some possible problems. Indeed, the very title – Why pay for antivirus? Windows 8 tops security software tests – demonstrates the same fondness for free AV that Consumer Reports has also shown in the past. Understandably: if you consider the best deal you can get your readership is Free, why would you recommend that they pay?

But is it really the best deal? In the Consumer Reports test I have in mind, they did at least recommend some other types of free security software that a parsimonious reader could combine in order to get a functional approximation to a security suite. In fact, while I don’t think price (let alone the absence thereof) is the best primary determining factor for selecting security software, you might feasibly do as well with a suitable combination of free products as with a commercial single-vendor suite. But only if you know exactly what you’re doing – in terms of configuration, as well as product selection – in which case you don’t need to rely on a consumer magazine review.

Which? has come up with a bizarre apples and oranges testing model: its choice of products to test includes:

• one operating system (Windows 8, apparently the top performer)
• one free AV/firewall combo (Zone Alarm)
• two free AV products (MSE for Windows 7 and Avira)
• six Internet security suites (Zone Alarm, Bullguard, F-Secure, Norton, Kaspersky, and Avira)

Right now, Windows 8 has a lot going for it from a security point of view. (Rather less if you don’t want a tablet-oriented operating system, like me and many business users, but we’re not the Which? target audience.) But those of us who remember the Vista security hype will be more sceptical about its long-term efficacy, especially given that the equivalent to its AV component performs fairly weakly in more established tests.

There isn’t much evidence of deep research here, either. AV products that don’t scan .ZIP files? Not many of those around… AV manufacturers may start looking at tablets and smartphones? I think most of them have been exploring that niche for quite a while, and Android users – for example – should be all too aware that the malware risk is far from ‘potential. This isn’t a minor consideration: readers of for-fee review services are entitled to expect accuracy in those reviews, though I hope they don’t assume it…

For all that, though, Moreton asks one very pertinent question: “As for PCs – would you trust security software built into your computer’s operating system?” Well, for many people the answer is clearly yes (Apple users for instance). The question is, should they?

Maybe one major justification for malware-oriented security software, patchy though its detection performance may sometimes be, is that it provides OS vendors and the vendors behind competing technologies with an incentive to try to keep ahead of malware (and AV). If we (the AV industry) all packed up tomorrow, and our researchers all retired or went to social media startups, I wonder what the impact would be on the overall detection and/or blocking of malware, long term. (Let’s not forget that those free products are effectively subsidized by commercial products, though they constitute a loss leader that may help to sell commercial products.) Call me cynical, but I suspect that you’d come to miss us…. (That’s actually a topic I touch on in an article due to appear in Virus Bulletin in February, by the way.)

David Harley CITP FBCS CISSP
Small Blue-Green World/Mac Virus/Anti-Malware Testing
ESET Senior Research Fellow