Posted by: David Harley | January 15, 2017

Testing the Internet of Things

The security community has long been concerned about the potential for compromise of the so-called Internet of Things (IoT). Recently, it has become commonplace to add Internet connectivity to objects that in the past functioned quite happily without connectivity. It seems that there are plenty of people who see advantages to being able to control all sorts of things from light bulbs to televisions to heating to kettles, though I sometimes wonder whether in some instances they’re mostly manufacturers rather than consumers, who may not be desperate to control everything in the house through a smart app. However, it’s been apparent time and time again that manufacturers in this market segment are not always giving security the attention it requires.

Well, I won’t bore you all (or both…) with another Luddite rant, in case I start to sound too much like a Jeffrey Deaver killer. However, I applaud AV-Test‘s initiative in setting up a site to record their testing of IoT device security. Right now the articles there seem to be focused on IP cameras, an area that AV-Test has also addressed more generally on its mothership site. It’s not an area I’m sufficiently conversant with to comment on AV-Test’s reviews, but given the organization’s recent experience in this field and its reputation in the field of anti-malware testing, I imagine they’ll be up to the usual high standards.

David Harley


Posted by: David Harley | December 10, 2016

Babel fish all round, please

I referred earlier to an AV-Comparatives ‘next-gen’ test, and intended to follow up by pointing to this article by Nikita Shvetsov for Kaspersky: Lost in Translation, or the Peculiarities of Cybersecurity Tests.

Nice. So-called next-gen vendors may be less keen. 🙂

David Harley

Posted by: David Harley | December 10, 2016

When DIY testing isn’t DIY

For several weeks now I’ve been meaning to write about Carl Gottlieb’s site TestMyAV. Well, not about that site so much as the pros and cons of companies setting up their own test labs.

It may surprise you to know that I actually started my career in security working for a medical research organization, and part of the job was, in fact, evaluating anti-malware products. Though in those days malware was nearly all viral. But long before I crossed the Great Divide and starting working with security companies rather than just using their products, I was scaling back on actual virus testing, as with the gradual escalation of the problem, I couldn’t give it the attention it required. Nowadays, since a sizeable proportionof my income comes from providing security companies with consultancy, I couldn’t ethically set myself up as an independent tester, even if I could find the time.

TestMyAV worries me (a lot). It suggests that testing is simple enough that anyone can do it with the help of the resources that TestMyAV provides, including some high-level advice and documentation on setting up a lab, but also offering samples. And it seems to me that if newbie testers are reliant on samples from a site that doesn’t disclose its sources, they have at least two problems. They have to assume that the samples are valid, in the absence of a documented validation process. And they don’t know whether the samples are sourced from one of the companies they plan to test, which is a methodological disaster. As Simon Edwards, one of the most scrupulous testers I know, observed on Twitter:

‘Testing anti-malware with malware provided by tested vendors (or related companies) is about as biased as testing can be. Don’t do it!’

Clearly, he’s referring to the fact that Carl Gottlieb is CTO of Cognition, which is a major Cylance reseller.

Well, I still intend to get back to this topic at greater length, though I’m making no promises about when or where. But in the meantime, it seems that Kevin Townsend has been worried about the site, too. In Anti-malware testing issues he lays stress on links between TestMyAV and Cognition. He emphasizes the number of pages there that offer an antivirus product recommendation. He summarizes the ongoing war of words between the mainstream used-to-be-antivirus industry and those companies that call themselves ‘next generation’. And he suggests a less contentious way of testing products.

I don’t agree with every word Kevin says: I think it’s pretty harsh to suggest baldly that independent testers aren’t independent, for instance, even though I’m not the testing industry’s biggest fan. The symbiotic relationship between testers and the mainstream security industry is complex and in some senses problematical, but both industries have – sometimes, at least – fought hard (in AMTSO and elsewhere) to strike the best possible balance in the interests of fair testing and the best outcome for the consumer. Nevertheless, he makes some very important points.

Carl Gottlieb evidently disagrees vehemently, but has said that he won’t ‘address the points publicly’.

David Harley

(Not speaking for any company or organization.)

Posted by: David Harley | December 3, 2016

AV-Test Report on Risk Scenarios

Long-established research/product testing organization AV-Test has published an interesting  document giving some background to the current malware scene, including consideration of threats on Windows, Mac, Android/mobile, Internet, PUA, and test statistics. Current Risk Scenario: AV-TEST Security Report Facts at a Glance

David Harley

Posted by: David Harley | November 7, 2016

AV-Comparatives ‘next-gen’ test

Independent testing of so-called ‘next-gen’ products is currently quite unusual: indeed, some next-gen vendors have suggested that their products cannot be tested by independent testers. Though apparently it’s OK to do your own tests with the methodologies and samples provided by a website affiliated with the reseller of  a next-gen product. The dangers of that approach are fairly obvious, but I’ll certainly be back to that topic in due course.

Independent tester AV-Comparatives, however, has gone its own way – as with its long experience of product testing, it’s certainly entitled to do – and tested four next-gen products:

  • Barracuda NextGen Firewall VF100 7.0.1
  • CrowdStrike Falcon Host
  • Palo Alto Traps
  • Sentinel One Endpoint Protection Platform

The overall reviews and the Malware Protection Tests were performed by AV-Comparatives themselves, while the Exploit Test was performed by MRG Effitas.

This review, and others, are available from the AV-Comparatives Business Reviews page.

David Harley

Older Posts »