Standards and Relevance

[Update: two mysteriously broken links repaired. Thanks, Kurt, for pointing one of them out! (You wouldn’t have spotted the other one because the link even more mysteriously disappeared altogether.)]

I was directed towards an interesting post (thanks, Julio!) by Danny Quist at Offensive Computing, commenting on the “Issues involved in the “creation” of samples for testing” document to be found on the AMTSO documents page at http://www.amtso.org/documents.html. The link in the Offensive Computing post doesn’t actually go anywhere, but the correct link for the paper is this.

He suggests that writing about testing is “a great way to draw the collective ire of the AV industry.” It may be for him, but I tend to find that for me, it’s a great way to draw the collective ire of people outside the AV industry. 😉 Still, he makes some more serious points that need addressing, so I’ll have a go, though I’m speaking for myself here, not for AMTSO.

As he also suggests, the AV industry is not keen on the public dissemination of samples. Indeed, nor is the mainstream testing industry. Oddly enough, I don’t agree that this is in order to discourage people from testing AV software before they buy it. In fact, there are quite a few issues that are relevant here, but the principle one is safety: putting live malware into the hands of the average user may not be quite as dangerous as putting fireworks into the hands of children, but it does have dangers. Some are obvious, such as accidental infection of the user’s system, or accidental release or triggering of a payload, so that other people’s systems are put at risk.

Some are less obvious. The ethical issue (which may not be important or comprehensible to people outside the industry, but is certainly important to us) around giving unrestricted access to samples to people outside our web of trust isn’t just about people whose honesty or good intentions are in doubt, or whose competence to handle samples safety is unproven.

Suppose I give you some samples to test products with. How well will you evaluate them? There are many ways of introducing a bias that makes one product look good, and many of them have been used… But let’s say that I give you a balanced set of sound samples which any product “should” detect. How easy do you think it is to test antivirus fairly and accurately? If your immediate response is “how difficult can it be?” I promise you that you have a lot to learn.

But this debate is supposed to be about creating samples, not about distributing them. Mr Quist thinks that AMTSO has missed something with respect to the techniques for creating samples referred to in the document.

He says that “Malware authors are using every single one of these techniques with spectacular success.” Of course they are. Which is why making more malware is, at the very least, redundant. Furthermore, if you’re attempting to create your own malware because you can’t get samples from other sources, the chances are that you don’t have the knowledge to create samples that represent real-world threats. A malware author’s lack of ethical principles is beside the point. A good tester does have ethical principles, and feels that he owes it to audience to be as accurate as possible in his testing. And he also knows that artificial samples are not “the real thing”…

David Harley