Feature Checking: from EICAR to AMTSO

Posted by: David Harley | June 5, 2013

Feature Checking: from EICAR to AMTSO

AMTSO, the Anti-Malware Testing Standards Organization, yesterday announced its new page for checking AV feature settings.

smoke detector

The new tools are a logical extension of the principle behind what is usually referred to as the EICAR test file. And in fact, as Neil Rubenking pointed out for PC Magazine SecurityWatch, two of them make direct use of the EICAR file.

I don’t really care to see that file referred to as a test file, because it may encourage people to use it inappropriately in formal testing contexts. For example, by changing the EICAR file to see if anti-malware products still detect it. That’s inappropriate because the EICAR file isn’t intrinsically malicious. If you change it so that it no longer meets the specification, there’s no longer any reason why it should be detected by security software. Eddy Willems, Lysa Myers and I wrote (for the AVAR conference) an extensive paper about that and some related issues around testing with simulations, and an AMTSO guidelines document along very similar lines was compiled subsequently: AMTSO Use and Misuse of Test Files.

What the EICAR file does do is give you a limited check – or test – as to whether an anti-virus scanner is installed and working. It’s a ‘test’ in the same sense that pushing the button on a smoke alarm is a test: it tells you that the battery isn’t dead and that it’s capable of making a noise to wake the dead, but it doesn’t tell you whether a specific fire/smoke event will set it off.

The AMTSO feature settings checks extend that functionality:

Clearly, the first two of them work by using the EICAR file. Which is fine, that’s how it was intended to be used. Since all mainstream AV products recognize the file and treat it more or less as if it were a genuinely malicious file, if it’s allowed to download and/or execute, it may indicate a problem with the installation of the software. Not necessarily, though.

Not all security products recognize the EICAR file, and those that do don’t all treat it in exactly the same way. And because these are extensions to the basic EICAR functionality, even products that do recognize EICAR may not react as expected in the context of the AMTSO feature check pages, though I suspect that any mainstream product will react appropriately to tests one or two, even if the vendor hasn’t specifically signed up for the feature check pages.

Number 5 uses the CloudCar test, which has been around for quite a while – it’s discussed in the AMTSO Use and Misuse of Test Files document. CloudCar is specific to the vendor so the feature check is only going to work for products where the vendor has signed up for it and generated the file/hash.

The other two pages are similar in concept: they’re not genuinely malicious but a vendor can agree to detect them as if they were – more or less, at any rate. (In fact, AV detection of EICAR isn’t necessarily exactly the same as detection of real malware is implemented.) If they don’t, it won’t work as a check that PUA or phishing detection is enabled, but a non-compliant product might have those functionalities enabled nonetheless. There’s no absolute requirement for any product to comply with any of these measures. If a product is in use that is compliant and the feature check doesn’t work as expected, then some users (or, more probably, system administrators) will find that information useful. But if the feature check works, all it tells you is that the product is compliant and the feature you’re checking is enabled. It doesn’t, of course, tell you anything about how effective the product is at detecting or blocking real attacks, and it does worry me that people might put more faith in these checks than is really appropriate.

Kevin Townsend asked me “What has configuration got to do with AV testing (mission creep?)”. Well, configuration has quite a lot to do with testing: you can invalidate a test completely by getting the configuration wrong. But I think I know what he means.

The only uses in formal testing that I can see for these pages (and the original EICAR file) are (1) for testing that products are actually compliant with the feature tests – a very limited but nevertheless legitimate test objective (2) as an additional check that the features being used in a test are actually activated. It’s a legitimate topic for AMTSO in any case, though, simply because of the possibility of misuse in real tests in the ways that the EICAR file has already been misused in the past.

It’s also been suggested that the fact that a number of vendors are supporting these initiatives indicates that AMTSO has evolved into a joint marketing tool for the AV industry. I don’t think that’s altogether fair comment. There’s no reason for a tester to sign up to this because it has no place (beyond the previous bullet) in formal testing. I don’t see that it has much use as a marketing tool anyway, except in so far as it brings AMTSO back into the public eye. It has to be jointly supported because its usefulness to the end user is even more limited if all – or at least most – vendors don’t sign up to it.

David Harley
Not speaking for anyone but himself.

Posted in AMTSO Guidelines Documents, Anti-Malware Testing Standards Organization, David Harley, EICAR, Kevin Townsend, Neil Rubenking, Testing Papers

Anti-Malware Testing