Posted by: David Harley | December 3, 2010

Guidelines for False Positive Testing

AMTSO has announced its guidelines for false positive testing.

False positives occur when a security product incorrectly flags a file or resource as malicious. Although false positives rarely have significant impact, the consequences of a severe incident can be more damaging than failing to detect a malicious file: for instance when access to one or more system-critical files is lost, as dramatically indicated by recent high-profile FP incidents.

The new guidelines suggest a series of criteria for testers to use in determining the magnitude of a false positive. Criticality looks at the impact of a false positive on the user. It categorizes the severity depending on the function of the affected resource within a system, network or application and assesses how critical it is to normal operation. Prevalence considers how many users would be impacted by a false positive – is it five or five thousand? Recoverability assesses how difficult it is to remediate the situation: has data been deleted and does the system need to be taken offline?

“False positives tend to have a greater visible impact on the customer than on a security product’s protection, so it’s surprising that not more anti-malware tests include false positives,” says Mark Kennedy of Symantec, who introduced the guidelines on behalf of AMTSO in papers for the Virus Bulletin and AVAR Conferences. “In recent times, the introduction of proactive technologies such as behavior blocking and generic signatures have dramatically increased the likelihood of false positives. The problem with current tests is that they are frequently too simplistic in their approach, presuming that all non-malicious files are equally important. However, when you break down a file’s specific function it’s clear that it this is simply not the case.”

Just as in its guidelines for testing detection rates of malicious files, AMTSO stresses that care must be taken to ensure that all samples to be tested are verified, that they are not misclassified and that the vendor has not added detection intentionally because it regards the file as “greyware”, “possibly unwanted” and so on. AMTSO also recommends that testers make it clear when FP testing is performed in conjunction with malware detection testing, as this may bias the results.

The Guidelines for false positive testing are available for free download here:


  1. […] At the last AMTSO workshop in Munich, a guidelines document on False Positive (FP) testing was approved, and is now available on the AMTSO documents page. More information on the AMTSO blog here: Guidelines for False Positive Testing. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.


%d bloggers like this: