Posted by: David Harley | June 16, 2010

The AMTSO Wheel of Pain*

Since I actually wrote an article for Virus Bulletin called AMTSOlutely Fabulous (though my tongue was in my cheek when I thought up that title) and am at present a member of AMTSO’s Board of Directors, I guess you wouldn’t expect me to have much sympathy for a blog post that asks “AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con?”

In fact, the post in question is an ambitious and thoughtful article by Kevin Townsend, a prolific writer and journalist who has some experience and knowledge of malware and testing, so I can’t help but take his question seriously. Actually, three questions:

  • Is this the anti-malware industry looking after itself?
  • Is AMTSO even representative of the anti-malware industry?
  • Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)

Well, that might be four questions, but I see where he’s coming from on that third bullet point, and those are connected questions.

So now I’m going to ask (and answer) a couple of rhetorical questions.

Q: Is Kevin’s post worth reading?

A: AMTSOlut – sorry. Absolutely. It’s a serious and well-grounded attempt to come to grips with some of the credibility problems that he believes AMTSO faces, and he has given me, Stuart Taylor of Sophos (also an AMTSO Board member), and Eric Sites of Sunbelt plenty of opportunity to address those issues.

Q: Do I agree with him?

A: Not in all respects, but he’s raised some essential topics for discussion.

Q: What don’t I agree with?

Well, I’d be perfectly happy to see Joe Wells in AMTSO personally, but I’m not sure AMTSO needs him for street cred. Joe is a great guy whose influence on the AV and testing industries has been very significant, but AMTSO is not the WildList Organization, though ICSAlabs is an AMTSO member (and quite a few of us have a long-standing association with WildList and ICSA Labs). 

The whole subject of WildList testing is pretty contentious, largely due to the incorrect assumption that all product certification is completely WildList-based. Leaving that aside for now (though it’s certainly a topic I’ll be back to),  any suggestion that AMTSO is wedded to some form of static testing based on WildCore is completely erroneous: otherwise, we wouldn’t have published guidelines on topics like “in-the-cloud” testing, dynamic testing, and testing of network-based products.

Finally, Kevin suggests that what we do is compromised because our membership is “incestuous” and doesn’t give a voice to end users. He has a point: we should be listening to users (though in our “real” jobs – we aren’t salaried by AMTSO, you know – we obviously are subject to market forces and vox populi, directly or indirectly). The AMTSO Advisory Board is intentionally made up of people who aren’t subject to the same commercial pressures that anti-malware and testing professionals have to accommodate, and that helps to “keep us honest”. This blog is another tentative step towards opening up a channel of communication between AMTSO and the user community, and we do, in fact, have individual members, though most interested parties will not, on an individual basis, want to pay the somewhat hefty subscription. (We’re not a for-profit organization, but you can’t keep an initiative like this afloat on the cheap!)

However, the content we deal with is sometimes sensitive and usually technical, and AMTSO includes some of the most technically knowledgeable people in the testing and anti-malware industries.  While I’m a keen advocate of better engagement with customers (corporate and consumer) and end users (not always the same thing), I also think that it would be counter-productive to give equal voting rights (even if that were procedurally or pragmatically feasible) to everyone, irrespective of their experience and expertise. I’m sorry if that sounds elitist, but I’m too old to have absolute faith in the wisdom of crowds.

David Harley CITP FBCS CISSP
Emphatically not speaking on behalf of AMTSO or ESET

http://www.securitymetrics.org/content/Wiki.jsp?page=Welcome_blogentry_040505_1

Advertisements

Responses

  1. […] posted a lengthy response on the AMTSO blog here: while it was a personal response rather than an official statement on behalf of AMTSO or the Board […]

  2. […] posted a lengthy response on the AMTSO blog here: while it was a personal response rather than an official statement on behalf of AMTSO or the Board […]

  3. […] posted a lengthy response on the AMTSO blog here: while it was a personal response rather than an official statement on behalf of AMTSO or the Board […]

  4. […] to Screw Up Testing By David Harley Since my blog a few days ago, a few people have asked about the “Top Ten Mistakes Made When Evaluating […]

  5. […] it was an AMTSO issue and most of the queries have related to an AMTSO blog post, I've returned to it (and slightly tweaked it) as another AMTSO […]

  6. […] it was an AMTSO issue and most of the queries have related to an AMTSO blog post, I've returned to it (and slightly tweaked it) Category: Nod32 Antivirus & Eset […]

  7. […] approved at Helsinki and published here, the Kevin Townsend blog considered at some length here, and the top ten testing screw-ups […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: