AMTSO, not ISO

There is so much misinformation around concerning AMTSO at this moment, I don’t know which attack to respond to first, let alone work out how to fit my responses round my real work, but here’s a point or two I can respond to fairly quickly.

What are the good things AMTSO “claims” to do? Well, I’m not going to get into “good” and “bad” at this point, but there’s a summary of its concrete “deliverables” here at Securiteam.

Mr. Townsend asks in his blog:

Would you be happy with a government that said to you, you don’t have the intelligence or knowledge to understand things; so we, the government, the police and the judiciary, are going to tell you how things are and how things are to be done and you don’t have any say in it?

Well, no, not personally. But that’s not a fair analogy to AMTSO’s position.

Firstly, AMTSO passes no laws, or, more to the point, doesn’t set standards in a formal sense like BSI or ISO (and if you say that in that case, the use of the the word “standards” in its name is misleading, I might even agree with you). AMTSO does not say who is or isn’t allowed to test, and does not prescribe testing methodologies. And while it’s been suggested that AMTSO aspires to be the authority on “all things AV”, that particular paranoid fantasy is not one that I’ve been invited to buy into, so I assume that it’s not on the Board’s agenda right now.

What AMTSO does do is provide guidance at varying levels of technical sophistication, put together and  approved by people with a great deal of expertise in aspects of testing. Kevin Townsend seems to be insisting that anyone’s opinion is just as valuable in framing such guidelines, or even that the “wisdom of crowds” trumps technical skill. I’m afraid I’m more sympathetic to Kurt Wismer’s comment that “truth, the goal of good science, is not a democracy.” (And, indeed, his position generally.)

But I’m not sure where this particular debate has come from. While AMTSO members do largely consist of security vendors and organizations, testers, and publishers, there are individual members (and several more are expressing interest at present), and the organization has an advisory board whose members are not part of the security industry. As Kevin Townsend should be aware, since I made that point when he first asked me about AMTSO. There is, in fact, no reason why any individual shouldn’t apply for membership, in principle, though I’m not sure we’ll be admitting botmasters.

But let’s talk about accountability.

The anti-malware industry is certainly accountable to its customers. If the anti-malware industry believes otherwise, I don’t belong in it, but I’ve never met anyone who works in that industry who believed that users have no rights. In fact, it could be argued (indeed, I have argued) that the anti-malware industry is less effective as a defence because it tries to meet all the expectations of its customers, however realistic. He who pays the piper…

The anti-malware industry is not exactly accountable to the testing industry, with which it has a complicated and symbiotic relationship, except in so far as the testing industry has some claim (implicit or otherwise) to represent the interests of customers. Conversely, I’ve always stated firmly that the testing industry should not be directly accountable to the anti-malware industry. But it should be accountable.

 Most people (and virtually everyone working for the media) are openly cynical about any suggestion that that the anti-malware industry represents anyone’s interests but its own. And it would be extraordinarily naive to argue that individual companies never think about the impact of negative testing on their revenue stream.  Nonetheless, testers (and the journalists and publishers who use their results as stories) should clearly be accountable to their audiences for the accuracy and relevance of their conclusions. AMTSO documentation goes some way towards giving those audiences tools to assess the validity of a test at an abstract level, but there is a more immediate and pragmatic way for a tester to demonstrate awareness of their responsibility. Sharing information and samples not only helps vendors to improve their products, but allows testers to improve their testing by introducing a check on its quality.

Some testers will not share information at all because of internal policy, or because they regard information sharing as a threat to their impartiality and independence. Some are hampered in that respect because of legal/disclosure  issues. While these issues may be genuine and compelling, they do (or should) compromise the perceived validity of a test if they deny all checks on how such issues as to how the test was conducted and whether the published conclusions follow logically from the data. (Yes, that’s all the stuff the AMTSO Principles document tries to address.)

While a testing business model that makes all sharing of information contingent upon a “consultancy” fee is not exactly conforming to the ideal of transparency that AMTSO advocates, we recognize that testers have to make a living too, though it’s of some concern when a tester makes misleading statements about the source of his income to bolster the impression of “independence”. It’s even more of a concern when testers are evasive when asked to share methodological and sample information, because this suggests that they don’t really think their methods will stand up to scrutiny from an informed source.

There’s another area of accountability that has troubled me in the past week or two. Most of the negative comment in the past few weeks seems to have come from journalists who have uncritically accepted the claims of a particular tester. After many years working with and (more recently in) the anti-malware industry, I’m used to journalistic assumptions about vendor hype, scaremongering, competence or the lack of it, and so on: still, it’s slightly alarming that the same journalists seem unaware that testers may also have a marketing agenda.

David Harley CITP FBCS CISSP
Not speaking on behalf of AMTSO