Testing, Terms and Conditions, and a Licence to Bill

I remember – a decade or two ago – there was a certain amount of fuss when a certain security company inserted into its licence agreements the stipulation that the product was not to be tested without the blessing of the company.

When I looked at the issue again recently as part of my research into the paper I’m presenting at Virus Bulletin the first week in October, I found that the same company’s licensing still contains a similar stipulation. But it’s not unique in that. In fact, Iain Thompson reports for The Register that:

More and more infosec software makers now include legal language in their T&Cs insisting that their products cannot be tested for usefulness if the results are going to be published. 

I was actually looking at this issue because of recent attempts by so-called next-gen vendors to control or block tests/reviews of their products, such as the unsuccessful attempt by Crowdstrike to repress the results of testing by NSS, an instance mentioned in Thompson’s article. The article also quotes John Strand’s keynote at DerbyCon at which he talks of “vendors acting like bullies ” and researchers terrified of litigation. He claims that “…most of the analysis of products you see is either from the vendor or vendor-approved.” In fact, ‘no testing allowed’ provisions are pretty hard to enforce in real life, but it doesn’t stop some vendors trying.

I have to agree that it’s naive to assume impartiality when one company compares the performance of its own products against that of its competitors, and there are all too many examples of tests and reviews that are heavily influenced (or even executed) by a participating or sponsoring vendor. I’m far from convinced that approval by a vendor is necessarily a bad thing, though, or that bullying in the testing arena is solely restricted to vendors. Though it has been argued that software vendors have always tended towards a high-handed attitude towards their customers with its preference for licensing (preferably shrinkwrap) of products rather than transfer of ownership.

Nevertheless, given my less-than-secret dislike of the exploitation by certain marketroids of an essentially illusory generation gap, it may surprise you to learn that I’m not totally unsympathetic to the concerns of security companies old and new that mistrust product testers and seek to block or control their public pronouncements. After all, I’ve seen a lot of incompetent tests in the last three decades, and it’s not surprising that vendors want to address issues with tests that don’t seem to treat their products fairly.

It’s not that I think that the testing industry should be directly accountable to or regulated by the security industry, of course: the testing industry should represent the interests of the consumer, not the vendor. And it’s to their audience/consumers that testers should be accountable for the accuracy and relevance of their conclusions, just as a security vendor is responsible to its consumers for its effectiveness.

That doesn’t mean that a tester shouldn’t be willing to cooperate with vendors to improve the quality and accuracy of a test, of course, in issues such as disputing/verifying samples. But you can’t take for granted the accuracy of a test that primarily serves the interests of a single vendor, whether it’s a test carried out by that vendor, or by a testing organization closely allied with that vendor, or even a test sponsored by a vendor. Testing is a ‘for-profit industry’, not a state-sponsored ombudsman or charitable institution. That may make it vulnerable to ‘bullying’ by vendors who withdraw their sponsorship or participation. But testers can bully, too. The combination of a ‘pay-to-play’ test with involuntary participation generally results in a scenario where vendors have to pay to participate, since otherwise they have no comeback when they’re penalized unfairly. If the vendor declines to offer a current licence for a product under test, the tester may acquire one indirectly, or even worse, test an obsolescent program version. This doesn’t seem fair, but the tester can argue that enforced participation is in the public interest. And there’s something in that, as long as the test is valid. But, sadly, it’s not safe to assume that a test is accurate or valid just because the tester says it is.

Even in AMTSO, which owes its existence to the willingness of testers and vendors to discuss their differences and come up with strategies that benefit the consumer, there have in the past been reports of vendors threatening testers by ‘reporting’ them to AMTSO when they don’t do well in tests, and of testers threatening an already fragile coalition by voting with their feet and enhancing the widespread view of AMTSO’s as a vendor clique, as well as withdrawing financial support. AMTSO still seems the best chance we have at present of maintaining a testing ecology, but there is a perpetual tension between conflicting financial imperatives:

• Because potential customers are influenced in their choice of security products, security vendors to some extent rely on looking good in tests. Sometimes that overrules the question of whether the test itself is adequate, and sometimes that results in overstepping ethical boundaries.
• Because testing (especially competent testing) is expensive to implement, most regular tests rely on financial input from the security industry, and sometimes that results in a testing organization’s overstepping ethical boundaries. Is there an echo in here?

How do we get round this? Well, it’s hard to disagree with Vesselin Bontchev on this. Actually, it’s usually pretty tough to disagree with him on anything. 🙂 In a blog article on the NSS/Crowdstrike bunfight, he observed that:

My personal opinion is that the companies whose products are being tested shouldn’t pay for the tests — the users of the tests should pay for the results.

That seems perfectly logical, but how feasible is it? Well, sometimes a potential corporate customer will pay a hefty sum to a tester for a comparative review. Sometimes a periodical or web site will pay a testing organization to conduct a test on its behalf, and pass on the cost of the test to its subscribers. But much of the time, the money comes from an individual vendor sponsoring a test, or a group of vendors participating in a group test. And hoping their results will be good enough to get a return on the investment.

Sometimes this all works to the advantage of the customer, especially within the framework of AMTSO’s principles.  Sometimes it doesn’t. We can but hope that AMTSO’s initiatives will continue to improve the situation.

David Harley