Testing: the Knight’s Tale

Strange how the whole AV testing issue seems to attract metaphors and analogies harking  back in some sense to mediaeval romance. For the past few days I’ve been muttering grimly about tilting at windmills, having recently come across an unhappy example of a test apparently relying on a simulation. (You’ll probably hear more from me on that, though I won’t necessarily be performing my impersonations of Don Quixote on this blog.)

And now I’ve come across a recent blog by Eugene Kaspersky in which he talks about The Holy Grail of AV Testing, and Why It Will Never Be Found: in fact, making a plea for something that sounds very similar to “whole product testing” as defined by AMTSO (which features strongly in his argument), and building on another of his articles, as I previously mentioned here. Essentially, he comes to the depressing conclusion that “…sadly, proper results from proper tests are these days simply nowhere to be found, despite their Holy Grail status.”

“Nowhere” might be a little harsh, but I have to admit that I’ve heard a lot of similar sentiments expressed recently. Yet here I am at yet another AMTSO workshop, halfway through a gruelling schedule of meetings. AMTSO is still here, and still trying to establish some form of chivalric code. Please keep any mutterings about honour among thieves to yourself: after many years in or on the fringes of the AV industry, I’ve heard it all, but I haven’t (altogether) lost faith in the ability of the vendor and tester communities to put personal differences and vested interests aside for the protection of the end user. Flashbacks to Monty Python notwithstanding.

David Harley