An AMTSO for biometrics?

Kevin Townsend and I exchanged email last month on the question as to whether the biometrics industry needed its own testing standards organization somewhat equivalent to AMTSO. Kevin isn’t AMTSO’s biggest fan, but he asked some pretty relevant questions about the purpose and value of AMTSO. The discussions came out of  an upcoming market report on biometrics and smartphones to be published in Q1 2011 by Goode Intelligence. I haven’t seen the actual report, but my understanding is that his thesis is that if accurate figures for conditions such as false positives and false negatives are collected from different biometric systems in a comparative context, this would constitute a viable if not optimal means of evaluation. The question (one that we’re all too familiar with in the anti-malware industry) is how to avoid introducing bias towards or against products and methodologies. It took many years for the AV and testing industries to take the initiative on defining and providing guidance on methodologies, and AMTSO is still very much in a formative stage. Could the biometrics industry benefit from our experiences and reach a point where self-regulation would tend to work to the advantage of the better products in the market place? I’m not, of course, saying that that’s where the anti-virus product and testing industries are right now, or attempt to address the question of how you define “better”.

The short answer, I suppose, is that I’m not sure how applicable AMTSO’s experience is to the biometric industry and product evaluation. The malware threatscape offers particular difficulties in terms of testing, like a threatbase that already runs to 40-50 million items and increases by tens or even hundreds of thousands of items a day, and the fact that vendors in this space are driven to secretiveness, not only in terms of competitive advantage, but also the pressures of an ongoing war or attrition with the bad guys. I’m not underestimating the technical difficulties that biometrics vendors face, but they don’t face them in an environment that’s more like an accelerator or a pressure cooker than a normal “see a problem, design a solution” environment. But stakeholders in that sector probably can learn from the problems AMTSO has faced in terms of harmonizing the  needs of vendors, testers, customers and the media, and certainly people more directly concerned with AV testing can. Perhaps we’re due for a “position statement” on what AMTSO has (or hasn’t) achieved that will help . I plan to discuss that assertion at the AMTSO workshop in San Mateo later this week, and I’ll report back here in due course.

David Harley