Back in 2008, EICAR rejected a paper proposed by Andrew Lee and myself discussing the state of anti-malware testing and how it might be improved, on the grounds that it was “advertising” the fledgling AMTSO (Anti-Malware Testing Standards Organization) initiative. You can decide for yourselves whether that criticism was justified: the same paper was accepted later in the year by Virus Bulletin and is available as “Who will test the testers?” from the ESET conference papers resource page.
I mention that paper because it makes for an interesting contrast with the paper I presented last week at EICAR 2012. Since the new paper is very much focused on AMTSO, I guess EICAR has got over its sensitivity to 'advertising' the other non-profit organization. (And in fact, there has been a fair amount of subsequent and rational discussion between individuals involved with both organizations.) Though I have to admit that it lacks some of the optimism of the earlier paper – unsurprisingly, given that an awful lot has happened in and to AMTSO in the interim. But it feels like a good time to ask whether AMTSO still has enough credibility to achieve substantially more than it already has. Can the organization go beyond the substantial repository of resources it’s already compiled, to resume monitoring and commenting on tests and testers? (The short answer is probably, but not all by itself, and in any case we'll have more idea about future directions after the discussions at the workshop that begins today: watch this blog for more information.)
Here’s the abstract for the new paper:
Imagine a world where security product testing is really, really useful.
When I snap your fingers, you will wake out of your trance, and we will consider how we could actually bring about this happy state of affairs. For a while, it looked as if AMTSO, the Anti-Malware Testing Standards Organization, might be the key (or at any rate one of the keys), and we will summarize the not inconsiderable difference that AMTSO has made to the testing landscape. However, it’s clear that the organization has no magic wand and a serious credibility problem, so it isn’t going to save the world (or the internet) all on its own. So where do we (the testing and anti-malware communities) go from here? Can we identify the other players in this arena and engage with them usefully and appropriately?
And here’s the abstract for the earlier paper.
Who Will Test The Testers? (2008 Abstract)
The anti-malware industry has been plagued since its earliest days by one poorly designed comparative test after another. In 2007, some of the best anti-malware researchers, comparative testers and product certification specialists took the first steps towards raising product testing standards with the formation of a group specifi cally focused on establishing standards and methodologies, educating both consumers and testers in discrimination between good and bad practice, and providing objective analyses of current testing practices. This paper summarizes current initiatives by the Anti-Malware Testing Standards Organization and other groups, but also considers next steps, going beyond objectifying methodology, educational issues and blowing away the fog of misinformation and fallacy, to the next level. Underlying these vital issues is a question: is it possible to make testers and certifying authorities more accountable for the quality of their testing methods and the accuracy of the conclusions they draw based on that testing?
David Harley CITP FBCS CISSP
ESET Senior Research Fellow