AV Testing, AMTSO and EICAR

Back in 2008, EICAR rejected a paper proposed by Andrew Lee and myself discussing the state of anti-malware testing and how it might be improved, on the grounds that it was “advertising” the fledgling AMTSO (Anti-Malware Testing Standards Organization) initiative. You can decide for yourselves whether that criticism was justified: the same paper was accepted later in the year by Virus Bulletin and is available as “Who will test the testers?” from the ESET conference papers resource page.

I mention that paper because it makes for an interesting contrast with the paper I presented last week at EICAR 2012. Since the new paper is very much focused on AMTSO, I guess EICAR has got over its sensitivity to 'advertising' the other non-profit organization.  (And in fact, there has been a fair amount of subsequent and rational discussion between individuals involved with both organizations.) Though I have to admit that it lacks some of the optimism of the earlier paper –  unsurprisingly, given that an awful lot has happened in and to AMTSO in the interim. But it feels like a good time to ask whether AMTSO still has enough credibility to achieve substantially more than it already has. Can the organization go beyond the substantial repository of resources it’s already compiled, to resume monitoring and commenting on tests and testers? (The short answer is probably, but not all by itself, and in any case we'll have more idea about future directions after the discussions at the workshop that begins today: watch this blog for more information.)

Here’s the abstract for the new paper:

After AMTSO: a funny thing happened on the way to the forum

Imagine a world where security product testing is really, really useful.

• Testers have to prove that they know what they’re doing before anyone is allowed to draw conclusions on their results  in a published review.
•  Vendors are not able to game the system by submitting samples that their competitors are unlikely to have seen, or to buy their way to the top of the rankings by heavy investment in advertising with the reviewing publication, or by engaging the testing organization for consultancy.
• Publishers acknowledge that their responsibility to their readers means that the claims they make for tests they sponsor should be realistic, relative to the resources they are able to put into them.
• Vendors don’t try to pressure testers into improving their results by threatening to report them to AMTSO.
• Testers have found a balance between avoiding being unduly influenced by vendors on one hand and ignoring informed and informative input from vendors on the other.
• Vendors don’t waste time they could be spending on enhancing their functionality, on tweaking their engines to perform optimally in unrealistic tests.
• Reviewers don’t magnify insignificant differences in test performance between products by  camouflaging a tiny sample set by using percentages, suggesting that a product that detects ten out of ten samples is 10% better than a product that only detects nine.
• Vendors don’t use tests they know to be unsound to market their products because they happened to score highly.
• Testers don’t encourage their audiences to think that they know more about validating and classifying malware than vendors.
• Vendors and testers actually respect each others work.

When I snap your fingers, you will wake out of your trance, and we will consider how we could actually bring about this happy state of affairs.  For a while, it looked as if AMTSO, the Anti-Malware Testing Standards Organization, might be the key (or at any rate one of the keys), and we will summarize the not inconsiderable difference that AMTSO has made to the testing landscape. However, it’s clear that the organization has no magic wand and a serious credibility problem, so it isn’t going to save the world (or the internet) all on its own. So where do we (the testing and anti-malware communities) go from here? Can we identify the other players in this arena and engage with them usefully and appropriately?

And here’s the abstract for the earlier paper.

Who Will Test The Testers? (2008 Abstract)

The anti-malware industry has been plagued since its earliest days by one poorly designed comparative test after another. In 2007, some of the best anti-malware researchers, comparative testers and product certification specialists took the first steps towards raising product testing standards with the formation of a group specifi cally focused on establishing standards and methodologies, educating both consumers and testers in discrimination between good and bad practice, and providing objective analyses of current testing practices. This paper summarizes current initiatives by the Anti-Malware Testing Standards Organization and other groups, but also considers next steps, going beyond objectifying methodology, educational issues and blowing away the fog of misinformation and fallacy, to the next level. Underlying these vital issues is a question: is it possible to make testers and certifying authorities more accountable for the quality of their testing methods and the accuracy of the conclusions they draw based on that testing?

David Harley CITP FBCS CISSP
ESET Senior Research Fellow