False Positives and Apportioning Blame

All this is potentially frightening and inconvenient (or worse) for a home user. And if it happens in a corporate environment, it can be very, very expensive to remedy. So while some of the public comments we see in the wake of such incidents may seem over the top, “FP rage” is certainly understandable.

AMTSO in the Media: the Prequel

As I mentioned here yesterday, I launched a new AMTSO in the Media page on the AMTSO blog page yesterday. Since then, Pedro Bustamente has kindly sent me a whole bunch of links relating to events leading up to the launch of AMTSO in 2008, so I’ve created a separate sub-page incorporating those links out

AMTSOspheric* Pressure

Who would have thought that an initiative aimed at increasing the accuracy and relevance of anti-malware testing would be quite so controversial? Well, it was to be expected that AMTSO (the Anti-Malware Testing Standards Organization) would generate a certain amount of controversy: clearly, the organization is not going to get everything right first time. And

Testing and Accountability

No-one believes that AMTSO has all the answers and can “fix” testing all by itself, but it has compiled and generated resources that have made good testing practice far more practicable and understandable. The way for testers (and others) to improve those resources is by talking to and working with AMTSO in a spirit of co-operation: the need for transparency is not going to go away.

I AMTSO confused….

…Somewhere in this welter of misinformation, well-meant but muddled thinking, and black propaganda, there are some issues that need clarifying… Watch this space for further information. And while you’re waiting, you might want to check the documentation and other resources at the AMTSO web site to see what the organization really proposes and what it is really trying to achieve…

Testing and how not to do it

Further to my "top ten of top tens" post, I was encouraged by some queries to revisit the “Top Ten Mistakes Made When Evaluating Anti-Malware Software” list quoted by Kevin Townsend here. As it was an AMTSO issue and most of the queries have related to an AMTSO blog post, I've returned to it (and

AMTSOlutely Fabulous or Utter BS?

Kevin Townsend asks whether AMTSO (the Anti-Malware Testing Standards Organization) is “a serious attempt to clean up anti-malware testing; or just a great big con?” I posted a lengthy response to that on the AMTSO blog here…

Test Toot Suite: Antivirus Vendors Blowing Own Horn

Of course, most vendors use in-house testing as a tool for monitoring and improving the capabilities of their own products. However, it’s also being used increasingly as a vehicle for showcasing a company’s own AV products in the best possible light.

AMTSO, Standards, and Relevance

[I told you these links were cursed: thanks to Daniel Schatz for pointing out a further problem. Tip of the hat to Kurt Wismer for pointing out the issue on the AMTSO blog, and another to Julio Canto for alerting me to the story in the first place.] Danny Quist posted an interesting article at


  Some of us are currently busily preparing for the AMTSO workshop in Helsinki on the 24th and 25th May 2010, just before the CARO workshop on 26th and 27th May (for which registration closes on 12th May). Before the Helsinki events, though, the EICAR conference in Paris includes some interesting testing-related material before and during the main conference.

New Documents

Just a quick note to draw your attention to a couple of new documents that have just become available. "AMTSOlutely fabulous" (sorry – it seemed like a good idea when I wrote it…) is a review of what the Anti-Malware Testing Standards Organization has achieved so far and what it might achieve in the future. It's

April 1st: Your Questions Answered

We're not really set up to use the ThreatBlog as a full strength Questions and Answers resource, but we got so many questions after my blog yesterday about April 1st hoaxes that I feel obliged to try to answer some of them. There is no truth in the rumour that the eCity of San Diego

NSS Labs: AMTSO’s Review Analysis

AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009. The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO

RSA, AMTSO, the Universe and Everything

There was an AMTSO (Anti-Malware Testing Standards Organization) panel session here at RSA, where Larry Bridwell, Righard Zwienenberg, Andreas Marx, Roel Schouwenberg and Neil Rubenking talked about AMTSO and what it does (and what it hopes to do). And I added to my list of qualifications for being involved with the organization: current vendor representative,


Greetings, friends and fiends. I've been uncharacteristically quiet for the past couple of weeks, due to the AMTSO workshop last week in Santa Clara. There was, as usual, some lively discussion: though no papers were approved at the meeting, some are close enough to finished to be voted on shortly. (See also the AMTSO blog

Cascading False Positives

 Security researchers work together and share information in many ways and in many contexts that aren't constrained by company boundaries, but it's unusual for security researchers working for different vendors to join forces in a company blog. However, John Leyden of The Register contacted us both when he was writing an article on the controversy following

Kaspersky, Virus Total, and Unacceptable Shortcuts

Larry Seltzer posted an interesting item yesterday.  The article on "SW Tests Show Problems With AV Detections " is  based on an "Analyst's Diary" entry called "On the way to better testing." Kaspersky did something rather interesting, though a little suspect. They created 20 perfectly innocent executable files, then created fake detections for ten of them.

AMTSOlute Elsewhere

We're now getting into preparations for the next meeting of AMTSO (Anti-Malware Testing Standards Organization), on 25th-26th February in Santa Clara. In the meantime, I wrote an article for Virus Bulletin called "AMTSOlutely Fabulous" about "the story so far". It's just appeared in the January edition of the magazine. Of course, it's only available to subscribers

A Thought or Two about Testing

The Hype-free blog at yesterday mentioned the latest AV-Comparatives round of test reports, including: The whole product dynamic test at The December 2009 performance test at The summary reports at I have a pretty jaundiced view of testing organizations in general: after all, I see some pretty awful tests proclaimed by the

The Curious Art of Anti-Malware Testing

I recently made a presentation to  the Special Interest Group in Software Testing of the BCS Chartered Institute for IT (formerly better known as the British Computer Society). The PDF version of the slide deck is now up at: The presentation outlines some of the problems with anti-malware testing and summarizes the mission and principles of

Follow Us

Automatically receive new posts via email:

Delivered by FeedBurner

2 articles related to:
Hot Topic
ESET Virus Radar


Select month
Copyright © 2014 ESET, All Rights Reserved.