All this is potentially frightening and inconvenient (or worse) for a home user. And if it happens in a corporate environment, it can be very, very expensive to remedy. So while some of the public comments we see in the wake of such incidents may seem over the top, "FP rage" is certainly understandable.
As I mentioned here yesterday, I launched a new AMTSO in the Media page on the AMTSO blog page yesterday. Since then, Pedro Bustamente has kindly sent me a whole bunch of links relating to events leading up to the launch of AMTSO in 2008, so I’ve created a separate sub-page incorporating those links out
Who would have thought that an initiative aimed at increasing the accuracy and relevance of anti-malware testing would be quite so controversial? Well, it was to be expected that AMTSO (the Anti-Malware Testing Standards Organization) would generate a certain amount of controversy: clearly, the organization is not going to get everything right first time. And
No-one believes that AMTSO has all the answers and can “fix” testing all by itself, but it has compiled and generated resources that have made good testing practice far more practicable and understandable. The way for testers (and others) to improve those resources is by talking to and working with AMTSO in a spirit of co-operation: the need for transparency is not going to go away.
...Somewhere in this welter of misinformation, well-meant but muddled thinking, and black propaganda, there are some issues that need clarifying... Watch this space for further information. And while you're waiting, you might want to check the documentation and other resources at the AMTSO web site to see what the organization really proposes and what it is really trying to achieve...
Further to my "top ten of top tens" post, I was encouraged by some queries to revisit the “Top Ten Mistakes Made When Evaluating Anti-Malware Software” list quoted by Kevin Townsend here. As it was an AMTSO issue and most of the queries have related to an AMTSO blog post, I've returned to it (and
Kevin Townsend asks whether AMTSO (the Anti-Malware Testing Standards Organization) is "a serious attempt to clean up anti-malware testing; or just a great big con?" I posted a lengthy response to that on the AMTSO blog here...
Of course, most vendors use in-house testing as a tool for monitoring and improving the capabilities of their own products. However, it’s also being used increasingly as a vehicle for showcasing a company’s own AV products in the best possible light.
[I told you these links were cursed: thanks to Daniel Schatz for pointing out a further problem. Tip of the hat to Kurt Wismer for pointing out the issue on the AMTSO blog, and another to Julio Canto for alerting me to the story in the first place.] Danny Quist posted an interesting article at
Some of us are currently busily preparing for the AMTSO workshop in Helsinki on the 24th and 25th May 2010, just before the CARO workshop on 26th and 27th May (for which registration closes on 12th May). Before the Helsinki events, though, the EICAR conference in Paris includes some interesting testing-related material before and during the main conference.
Just a quick note to draw your attention to a couple of new documents that have just become available. "AMTSOlutely fabulous" (sorry – it seemed like a good idea when I wrote it…) is a review of what the Anti-Malware Testing Standards Organization has achieved so far and what it might achieve in the future. It's
We're not really set up to use the ThreatBlog as a full strength Questions and Answers resource, but we got so many questions after my blog yesterday about April 1st hoaxes that I feel obliged to try to answer some of them. There is no truth in the rumour that the eCity of San Diego
AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009. The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO
There was an AMTSO (Anti-Malware Testing Standards Organization) panel session here at RSA, where Larry Bridwell, Righard Zwienenberg, Andreas Marx, Roel Schouwenberg and Neil Rubenking talked about AMTSO and what it does (and what it hopes to do). And I added to my list of qualifications for being involved with the organization: current vendor representative,
Greetings, friends and fiends. I've been uncharacteristically quiet for the past couple of weeks, due to the AMTSO workshop last week in Santa Clara. There was, as usual, some lively discussion: though no papers were approved at the meeting, some are close enough to finished to be voted on shortly. (See also the AMTSO blog
Security researchers work together and share information in many ways and in many contexts that aren't constrained by company boundaries, but it's unusual for security researchers working for different vendors to join forces in a company blog. However, John Leyden of The Register contacted us both when he was writing an article on the controversy following
Larry Seltzer posted an interesting item yesterday. The article on "SW Tests Show Problems With AV Detections " is based on an "Analyst's Diary" entry called "On the way to better testing." Kaspersky did something rather interesting, though a little suspect. They created 20 perfectly innocent executable files, then created fake detections for ten of them.
We're now getting into preparations for the next meeting of AMTSO (Anti-Malware Testing Standards Organization), on 25th-26th February in Santa Clara. In the meantime, I wrote an article for Virus Bulletin called "AMTSOlutely Fabulous" about "the story so far". It's just appeared in the January edition of the magazine. Of course, it's only available to subscribers
The Hype-free blog at http://hype-free.blogspot.com/2009/12/congratulation-to-av-comparatives.html yesterday mentioned the latest AV-Comparatives round of test reports, including: The whole product dynamic test at http://www.av-comparatives.org/comparativesreviews/dynamic-tests The December 2009 performance test at http://www.av-comparatives.org/comparativesreviews/performance-tests The summary reports at http://www.av-comparatives.org/comparativesreviews/main-tests/summary-reports I have a pretty jaundiced view of testing organizations in general: after all, I see some pretty awful tests proclaimed by the
I recently made a presentation to the Special Interest Group in Software Testing of the BCS Chartered Institute for IT (formerly better known as the British Computer Society). The PDF version of the slide deck is now up at: http://www.eset.com/download/whitepapers/Curious_Act_Of_Anti_Malware_Testing.pdf The presentation outlines some of the problems with anti-malware testing and summarizes the mission and principles of