Product Testing: Simulation, dissimulation, exasperation
Malware simulators are not an appropriate basis for testing product detection capabilities.
Testing
Better Mac Testing: Static versus Dynamic Testing
Dynamic or on-access Mac testing of AV products is problematical with samples for which Apple has implemented signature detection.
Better Mac Testing? How OS security can make AV testing harder
As Mac malware increases in prevalence, testing security software that supplements OS X internal security gets more important and more difficult.
Why Anti‑Virus is not a waste of money
It has happened before, it just happened again and it will happen in the future. It is inevitable! Some company that needs to get some press coverage or public visibility will release yet another statement on how worthless Anti-Virus is, based on its own dysfunctional test. For this “test”, they used the VirusTotal service. VirusTotal
AMTSO's New Direction (and some resources updates)
AMTSO's discussions on its own new directions, and updates to its testing-related resources.
AV Testing, AMTSO and EICAR
A new conference paper discusses whether AMTSO has the credibility to achieve its aims of raising testing standards on its own.
VirusTotal, Useful Engines, and Useful AV
The paper by Julio Canto and myself on the use and misuse of multi-scanner malware-checking resources like VirusTotal is now available.
Testing presentation slides: old whine in new bottle
The slides from an AMTSO-oriented presentation by Larry Bridwell and myself at this year's Virus Bulletin conference, on "'Daze of whine and neuroses (but testing is FINE)" are now available on the Virus Bulletin site are now available here (along with some other excellent presentations). The paper on which the presentation is based is on the ESET white papers
PUAs: ESET’s Most Unwanted List
Aryeh Goretsky interviewed, as his paper on Possibly Unwanted Applications is published.
AMTSO’s helping hand to testers
...AMTSO's members have approved a document that offers guidelines to vendors on ways in which they can make it easier to test products accurately....
Simulation Testing and the EICAR test file
Summary of and link to an AVAR paper addressing some of the pitfalls of using malware simulation in product testing.
Virus Bulletin 2010 papers
By kind permission of Virus Bulletin, we've already put two of the papers written or co-authored by ESET researchers up on the White Papers page.
Testing and Accountability
No-one believes that AMTSO has all the answers and can “fix” testing all by itself, but it has compiled and generated resources that have made good testing practice far more practicable and understandable. The way for testers (and others) to improve those resources is by talking to and working with AMTSO in a spirit of co-operation: the need for transparency is not going to go away.
Testing and how not to do it
Further to my "top ten of top tens" post, I was encouraged by some queries to revisit the “Top Ten Mistakes Made When Evaluating Anti-Malware Software” list quoted by Kevin Townsend here. As it was an AMTSO issue and most of the queries have related to an AMTSO blog post, I've returned to it (and
Top Ten of Top Tens
Well, not exactly, though actually a top ten of top tens isn't a bad idea: apparently, top tens usually attract plenty of readers. As do top fives. twenties etc, though probably not top thirteens. Security Memes a Lot to Me Still, there is a touch of recursion to this post. I got a notification from
AMTSOlutely Fabulous or Utter BS?
Kevin Townsend asks whether AMTSO (the Anti-Malware Testing Standards Organization) is "a serious attempt to clean up anti-malware testing; or just a great big con?" I posted a lengthy response to that on the AMTSO blog here...
Test Toot Suite: Antivirus Vendors Blowing Own Horn
Of course, most vendors use in-house testing as a tool for monitoring and improving the capabilities of their own products. However, it’s also being used increasingly as a vehicle for showcasing a company’s own AV products in the best possible light.
NSS Labs: AMTSO’s Review Analysis
AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009. The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO
I Have a Little (Wild)List*
* http://math.boisestate.edu/gas/mikado/webopera/mk105a.html Kevin Townsend posted a blog in response to a piece by Mike Rothman at Securosis. Mike’s piece on “The Death of Product Reviews” makes some pretty good points about security product reviews in general. Kevin’s piece is more specific to anti-malware. He too makes some useful discussion points about the value or otherwise
Kaspersky, Virus Total, and Unacceptable Shortcuts
Larry Seltzer posted an interesting item yesterday. The article on "SW Tests Show Problems With AV Detections " is based on an "Analyst's Diary" entry called "On the way to better testing." Kaspersky did something rather interesting, though a little suspect. They created 20 perfectly innocent executable files, then created fake detections for ten of them.