NSS Labs: AMTSO’s Review Analysis

AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009.

The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO Principles:

  • Principle 6 “Testing methodology must be consistent with the testing purpose”
  • Principle 7 “The conclusions of the test must be based on the test results”.

The stated purpose of the test was only to determine the protection of the products tested against socially-engineered malware. It did not consider other layers of protection, so the conclusion in the Executive Summary of the report – “Products that earn a caution rating from NSS Labs should not be short-listed or renewed” – is unprofessional and misleading. As the Review Analysis report states:

“This is clearly a conclusion that you can't make out of the detection for socially-engineered malware only, as the products have other layers of protection that the test did not evaluate.”

This conclusion was supported and clarified in AMTSO’s answer to a supplementary question:

“Does the interpretation of the results follow logically from the data as presented? – No. As above, the conclusion is too general in its recommendations and condemnations, considering that only a portion of each product's functionality was tested.”

AMTSO does not currently undertake to verify the test samples used in tests that it reviews. Nor has ESET been able to obtain any of the samples used in this test from NSS. We are thus unable to verify exactly what samples were used, let alone their quality, relevance and representativeness in terms of how they were selected. This evasiveness on the tester’s part in itself casts doubts on the test’s openness and transparency according to AMTSO Principle 3. AMTSO’s Review Analysis Committee was not able to agree unanimously on whether the test was compliant with this Principle.

NSS Labs is no longer a member of the Anti-Malware Testing Standards Organization, and so no longer has a representative on the Review Analysis Board.

Andrea Kokavcova
Senior Market Research Analyst

Author , ESET

  • Ed Ferguson

    I have begun installing ESET on my customers networks and have been happier than ever using an antivirus product.  In the past few years I have been using Trend Micro and AVG due to reports of how good these products are and increasingly find myself spending hours upon hours of time cleaning malware off of systems that were undetected by these products or were detected by unable to remove.
    Since using ESET things have finally begun to quiet down and its not because malware is not being detected and therefore I don't know that it is there.  If these programs are there then they do things to the system that make it impossible to work and so its obvious they are present.  But with ESET I have not had these programs get through.  And the greatest part is that people are not complaining anymore about how slow there systems have become after I upgraded their antivirus software.
    So I don't know either what this report is trying to say.  But in my work I have experienced a huge drop in having to spend time worrying about problems that my antivirus solution is causing.

Follow us

Copyright © 2017 ESET, All Rights Reserved.