NSS Labs: AMTSO’s Review Analysis

NSS Labs: AMTSO’s Review Analysis

AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009. The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO

AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009. The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO

AMTSO (the Anti-Malware Testing Standards Organization) has published its review analysis of the Endpoint Security Test that was published by NSS Labs on September 8, 2009.

The Review Analysis published on March 17, 2010 compared AMTSO’s Fundamental Principles of Testing to the NSS Labs report and found that it doesn’t comply with two of the nine AMTSO Principles:

  • Principle 6 “Testing methodology must be consistent with the testing purpose”
  • Principle 7 “The conclusions of the test must be based on the test results”.

The stated purpose of the test was only to determine the protection of the products tested against socially-engineered malware. It did not consider other layers of protection, so the conclusion in the Executive Summary of the report – “Products that earn a caution rating from NSS Labs should not be short-listed or renewed” – is unprofessional and misleading. As the Review Analysis report states:

“This is clearly a conclusion that you can't make out of the detection for socially-engineered malware only, as the products have other layers of protection that the test did not evaluate.”

This conclusion was supported and clarified in AMTSO’s answer to a supplementary question:

“Does the interpretation of the results follow logically from the data as presented? – No. As above, the conclusion is too general in its recommendations and condemnations, considering that only a portion of each product's functionality was tested.”

AMTSO does not currently undertake to verify the test samples used in tests that it reviews. Nor has ESET been able to obtain any of the samples used in this test from NSS. We are thus unable to verify exactly what samples were used, let alone their quality, relevance and representativeness in terms of how they were selected. This evasiveness on the tester’s part in itself casts doubts on the test’s openness and transparency according to AMTSO Principle 3. AMTSO’s Review Analysis Committee was not able to agree unanimously on whether the test was compliant with this Principle.

NSS Labs is no longer a member of the Anti-Malware Testing Standards Organization, and so no longer has a representative on the Review Analysis Board.

Andrea Kokavcova
Senior Market Research Analyst

Discussion