Why Anti-Virus is not a waste of money

It has happened before, it just happened again and it will happen in the future. It is inevitable! Some company that needs to get some press coverage or public visibility will release yet another statement on how worthless Anti-Virus is, based on its own dysfunctional test.

For this “test”, they used the VirusTotal service. VirusTotal is a great service, but not suitable for testing Anti-Virus performance in this way. For a start, VirusTotal only uses the command-line scanner versions of the products that support VirusTotal. Even if a new threat is not detected by the command-line scanner, it does not mean that the threat would not be stopped by another Security Suite module. Nowadays there is no mainstream vendor anymore that only offers its customers an on demand command-line scanner: modern solutions are Security Suites with many additional components such as Anti-Phishing, Anti-Spam, IDS, (H)IPS, etc., all kinds of security technologies NOT reflected in the VirusTotal result.

Furthermore, some of the solutions included in VirusTotal are configured according to the parameters requested by the vendor, with a more aggressive level of heuristic detection than the official end-user default configuration would offer.  This may result in a detection whereas the official end-user configuration would never detect the threat, so VirusTotal can be really, really misleading if used to compare performance between products..  

Having the different products run with different parameters, resulting in different levels of heuristic paranoia is common practice during anti-malware tests, but then these differing parameters are the result of using so-called out-of-the-box settings: in other words, as the vendors have prepared them for what they consider to be the best end-user experience. It’s not unreasonable to test this way as long as its clear to the audience that what is being tested is not absolute detection, but configuration philosophy. A better way of testing overall detection is to modify individual products so that the level of heuristic aggression is equivalent for each product. However, tweaking some products and not for others will make the result equivalent to comparing apples and oranges. VirusTotal is unsuitable for comparing products because essentially, this is what it does. But that, of course, is fine because comparing products is not what VirusTotal is for. The importance of comparing like with like in the testing context has been described in 2008 already in the “Guidelines to the Fundamental Principles of Testing” document of the Anti?Malware Testing Standards Organization (AMTSO).

Similarly it is not correct to compare consumer versions against corporate versions. Imperva actually suggests for companies to use the free Avast! solution. Whereas there is nothing wrong with the Avast! product by itself, the free version is meant for consumers and not for businesses. These two branches of the security tree should not be directly compared to each other. In a corporate environment anti-virus solutions work with a central management system for enterprise wide reporting and configuration, an integration which is absent in consumer products. In corporate environments, thresholds for heuristics may be set higher than in the free consumer-targeted versions. In other words, heuristic detection in a corporate environment is likely to be less aggressive, to lessen the risk of false positives. A false positive in a corporate environment can make the entire network inoperable and cause significantly more damage: for a business, even transient inconvenience can be expensive. A lot of the time, the thresholds for heuristics are set much lower for free antivirus products. A false positive in a free consumer product is generally seen as not so severe in impact, and feedback from an affected system may provide valuable data for the corporate version.

David Harley has published a paper that analyses in detail the problems with one test that displayed a certain inconsistency between the test objective described and the methodology used.

Julio Canto of VirusTotal/Hispasec Sistema together with ESET’s David Harley wrote a paper on this topic called “Man, Myth, Malware and Multi-Scanning”. In the paper they documented: “VirusTotal was not designed as a tool to perform AV comparative analyses, but to check suspicious samples with multiple engines, and to help AV labs by forwarding them the malware they failed to detect."

"How not to use VT: VirusTotal uses a group of very heterogeneous engines. AV products may implement roughly equivalent functionality in enormously different ways, and VT doesn’t exercise all the layers of functionality that may be present in a modern security product. VirusTotal uses command-line versions: that also affects execution context, which may mean that a product fails to detect something it would detect in a more realistic context. It uses the parameters that AV vendors indicate: if you think of this as a (pseudo)test, then consider that you’re testing vendor philosophy in terms of default configurations, not objective performance. Some products are targeted for the gateway: gateway products are likely to be configured according to very different presumptions to those that govern desktop product configuration."

"Conclusion

VirusTotal is self-described as a TOOL, not a SOLUTION: it’s a highly collaborative enterprise, allowing the industry and users to help each other. As with any other tool (especially other public multi-scanner sites), it’s better suited to some contexts than others. It can be used for useful research or can be misused for purposes for which it was never intended, and the reader must have a minimum of knowledge and understanding to interpret the results correctly. With tools that are less impartial in origin, and/or less comprehensively documented, the risk of misunderstanding and misuse is even greater."

These are just some examples illustrating why using VirusTotal for antivirus testing is a bad idea. Our colleagues on the Prevx team also made an entry in their blog discussing the matter.

Why is anti-virus not a waste of money? The service, the support, the timely updates, the research into (future) threats, etc. There is NO such thing as a free anti-virus. Because of the work that is put into those ‘free’ products, the developer needs to get some return on his investment (ROI). Most often this is done by installing “complementary” toolbars, utilities containing adware-like functionality, and so forth, where the client is monitored and served with information “you need.”  These add-on programs, subsidize the cost of the “free” anti-virus potentially at the expense of your privacy.  Additionally, once an anti-virus company includes a toolbar with their “free” offering, they may be pressured by the toolbar vendor to exclude detection of other products bundling the toolbar vendor’s software, which may be more intrusive in nature and cross the line from grey into black.

The following two white papers provide additional information about such questionable software:

  1. Lawyer in the lab
  2. Problematic-Unloved-Argumentative

There is an unfortunate tendency to believe, especially amongst consumers, that anti-virus software serves as a magic forcefield which protects their computer from goblins and other things which go bump in the night.  While this sometimes is the case, anti-virus software is more often like automobile insurance:  You may not like purchasing it, but when an “accident” occurs, you’ll be glad that you bought the plan with the best support.

 

Author Righard Zwienenberg, ESET

  • Brian

    How would you say AV performance is at detecting exploits (as opposed to malware payloads)?  Some say that they aren't designed to detect exploits and yet many of them attempt to.

    • David Harley

      Attempt is the key word. We certainly detect some known exploits (for instance, CVE-2012-0507, detected as Java/Exploit.CVE-2012-0507.DX, used by OSX/Dockster et al).But AV isn’t a substitute for OS or application patching, and you can’t rely on it to detect vulnerabiiities that might be exploited, either. Horses for courses.

  • Bill Pytlovany

    Thanks for another thought provoking discussion. I've always been a big fan of VirusTotal. There have been times I've suggested users upload a file they know is bad as a way to figure out which AV solutions know about the threat.

    Obviously, this may not always be a clear cut way to find an automated cleanup tool.  I always say which AV solution I recommend can change monthly.  Based on recent tests I'm pleased to say this month ESET is a must have for the folks who ask me for recommendations.

    Bill

     

    • David Harley

      Actually, it is clearcut. A VirusTotal report doesn‘t tell you which solutions know about a threat. It tells you which (if any) solutions will flag it as a threat under very restricted conditions that don’t reflect real-world conditions. A useful comparative test has to be much more rigorous than that, but VT was never intended as a way of testing AV.

  • Brian

    Is there a service that will test the full desktop versions of various AV programs rather than just the CLI file scanners so we could make more representative tests?

    • Righard Zwienenberg

       
      That would be problematic. First of all, the entire security suite has to be installed – for all products. That would require a lot of machines and or VM’s. Installing them all together on one machine is not possible. Second, it would require resource intensive testing. Most full suites do require interaction.
      And then I’m not even touching other implementation issues as e.g. when something is a network worm and will never be written to the harddisk (e.g. CodeRed comes to mind), you would have to “inject” the malicious packages into the network stream. Not at all a realistic approach.

  • Unknown

    i would like to ask that can we use two antivirus on a single workstation.
    i want to use ESET Smart Security With Emsisoft Anti-Malware

    • David Harley

      I don’t know much about Emsisoft, but I wouldn’t recommend running two antivirus products of the same type at the same time on a single workstation. The additional security that I presume you’re hoping for wouldn’t necessarily compensate for the additional load on system resources. Where you have two products that work in a somewhat similar way, there’s a risk that having both in memory at the same time will actually lessen their effectiveness, even if there aren’t serious incompatibilities. (Some would probably say that just the risk is a serious incompatibility!)

  • GlennJ

    Someone above mentioned that AV solutions do not try to block exploits, but some claim to do so anyway.  In the case of AVG the AVGlinkscanner component (not part of VirusTotal testing) blocks exploits such as the Blackholes and so forth in real time and it does pretty well at that.
    Using VirusTotal for testing this capability of the AVG product suite would be Totally misleading, which supports Righard's thesis in this article.

Follow Us

Automatically receive new posts via email:

Delivered by FeedBurner

4 articles related to:
Hot Topic
04 Dec 2012
ESET Virus Radar

Archives

Select month
Copyright © 2014 ESET, All Rights Reserved.