VirusTotal is not a Comparative Analysis Tool!

[Dead link changed 27th November 2012]

Most of us have been in Estonia for the past few days for a couple of conferences. You may hear more about that later, when Normal Service is resumed. One thing I wanted to remark on now, though (partly because it relates directly to some presentations I've been doing) is a spike in the use of VirusTotal as a tool for comparing detection performance. This is a topic we (and the guys at VirusTotal/Hispasec themselves, who are a really good bunch) are rather sensitive about.

I'll probably come back to this in the near future, but the gist of the problem is this. VirusTotal is a tool many people find very useful as a shortcut to checking a possibly malicious file, but it isn't a detection test. Most importantly, it submits the files you submit to a battery of command-line scanners. This gives you a good chance of identifying a known malicious program, but the fact that a scanner doesn't identify a file as malware does not mean it isn't malicious, obviously. However, if a file is identified as malicious by one group of scanners but not another, it doesn't necessarily mean that the second group is less competent at detection, either. Scanners that use sophisticated behaviour analysis, active heuristics and so on are disadvantaged by this misuse as a comparative test tool, since there is no behaviour to analyse. Generally, command-line scanners simply look at the code passively, rather than running it in a safe environment to see what it does in practice, so products that are heavily dependent on signature detection may seem to do better than products with advanced heuristics. In the real world, however, where on-access scanning is the first line of defence for most people, the advantage tends to swing the other way.

You might want to check out what Hispasec/VirusTotal have to say themselves at   https://www.virustotal.com/about in the section "BAD IDEA: VirusTotal for antivirus/URL scanner testing".

Alas, I'm sure I'll be back to this topic sooner rather than later, and in appreciably more detail.

David Harley BA CISSP FBCS CITP
Director of Malware Intelligence

Author David Harley, ESET

  • Johnson

    Yes,Virustotal is not a Analysis tool,I find many people very depend on virustotal’s result to analyze if the files which they uploaded are malware,if many avs can detect them,they think the files must be malware,but in fact,I often see many avs detect normal files as malware.

    Some organizations and individuals use virustotal’s result to do some av tests,but they can’t analyze if these samples are real malware or normal files before,so I can’t trust these tests which based on virustotal.

  • http://pumpkenhead@hotmail.com Emilienne Morais Lebrun

    Nod 32,I have nod 32 since Jan. or Feb.2007 and been satisfied but now you want to sell me something else or what I have.You downloaded my version and if I have to buy it again,I hope you will do the same again because I`not too technical qhen it comes to computer or any electronique
    because I am mostly visual.So please reply and tell me what is my situation here o.k?I havealso a hotmail addres:pumpkenhead@hotmail.com,thanks for reading me.
    Emilienne Morais Lebrun

  • http://pumpkenhead@hotmail.com Emilienne Morais Lebrun

    May I have more info on buying nod 32?
    Emilienne

  • http://www.smallblue-greenworld.co.uk David

    Hello, Emilienne.

    If you go to http://www.eset.com/purchase, you should be directed from there to the appropriate web page. If you need information on installation and such, you can also try http://www.eset.com/support.

    David Harley
    Malware Intelligence Team

  • http://DidierStevens.com Didier Stevens

    You’re correct, VirusTotal is for testing files, not AV products. I interviewed Julio Canto from VirusTotal for a blogpost about this subject:

    http://blog.didierstevens.com/2008/04/21/only-x-out-of-32-antivirus-products-detect-this/

  • http://www.smallblue-greenworld.co.uk David

    Thanks for that, Didier. Good blog post. Julio Canto is a good guy for sure, very knowledgeable and unfailingly helpful.

    David Harley
    ESET Research Team

  • http://extremesecurity.blogspot.com Aa’ed Alqarta

    VirusTotal is just a small step in your virus analysis process. You shouldn’t relay 100 % on the results, because sooner or later, virus writers will figure out a way to trick virustotal to give you missleaded results.

    http://extremesecurity.blogspot.com

  • http://www.filterbit.com metasca

    I think that http://www.filterbit.com is much much faster

  • Randy Abrams

    Filterbit aslo is not a testing tool. Filterbit is much faster because it uses far less scanners and does not have the traffic that Virus total has.

    online virus scannign services are uselss for testing in terms of comparing scanners. Online sevices fail to discriminate against false positives. If I write program that says every file is infected, then my useless program will be the one that filterscan and otehr services say is the best. It really is that easy.

    Randy Abrams
    Director of Technical Education

  • http://www.filterbit.com Metascan

    Well Filterbit is faster because it is using Metascan
    I’d bet you that if Virus total would be using it will be much faster!!

  • http://www.smallblue-greenworld.co.uk David Harley

    Perhaps. (He said diplomatically.) But speed of submission to multiscanner sites isn’t the issue. The point is that multiscanner sites aren’t an appropriate way to rank scanner performance as an alternative to detection testing, because they don’t constitute a full test of a scanner’s detection ability.

Follow Us

Automatically receive new posts via email:

Delivered by FeedBurner

4 articles related to:
Hot Topic
12 Sep 2008
ESET Virus Radar

Archives

Select month
Copyright © 2014 ESET, All Rights Reserved.