VirusTotal is not a Comparative Analysis Tool!

[Dead link changed 27th November 2012]

Most of us have been in Estonia for the past few days for a couple of conferences. You may hear more about that later, when Normal Service is resumed. One thing I wanted to remark on now, though (partly because it relates directly to some presentations I’ve been doing) is a spike in the use of VirusTotal as a tool for comparing detection performance. This is a topic we (and the guys at VirusTotal/Hispasec themselves, who are a really good bunch) are rather sensitive about.

I’ll probably come back to this in the near future, but the gist of the problem is this. VirusTotal is a tool many people find very useful as a shortcut to checking a possibly malicious file, but it isn’t a detection test. Most importantly, it submits the files you submit to a battery of command-line scanners. This gives you a good chance of identifying a known malicious program, but the fact that a scanner doesn’t identify a file as malware does not mean it isn’t malicious, obviously. However, if a file is identified as malicious by one group of scanners but not another, it doesn’t necessarily mean that the second group is less competent at detection, either. Scanners that use sophisticated behaviour analysis, active heuristics and so on are disadvantaged by this misuse as a comparative test tool, since there is no behaviour to analyse. Generally, command-line scanners simply look at the code passively, rather than running it in a safe environment to see what it does in practice, so products that are heavily dependent on signature detection may seem to do better than products with advanced heuristics. In the real world, however, where on-access scanning is the first line of defence for most people, the advantage tends to swing the other way.

You might want to check out what Hispasec/VirusTotal have to say themselves at in the section “BAD IDEA: VirusTotal for antivirus/URL scanner testing”.

Alas, I’m sure I’ll be back to this topic sooner rather than later, and in appreciably more detail.

David Harley 
Director of Malware Intelligence

Author David Harley, ESET

  • Johnson

    Yes,Virustotal is not a Analysis tool,I find many people very depend on virustotal’s result to analyze if the files which they uploaded are malware,if many avs can detect them,they think the files must be malware,but in fact,I often see many avs detect normal files as malware.

    Some organizations and individuals use virustotal’s result to do some av tests,but they can’t analyze if these samples are real malware or normal files before,so I can’t trust these tests which based on virustotal.

  • Nod 32,I have nod 32 since Jan. or Feb.2007 and been satisfied but now you want to sell me something else or what I have.You downloaded my version and if I have to buy it again,I hope you will do the same again because I`not too technical qhen it comes to computer or any electronique
    because I am mostly visual.So please reply and tell me what is my situation here o.k?I havealso a hotmail,thanks for reading me.
    Emilienne Morais Lebrun

  • May I have more info on buying nod 32?

  • Hello, Emilienne.

    If you go to, you should be directed from there to the appropriate web page. If you need information on installation and such, you can also try

    David Harley
    Malware Intelligence Team

  • You’re correct, VirusTotal is for testing files, not AV products. I interviewed Julio Canto from VirusTotal for a blogpost about this subject:

  • Thanks for that, Didier. Good blog post. Julio Canto is a good guy for sure, very knowledgeable and unfailingly helpful.

    David Harley
    ESET Research Team

  • VirusTotal is just a small step in your virus analysis process. You shouldn’t relay 100 % on the results, because sooner or later, virus writers will figure out a way to trick virustotal to give you missleaded results.

  • I think that is much much faster

  • Randy Abrams

    Filterbit aslo is not a testing tool. Filterbit is much faster because it uses far less scanners and does not have the traffic that Virus total has.

    online virus scannign services are uselss for testing in terms of comparing scanners. Online sevices fail to discriminate against false positives. If I write program that says every file is infected, then my useless program will be the one that filterscan and otehr services say is the best. It really is that easy.

    Randy Abrams
    Director of Technical Education

  • Well Filterbit is faster because it is using Metascan
    I’d bet you that if Virus total would be using it will be much faster!!

  • Perhaps. (He said diplomatically.) But speed of submission to multiscanner sites isn’t the issue. The point is that multiscanner sites aren’t an appropriate way to rank scanner performance as an alternative to detection testing, because they don’t constitute a full test of a scanner’s detection ability.

Follow us

Copyright © 2017 ESET, All Rights Reserved.