VirusTotal is not a Comparative Analysis Tool!

VirusTotal is not a Comparative Analysis Tool!

VirusTotal is a tool many people find very useful as a shortcut to checking a possibly malicious file, but it isn't a detection test

VirusTotal is a tool many people find very useful as a shortcut to checking a possibly malicious file, but it isn’t a detection test

[Dead link changed 27th November 2012]

Most of us have been in Estonia for the past few days for a couple of conferences. You may hear more about that later, when Normal Service is resumed. One thing I wanted to remark on now, though (partly because it relates directly to some presentations I’ve been doing) is a spike in the use of VirusTotal as a tool for comparing detection performance. This is a topic we (and the guys at VirusTotal/Hispasec themselves, who are a really good bunch) are rather sensitive about.

I’ll probably come back to this in the near future, but the gist of the problem is this. VirusTotal is a tool many people find very useful as a shortcut to checking a possibly malicious file, but it isn’t a detection test. Most importantly, it submits the files you submit to a battery of command-line scanners. This gives you a good chance of identifying a known malicious program, but the fact that a scanner doesn’t identify a file as malware does not mean it isn’t malicious, obviously. However, if a file is identified as malicious by one group of scanners but not another, it doesn’t necessarily mean that the second group is less competent at detection, either. Scanners that use sophisticated behaviour analysis, active heuristics and so on are disadvantaged by this misuse as a comparative test tool, since there is no behaviour to analyse. Generally, command-line scanners simply look at the code passively, rather than running it in a safe environment to see what it does in practice, so products that are heavily dependent on signature detection may seem to do better than products with advanced heuristics. In the real world, however, where on-access scanning is the first line of defence for most people, the advantage tends to swing the other way.

You might want to check out what Hispasec/VirusTotal have to say themselves at   https://www.virustotal.com/about in the section “BAD IDEA: VirusTotal for antivirus/URL scanner testing”.

Alas, I’m sure I’ll be back to this topic sooner rather than later, and in appreciably more detail.

David Harley 
Director of Malware Intelligence

Discussion