Everyone hates false positives (FPs). Well, nearly everyone. For purveyors of fake anti-malware products, deliberate FPs are a source of income...
However, real security vendors hate them because every false positive is a significant detection failure, even if no-one notices (it's quite possible that most FPs pass unnoticed by anyone because the circumstances under which the scanner would mis-diagnose are so unusual). If they are noticed, it can be very bad news indeed for the customer. One of the worst scenarios is where a very widely-used version of a system or application file is misdetected, because of the number of customers it can affect.
And while some would say it's richly deserved, a widely publicized FP incident is a PR disaster for a security company, at least in the short term. What's worse, it can happen to anyone, which is why (with the occasional dishonourable exception) antivirus companies don't attempt to capitalize on a competitor's misfortune, because sooner or later, it's going to happen to them too. It's pretty much inevitable, given the nature of the technology and the constant pressure to deal immediately with ridiculous volumes of malware - (many) tens of thousands of unique malicious binaries are seen by virus labs daily, and the increasing need to make detections as proactive and generic as possible.
All this is potentially frightening and inconvenient (or worse) for a home user. And if it happens in a corporate environment, it can be very, very expensive to remedy. So while some of the public comments we see in the wake of such incidents may seem over the top, "FP rage" is certainly understandable.
I've talked about VirusTotal here before: it's the best known (and arguably the best) of a number of services that can evaluate the likelihood of a given file's being malicious by submitting it to a battery of antivirus scanners to see what they report.
It seems that the guys at VT get a lot of questions about false positives nowadays. Well, that's not unreasonable: if you're using VT to get a handle on whether a given file/object is really malicious, which I presume is the primary reason most people use it, then you probably should be concerned about the possibility of FPs in a VT report. As the VT guys themselves say:
VirusTotal is a service that analyzes suspicious files and facilitates the quick detection of viruses, worms, trojans, and all kinds of malware detected by antivirus engines.
Hispasec, who provide VT as a free service, have a great deal of experience and expertise in this area, and I know they go out of their way to answer queries if they can.
But if the AV industry, with all its experience of detection and determination to avoid FPs, is unable to prevent them from happening from time to time , it seems particularly unfair to expect a third party, even a group with VT's experience, to take responsiblity for problems that should be owned by the vendors. So I was surprised to hear that from time to time VT also gets the sort of abuse and threats of litigation that AV companies sometimes get after a highly-publicised FP.
I have to wonder whether this is another instance of the VirusTotal service being misused to support conclusions on issues it wasn't really meant to address. We already see it being used inappropriately as a substitute for validation in testing and product evaluation, or as a marketing tool used by antivirus and other security companies to bash the competition, or as a tool used by other researchers for generating dubious research statistics about specific threats. Well, perhaps misuse is a little harsh: there's nothing wrong with using VT as a research tool, as long as you're aware of the limitations of the service and tailor your conclusions so that they're not misleading.
I don't know whether VT have ever considered offering a formal sample validation service that would offer, for instance, checking of possible FPs. It sounds as if some of the people who use the service might appreciate it. However, the practical difficulties of implementing such a service would be considerable: I'm not sure that VT or anyone else with the ability to do it properly would be able to do it on a large scale for free.
Symantec's Mark Kennedy has contributed a very relevant paper under the auspices of AMTSO for this year's Virus Bulletin conference. I very much look forward to reading it.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
