False Positives: A Round of Applause...

False Positives: A Round of Applause…

The anti-malware industry isn’t a suitable environment for the thin-skinned. We get used to receiving “more kicks than ha’pence” (see http://www.virusbtn.com/spambulletin/archive/2006/11/vb200611-OK).. In particular, I’ve grown accustomed to the fact that many people expect all the following from an AV product: Absolute Protection Absolute Convenience Absolutely no  False Positives Absolutely no charge False positives (FPs) are

The anti-malware industry isn’t a suitable environment for the thin-skinned. We get used to receiving “more kicks than ha’pence” (see http://www.virusbtn.com/spambulletin/archive/2006/11/vb200611-OK).. In particular, I’ve grown accustomed to the fact that many people expect all the following from an AV product: Absolute Protection Absolute Convenience Absolutely no  False Positives Absolutely no charge False positives (FPs) are

The anti-malware industry isn’t a suitable environment for the thin-skinned. We get used to receiving “more kicks than ha’pence” (see http://www.virusbtn.com/spambulletin/archive/2006/11/vb200611-OK)..

In particular, I’ve grown accustomed to the fact that many people expect all the following from an AV product:

  • Absolute Protection
  • Absolute Convenience
  • Absolutely no  False Positives
  • Absolutely no charge

False positives (FPs) are not a minor issue: my experience is that many people (especially in corporate environments) are more infuriated by an FP than by a false negative (i.e. where security software fails to detect real malware).

So it was a pleasant surprise to come across a blog from a source not usually associated with fulsome praise of the AV industry that was actually rather positive, in a backhanded sort of way. The author states “…while I rant quite a bit about the AV industry, you have to give this one to them: the number of false positives is really low. For example, in the AV-Comparatives test 20 false positives is considered many, even though the collection is over 1 500 000 samples (so the acceptable FP rate is below 0.0015%!).” [Sadly, this calculation doesn’t altogether make sense, either to us or to AV-Comparatives. In fact, it seems to be based on comparing the number of “acceptable” false positives to the AV-Comparatives malware test set. It isn’t based on the AV-Comparatives clean set: AV-C don’t publish that information, as they consider FP ratios in percentages to be potentially misleading.]

The blog isn’t actually about us, I should make clear: it’s about the importance of false positives in intrusion detection, an area with just enough methodology and terminology shared with anti-malware to be confusing – for instance, both technologies talk about FPs and signatures, but often mean something slightly different by them. This led me to another blog that makes a useful distinction between “real” false positives – misdetection of innocent objects as being malicious – and “noise” such as “inappropriate traffic that is legitimately detected that we just don’t care about.” Actually, because modern anti-malware has a much wider remit than traditional virus detection, that is a concern we share with IDS and IPS vendors – of course, there’s an overlap – most AV products now include some IPS (Intrusion Protection System) functionality. But it also has an analogue in more traditional anti-malware detection, which tends to cover “noise” programs such as adware and “possibly unwanted” code as well as out-and-out malware.

Then there’s the miscellaneous detritus that may be left behind when malware is removed: it may be harmless in itself, but what happens when two security programs are used on the same system and one is more cautious about what it removes than the other? Which is “correct” is a subjective judgement rather than a right/wrong issue.

To take another common case, no-one wants to see those misidentifications of innocent system files as malicious that affect even the most cautious products from time to time, but what about files that are detected as malicious because they’re packed with a run-time packer that’s often (but not necessarily always) used by malware authors? Well, that’s not an issue I’m going to be able to give a short and simple answer to here. But I will say this (and often have before): given the complexity of the malware problem, not to mention the sheer volume of samples, the wonder is that mainstream anti-malware generates so few FPs. And it’s a real pleasure to find someone agreeing on that point who can’t be accused of undue bias in favour of this industry.

By the way, these blogs reminded me of an old but still relevant paper on “The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection” by Stefan Axelsson. If you’re interested in the false positive conundrum and  the words “Bayes Theorem” don’t strike terror to your heart, you might find it well worth reading.

David Harley
Director of Malware Intelligence

 

Discussion