funsec mailing list archives
Re: Spam cube
From: Nick FitzGerald <nick () virus-l demon co uk>
Date: Mon, 20 Mar 2006 09:12:15 +1200
Predrag Ivanovic to Drsolly to ???:
percent of viruses discovered/removed?You would not believe how difficult this one is to measure.IIRC,methodology used for one of the reviews was: 1.put as many malware on computer as you can
As Alan has already indicated, you make that sound so easy... In reality, if you were to spend 100 hours setting up and running a test from scratch, never having done one before _and_ wnating to ensure you did a modestly technically competent test, you'd spend the first 10- 20,000 of those 100 hours assembling your test-set. The devil is, as they say, in the details, and the details of assembling a "good" malware test-set for an AV detection test are _immense_. Yes, you can go to several web-sites and download what seem like large- ish collections of malware, but much of the contents of many of those collections are well-known garbage files. Some vendors protect themselves from incompetent testers by adding detection of those broken, non-functioning and non-malicious files, so thinking you are "improving" your testing by running several "known to have hiugh detection" scanners against your new collection and dropping any "samples" that no, or only one, or less than three, or whatever, of your "premium" scanners detect will still result in a very bad test set. First, it will still contain a lot of "garbage" files that many vendors refuse to detect for "ethical" reasons. Second, you have now seriously biased the test-set to favour your presumption that scanners X, Y and Z "have the best detection". (Don't get me wrong -- you _can_ use the results from several "very good" scanners to _help_ winnow crud from such inforamlly assembled test- sets, BUT to do so you have to know a lot about the history of the specific scanners, the vagaries of what most consider insignificant or trivial wording differences in reported deetctions, the meaning of a file being detected/reported one way when scanned with default options and another when scanned in "guru" and other even less-documented modes, and so on. In short, you have to be very experienced in the malware analysis and development field to know how to to do this at all well, and then you still have the issue of analysing all the remaining grey and other edge cases.) Third, you will almost certainly have no "challenging" samples. Some products are notorious for having trouble with some malware (particularly polymorphic and metamorphic viruses, but there are many other difficult cases). One way for AV developers to protect themselves from bad reviewers is for them to grab all these publicly available "collections" and make sure that any samples of any of these "problem" malwares are in their QA test-sets so anyone "fiddling with" (aka "trying to fix") one of these problem detections cannot break it so badly as to prevent the known, publicly available samples of that malware from being detected. I could go on, but I don't have 10,000 hours to spare to point out all the major gotchas you have to be aware of when contemplating doing such a test from scratch...
2.install antivirus foo,with latest updates 3.scan the system 4.wipe the system,reinstall from image 5.put another AV on it 6.repeat And at the end,calculate percentages.
This will take the remaining 2-400 hours of your allotted time, for as sure as eggs are eggs, you'll find all kinds of issues, weirdness, incompatibility, instability, etc, etc (and if you don't, you certainly are doing a _very crap_ test or are too inattentive to detail for your results to be worth writing down, let alone anyone else reading). Aside from having had a general to advanced technical interest in all AV product testing issues for a large part of the last ~15 years, I also worked in independent AV product testing for a couple of years and dealt with all these things on an almost daily basis. Regards, Nick FitzGerald _______________________________________________ Fun and Misc security discussion for OT posts. https://linuxbox.org/cgi-bin/mailman/listinfo/funsec Note: funsec is a public and open mailing list.
Current thread:
- RE: Re: The AV., (continued)
- RE: Re: The AV. Larry Seltzer (Mar 20)
- Re: Spam cube Valdis . Kletnieks (Mar 19)
- Re: Spam cube Drsolly (Mar 20)
- Re: Spam cube Valdis . Kletnieks (Mar 20)
- Re: Spam cube Drsolly (Mar 20)
- Re: Spam cube Stephen J. Smoogen (Mar 20)
- Re: Spam cube Valdis . Kletnieks (Mar 19)
- Re: Spam cube Drsolly (Mar 20)
- Re: Spam cube Valdis . Kletnieks (Mar 20)
- Re: Spam cube Drsolly (Mar 20)
- Re: Spam cube Nick FitzGerald (Mar 19)
- Re: Spam cube Drsolly (Mar 19)
- Re: Spam cube Nick FitzGerald (Mar 19)
- Re: Spam cube Predrag Ivanovic (Mar 27)
- Re: Spam cube Drsolly (Mar 27)
- Re: Spam cube Predrag Ivanovic (Mar 19)