Re: Spam cube

[Drsolly <drsollyp () drsolly com>]

And when you've done all that ...

Then you need to include "The Perfect Antivirus" in your product tests, 
and if you find that "The Perfect Antivirus" scores highly in your tests, 
then you know that your tests are utter crap.



On Mon, 20 Mar 2006, Nick FitzGerald wrote:

> Predrag Ivanovic to Drsolly to ???:
>
> > > > percent of viruses discovered/removed? 
> > >
> > > You would not believe how difficult this one is to measure.
> >
> > IIRC,methodology used for one of the reviews was:
> > 1.put as many malware on computer as you can
>
> As Alan has already indicated, you make that sound so easy...
>
> In reality, if you were to spend 100 hours setting up and running a 
> test from scratch, never having done one before _and_ wnating to ensure 
> you did a modestly technically competent test, you'd spend the first 10-
> 20,000 of those 100 hours assembling your test-set.
>
> The devil is, as they say, in the details, and the details of 
> assembling a "good" malware test-set for an AV detection test are 
> _immense_.
>
> Yes, you can go to several web-sites and download what seem like large-
> ish collections of malware, but much of the contents of many of those 
> collections are well-known garbage files.  Some vendors protect 
> themselves from incompetent testers by adding detection of those 
> broken, non-functioning and non-malicious files, so thinking you are 
> "improving" your testing by running several "known to have hiugh 
> detection" scanners against your new collection and dropping any 
> "samples" that no, or only one, or less than three, or whatever, of 
> your "premium" scanners detect will still result in a very bad test 
> set.
>
> First, it will still contain a lot of "garbage" files that many vendors 
> refuse to detect for "ethical" reasons.
>
> Second, you have now seriously biased the test-set to favour your 
> presumption that scanners X, Y and Z "have the best detection".  (Don't 
> get me wrong -- you _can_ use the results from several "very good" 
> scanners to _help_ winnow crud from such inforamlly assembled test-
> sets, BUT to do so you have to know a lot about the history of the 
> specific scanners, the vagaries of what most consider insignificant or 
> trivial wording differences in reported deetctions, the meaning of a 
> file being detected/reported one way when scanned with default options 
> and another when scanned in "guru" and other even less-documented 
> modes, and so on. In short, you have to be very experienced in the 
> malware analysis and development field to know how to to do this at all 
> well, and then you still have the issue of analysing all the remaining 
> grey and other edge cases.)
>
> Third, you will almost certainly have no "challenging" samples.  Some 
> products are notorious for having trouble with some malware 
> (particularly polymorphic and metamorphic viruses, but there are many 
> other difficult cases).  One way for AV developers to protect 
> themselves from bad reviewers is for them to grab all these publicly 
> available "collections" and make sure that any samples of any of these 
> "problem" malwares are in their QA test-sets so anyone "fiddling with" 
> (aka "trying to fix") one of these problem detections cannot break it 
> so badly as to prevent the known, publicly available samples of that 
> malware from being detected.
>
> I could go on, but I don't have 10,000 hours to spare to point out all 
> the major gotchas you have to be aware of when contemplating doing such 
> a test from scratch...
>
> > 2.install antivirus foo,with latest updates
> > 3.scan the system
> > 4.wipe the system,reinstall from image
> > 5.put another AV on it
> > 6.repeat  
> > And at the end,calculate percentages.    
>
> This will take the remaining 2-400 hours of your allotted time, for as 
> sure as eggs are eggs, you'll find all kinds of issues, weirdness, 
> incompatibility, instability, etc, etc (and if you don't, you certainly 
> are doing a _very crap_ test or are too inattentive to detail for your 
> results to be worth writing down, let alone anyone else reading).
>
>
> Aside from having had a general to advanced technical interest in all 
> AV product testing issues for a large part of the last ~15 years, I 
> also worked in independent AV product testing for a couple of years and 
> dealt with all these things on an almost daily basis.
>
>
> Regards,
>
> Nick FitzGerald
>
> _______________________________________________
> Fun and Misc security discussion for OT posts.
> https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
> Note: funsec is a public and open mailing list.

_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.