Re: Consumer Reports Slammed for Creating 'Test' Viruses

[Drsolly <drsollyp () drsolly com>]

On Thu, 17 Aug 2006, Blue Boar wrote:

> Drsolly wrote:
> > No, it's one of the worst ways, about on a par with throwing dice.
>
> If I were to write a new virus, I'm pretty confident that I could 
> accurately predict the results of throwing it at 30 virus scanners.

I'm pretty confident that you couldn't. But anyway that doesn't actually 
tell anyone about how likely they are to detect a new virus using any of 
those scanners, because it isn't a stochastic process.
 
> For the occasion claim that some AV package can detect new unknown 
> viruses, or that some hueristic package can do so, creating a new virus 
> in lab conditions is certainly a valid test.  It's a crap shoot because 
> that's how (in)effective AV is at spotting new things, not because the 
> test is invalid.

Yes, I agree that current AV products are a crap shoot. 
 
> > I agree - the only test method that comes anywhere near being able to 
> > work, is to run a three-month-old product against the current crop of 
> > viruses (and even that isn't as easy as it sounds).
>
> OK, so if I write a virus today and test today's signature files... it's 
> not a valid test. 

Correct.

> However, if I save today's signature files, let 
> *other people* volunteer to write a bunch of viruses, and then test 
> those, it is.

Yes, you've got it.
 
> You're not arguing against the validity of the test method, you're

No, I'm arguing that the test method is about as valid as Trial by Combat.

 
> saying that you don't want additional viruses being created, because you 
> don't like it.
>
> I'm not saying you have to like it.

No, I'm saying that there's an Intelligent Designer behind the viruses, 
and your purpose isn't the purpose of the virus authors, and you would 
design different viruses from the ones they would design.

I have been on the wrong end of so many severely faulty AV product tests 
done by people who dn'e really understand what they're doing, I became 
very cynical about all product tests (I think that other products don't 
get sensibly tested either). Iused to do product testing for magazines - 
the products that I tested ranged from not very good to downright 
dangerous. I remember testing ten backup products, of which three coupld 
actually do backups, and three more could do a backup of my test computer, 
but then couldn't do a restore.

_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.