RE: Consumer Reports Slammed for Creating 'Test' Viruses

[Drsolly <drsollyp () drsolly com>]

On Mon, 21 Aug 2006, Larry Seltzer wrote:

> > > You missed the main argument against what CR did. The results they got
> will randomly favour whichever products are better that their choice of
> files. They didn't write viruses from scratch, they modified existing
> ones, right? 
>
> I was careful to say that I don't know if they did a good or bad job,
> just that I don't think writing your own viruses necessarily means
> you'll do a bad one.
>
> Yes, it's possible to do a bad job by writing the wrong viruses. If I
> were to test AV products with a library of existing malware I'd also be
> favoring the ones that do better with my selection. So I don't see your
> argument as one against writing viruses, just against writing the wrong
> ones.
 
Well, yes. I didn't say I was against writing viruses - I'm against bad 
product testing. It's pretty easy (if you know what you're doing) to avoid 
virus escapes.

Interesting question, though. When you've finished the tests, do you 
delete all copies of the viruses, and the source code, and the generator 
program you used, and the source code of that? Because if you do, then if 
there's any questions afterwards, you won't be able to answer them.

In the past, when I was the victim of an appallingly poor product test, I 
was able to examine the test set, to show the tester where they'd gone 
wrong.

The usual failure, was that some (in one remarkable case, ALL) of the
files were not actually viruses (or even malicious software). In another
case, the tester ran a product which did a repair on all the viruses it
found without then knowing. Unsurprisingly, all the products they ran
after that one, performed very poorly.

If you delete the test set, then such forensic examination, isn't 
possible. If you don't delete the test set, then you have the problem of 
long term secure storage (which is solvable, but isn't trivial).

Would CR be willing to subject their methodology to proper expert
examination?  Or are they 100% confident that there couldn't possibly be
any problems?

_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.