Advice testing IDS products

["Bridges, Robert A." <0000008d8011d045-dmarc-request () LISTSERV EDUCAUSE EDU>]

Hi,
We’re in the process of designing a test of some IDS products and are looking for any advice from those who have 
designed and run experiments of new tools. We are looking into IDS tools that complement traditional 
(signature/rule-based) IDS that is they should complement host-based AV, firewalls, tools like Snort). Some of these 
tools sit at the network level and perform file extraction / packet reconstruction to provide detection of malware or 
of fileless malware. Some rely on host agents.

Some questions:
·         We’ll need both benign files and malicious files and fileless malware. Are there good, up-to-date libraries 
anyone recommends?
·         We’ll need network traffic sending the files—are there prescripted attack scenarios we can use or should we 
work from scratch?
·         We are trying to test the ability of these tools to get never-before-seen malware. Here’s some ideas for 
obtaining such files
o   Augment malware (manually create variants)
o   Configure the tool, but wait a few months w/o updating it and use malware that was discovered in that time period
o   Time travel (😝)
Are there other good solutions?

Any general advice for testing these tools is recommended!

Thanks,
Bobby


Robert A. Bridges, PhD, Cyber Security Research Mathematician, Oak Ridge National Laboratory