Re: Advice testing IDS products

["Bridges, Robert A." <0000008d8011d045-dmarc-request () LISTSERV EDUCAUSE EDU>]

Thanks.
bb



Robert A. Bridges, PhD, Cyber Security Research Mathematician, Oak Ridge National Laboratory

From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU> on behalf of Tuukka Vainio 
<tuukka () UTU FI>
Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU>
Date: Monday, February 18, 2019 at 12:21 PM
To: "SECURITY () LISTSERV EDUCAUSE EDU" <SECURITY () LISTSERV EDUCAUSE EDU>
Subject: Re: [SECURITY] Advice testing IDS products

I did not simulate any attacks. I would rely more on professional 3rd party testing (à la ICSA Labs) than do that 
myself, and then see how they fare in a real environment where not everything is an attack, just to see if the 
solutions find stuff only when set to paranoid settings that aren’t usable with real workloads.

--
Tuukka Vainio
University of Turku

From: Bridges, Robert A. [mailto:0000008d8011d045-dmarc-request () LISTSERV EDUCAUSE EDU]
Sent: perjantai 15. helmikuuta 2019 17.31
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: Re: [SECURITY] Advice testing IDS products

Tuukka, thanks for the info. How did you set up mock-malicious data breaches?



Robert A. Bridges, PhD, Cyber Security Research Mathematician, Oak Ridge National Laboratory

From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>> on behalf of Tuukka Vainio <tuukka () UTU FI<mailto:tuukka () UTU FI>>
Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>>
Date: Friday, February 15, 2019 at 1:42 AM
To: "SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>" <SECURITY () LISTSERV EDUCAUSE 
EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>>
Subject: Re: [SECURITY] Advice testing IDS products

Hi Bobby,

I tested three different big name breach detection systems (BDS) in 2014 by mirroring (not in-line) the same production 
traffic from our Internet border to all three devices simultaneously. This way I could see their differences in their 
detections and also how much they get false positives.

IIRC there was maybe 2-3 detections that at least two noticed but otherwise they all alerted about different things. 
And one of them was really bad with false positives.

--
Tuukka Vainio
University of Turku

Bridges, Robert A. <0000008d8011d045-dmarc-request () listserv educause edu<mailto:0000008d8011d045-dmarc-request () 
listserv educause edu>> kirjoitti 14.2.2019 kello 20.53:
Hi,
We’re in the process of designing a test of some IDS products and are looking for any advice from those who have 
designed and run experiments of new tools. We are looking into IDS tools that complement traditional 
(signature/rule-based) IDS that is they should complement host-based AV, firewalls, tools like Snort). Some of these 
tools sit at the network level and perform file extraction / packet reconstruction to provide detection of malware or 
of fileless malware. Some rely on host agents.

Some questions:
·         We’ll need both benign files and malicious files and fileless malware. Are there good, up-to-date libraries 
anyone recommends?
·         We’ll need network traffic sending the files—are there prescripted attack scenarios we can use or should we 
work from scratch?
·         We are trying to test the ability of these tools to get never-before-seen malware. Here’s some ideas for 
obtaining such files
o   Augment malware (manually create variants)
o   Configure the tool, but wait a few months w/o updating it and use malware that was discovered in that time period
o   Time travel (😝)
Are there other good solutions?

Any general advice for testing these tools is recommended!

Thanks,
Bobby


Robert A. Bridges, PhD, Cyber Security Research Mathematician, Oak Ridge National Laboratory