Re: Hyperscan pattern matcher integration for Snort 2.9.8.2

["Viiret, Justin" <justin.viiret () intel com>]

Hi Vladimir,

Thanks for trying it out!

Depending on the pattern set, Hyperscan’s compilation process can take time – and with a  ruleset of that size, Snort 
will build many (perhaps thousands of) Hyperscan databases. We do track and try to minimize compile time; if you’re 
able to share your pattern set off-list we can take a look at it and see if there are improvements that can be made.

Hyperscan does have a straightforward serialization API, so I’m sure it would be possible to use it to cache previously 
compiled databases as you suggest. Most of the work would be in hashing the input patterns and ensuring that you load 
the correct database, plus the configuration and filesystem handling for the caching mechanism. This is too large a 
change to Snort for us to make as the Hyperscan team, though.

Best regards,
    Justin


From: Vladimir Kunschikov [mailto:kunschikov () gmail com]
Sent: Friday, June 10, 2016 6:54 PM
To: Viiret, Justin
Cc: snort-devel () lists sourceforge net
Subject: Re: [Snort-devel] Hyperscan pattern matcher integration for Snort 2.9.8.2

Very nice patch. I applied it to snort-2.9.8.0 without any problems. The throughput of the Snort improved for sure.
 Is there a way to speed up loading of the ruleset with 'search-method hyperscan' option enabled?
 My set of the rules is quite a big one - over 20 thousands of rules. Can hyperscan objects be serialized at first run 
and then loaded at subsequent runs of the Snort? Is it hard to implement such serialization by usage of the Hyperscan 
library API?

On Thu, Jun 9, 2016 at 3:57 AM, Viiret, Justin <justin.viiret () intel com<mailto:justin.viiret () intel com>> wrote:
Hi all,

Hyperscan is a high-performance regular expression matching library from Intel, released as open source software under 
a 3-clause BSD license. Although there is already some use of Hyperscan in Snort++, we at Intel have also received 
requests for a Hyperscan integration into Snort 2.9.x.

Accordingly, we have put together a patch against Snort 2.9.8.2 which adds support for using the Hyperscan library to 
accelerate some of the pattern matching tasks in Snort. This integrates Hyperscan library usage into three places:

1. A new multi-pattern literal matcher (MPSE module) called "hyperscan".
2. A faster single-pattern content string matcher, replacing the Boyer-Moore approach used by default.
3. A prefilter for PCRE, where Hyperscan is used as a prefilter check for regex options before PCRE is run. Expressions 
that are expensive to evaluate in PCRE may be avoided entirely depending on the result of the prefilter.

You can find the patch here, including a README with instructions for its use:

    https://01.org/hyperscan/downloads/hyperscan-integration-snort-2.9.8.2

You can find more information about the Hyperscan library here:

Website: https://01.org/hyperscan
Github: https://github.com/01org/hyperscan

Please get in touch if you have any feedback on the patch!

Best regards,
    Justin


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net<mailto:Snort-devel () lists sourceforge net>
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!