Snort mailing list archives
Re: Multi Pattern Search Engine Plugin
From: Vlad Ulmeanu via Snort-devel <snort-devel () lists snort org>
Date: Wed, 10 Apr 2024 22:38:16 +0300
Hi, back with some bugs: * lowmem seems to treat every pattern as if they have `nocase == true`. All calls to `KTriePrefixMatch` pass `Tnocase` as the useful parameter. Is this intended? * For the following text (in `uint8_t` format): ``` (n = 24) T = 0 0 0 0 243 127 95 75 189 112 255 71 180 46 93 169 167 197 0 248 21 0 0 0 ``` Tested against the following dictionary <https://pastebin.com/raET1dJR> (originally `63` entries, only `50` are distinct (considering the pattern array and the `nocase`, `negated` tags)): My solution <https://github.com/vlad-ulmeanu01/ExpoSizeStringSearch/blob/main/snort_benchmark/snort3_extra/src/search_engines/bruteforce/bruteforce.cc> finds only one match: `pat = 0 0 0 0`, with the first index that doesn't match being `4` (counting from `0`). Lowmem <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/sfksearch.cc#L579> finds two matches: ``` LOWMEM index 4 LOWMEM: user 0x55a5a90e9540, pattern len 4, index 4, nocase 1, negative 0, n 20. P nocase = 0 0 0 0 LOWMEM index 25 LOWMEM: user 0x55a5a90e9540, pattern len 4, index 25, nocase 1, negative 0, n 17. P nocase = 0 0 0 0 search finished: matches = 2 ``` The first one is the same as mine. The second one matches the same pattern, but ends outside of `T`, as if it also compares an extra `\0`? I do not know why the value of `n` is so high for the second match, I believe it should have been `-1`. `index + (remaining) n` should be equal to the original value of `n`, `24`. Is this behaviour correct? How can it be explained? Also, is there any faster way to communicate? Thank you. În vin., 15 mar. 2024 la 14:11, Vlad Ulmeanu <vlad.ulmeanu01 () gmail com> a scris:
Thank you very much for your answer! I called build_tree for each pattern, and passed the current pattern's associated tree object to the match function. The unit test passes now. I have some follow-up questions: * What are the literal, multi_match and flags variables from the PatternDescriptor <https://github.com/snort3/snort3/blob/bd6cbf1bbd3dcad9cd09261786b664d819357d94/src/framework/mpse.h#L65> supposed to do? lowmem ignores them. Does literal mean one byte characters? Is the order in which patterns must be tried to be matched conditioned by multi_match? * Locally, snort spawns 18 instances of LowmemMpse <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/lowmem.cc#L42>. It calls add_pattern(s), then prep_patterns and get_pattern_count on all of them, but only calls _search on a single one. Is this related to the first paragraph in here <https://github.com/snort3/snort3/blob/master/src/detection/dev_notes.txt> (creating an MPSE instance for each ...)? No further _search calls are made if the first one doesn't return a match. * Am I supposed to interact with int *current_state from _search? Thank you again! În joi, 14 mar. 2024 la 13:48, Russ Combs (rucombs) <rucombs () cisco com> a scris:Vlad, rule_tree_queue is the only implementation of MpseMatch, which is a call back provided so that your MPSE can report matches to the detection engine. It is not a side-effect and it is not intended to be overridden. If you break in rule_tree_queue, you will see that KTriePrefixMatch is calling rule_tree_queue on the match via the match callback which was set in fp_partial (called by fp_full). There is scant documentation on this so we will improve that. Also, that demo is a little off. The README says it is high performance, but lowmem is decidedly low performance. It is also throws an unrelated 119:43 for no good reason so ignore that. 1:1 is the one you are after. Hope that helps. Russ ------------------------------ *From:* Snort-devel <snort-devel-bounces () lists snort org> on behalf of Vlad Ulmeanu via Snort-devel <snort-devel () lists snort org> *Sent:* Sunday, March 10, 2024 4:57 AM *To:* snort-devel () lists snort org <snort-devel () lists snort org> *Subject:* [Snort-devel] Multi Pattern Search Engine Plugin Hi all, I'm trying to plug in my Multi Pattern Search Engine <https://github.com/vlad-ulmeanu01/ExpoSizeStringSearch> into snort3 and run some benchmarks. I have run into some problems <https://stackoverflow.com/questions/78121441/snort3-where-is-the-default-implementation-for-mpsematch> with the setup: tried to rewrite the lowmem <https://github.com/snort3/snort3_extra/tree/master/src/search_engines/lowmem> example in snort3_extra <https://github.com/snort3/snort3_extra>, but there is a side effect occurring in lowmem's _search <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/lowmem.cc#L65C9-L65C16> function (that triggers another "allow <https://github.com/snort3/snort3_demo/blob/3fdada8224f8ec5ecea4649fdad144edec7a9c9e/tests/search_engines/ac_bnfa/expected#L2>" in the snort3_demo <https://github.com/snort3/snort3_demo/tree/master/tests/search_engines/ac_bnfa> example <https://github.com/snort3/snort3_demo/tree/master/tests/search_engines/ac_bnfa>) when calling match <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/sfksearch.cc#L579> (MpseMatch <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/search_engines/search_common.h#L39> -> rule_tree_queue <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L865> (I suppose this is the default implementation of MpseMatch that lowmem ends up using) -> MpseStash::push <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L773> -> MpseStash::process <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L832> -> rule_tree_match <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L375>). Unfortunately, things get quite complicated, and I couldn't pinpoint the reason for the side effect. How can I deal with this side effect? I assume that I should call match with a non-nullptr argument for tree, but I don't really understand its meaning. Also, where can I find a good documentation source for snort3_extra? The best I could find is this <https://fossies.org/dox/snort3_extra-3.1.78.0/classLowmemMpse.html>. Thank you, Vlad Ulmeanu
_______________________________________________ Snort-devel mailing list Snort-devel () lists snort org https://lists.snort.org/mailman/listinfo/snort-devel Please visit http://blog.snort.org for the latest news about Snort!
Current thread:
- Re: Multi Pattern Search Engine Plugin Vlad Ulmeanu via Snort-devel (Apr 11)
- Re: Multi Pattern Search Engine Plugin Russ Combs (rucombs) via Snort-devel (Apr 11)
- Re: Multi Pattern Search Engine Plugin Vlad Ulmeanu via Snort-devel (Apr 12)
- Re: Multi Pattern Search Engine Plugin Russ Combs (rucombs) via Snort-devel (Apr 19)
- Re: Multi Pattern Search Engine Plugin Vlad Ulmeanu via Snort-devel (Apr 29)
- Re: Multi Pattern Search Engine Plugin Vlad Ulmeanu via Snort-devel (Apr 12)
- Re: Multi Pattern Search Engine Plugin Russ Combs (rucombs) via Snort-devel (Apr 11)