Re: Multi Pattern Search Engine Plugin

[Vlad Ulmeanu via Snort-devel <snort-devel () lists snort org>]

Hi, back with some bugs:

* lowmem seems to treat every pattern as if they have `nocase == true`. All
calls to `KTriePrefixMatch` pass `Tnocase` as the useful parameter. Is this
intended?

* For the following text (in `uint8_t` format):

```
(n = 24)
T = 0 0 0 0 243 127 95 75 189 112 255 71 180 46 93 169 167 197 0 248 21 0 0
0
```

Tested against the following dictionary <https://pastebin.com/raET1dJR>
(originally `63` entries, only `50` are distinct (considering the pattern
array and the `nocase`, `negated` tags)):

My solution
<https://github.com/vlad-ulmeanu01/ExpoSizeStringSearch/blob/main/snort_benchmark/snort3_extra/src/search_engines/bruteforce/bruteforce.cc>
finds only one match: `pat = 0 0 0 0`, with the first index that doesn't
match being `4` (counting from `0`).

Lowmem
<https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/sfksearch.cc#L579>
finds two matches:

```
LOWMEM index 4
LOWMEM: user 0x55a5a90e9540, pattern len 4, index 4, nocase 1, negative 0,
n 20. P nocase = 0 0 0 0
LOWMEM index 25
LOWMEM: user 0x55a5a90e9540, pattern len 4, index 25, nocase 1, negative 0,
n 17. P nocase = 0 0 0 0
search finished: matches = 2
```

The first one is the same as mine. The second one matches the same pattern,
but ends outside of `T`, as if it also compares an extra `\0`? I do not
know why the value of `n` is so high for the second match, I believe it
should have been `-1`. `index + (remaining) n` should be equal to the
original value of `n`, `24`.

Is this behaviour correct? How can it be explained?

Also, is there any faster way to communicate?

Thank you.

În vin., 15 mar. 2024 la 14:11, Vlad Ulmeanu <vlad.ulmeanu01 () gmail com> a
scris:

> Thank you very much for your answer!
>
> I called build_tree for each pattern, and passed the current pattern's
> associated tree object to the match function. The unit test passes now.
>
> I have some follow-up questions:
> * What are the literal, multi_match and flags variables from the
> PatternDescriptor
> <https://github.com/snort3/snort3/blob/bd6cbf1bbd3dcad9cd09261786b664d819357d94/src/framework/mpse.h#L65>
> supposed to do? lowmem ignores them. Does literal mean one byte
> characters? Is the order in which patterns must be tried to be matched
> conditioned by multi_match?
>
> * Locally, snort spawns 18 instances of LowmemMpse
> <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/lowmem.cc#L42>.
> It calls add_pattern(s), then prep_patterns and get_pattern_count on all
> of them, but only calls _search on a single one. Is this related to the
> first paragraph in here
> <https://github.com/snort3/snort3/blob/master/src/detection/dev_notes.txt>
> (creating an MPSE instance for each ...)? No further _search calls are
> made if the first one doesn't return a match.
>
> * Am I supposed to interact with int *current_state from _search?
>
> Thank you again!
>
> În joi, 14 mar. 2024 la 13:48, Russ Combs (rucombs) <rucombs () cisco com> a
> scris:
>
> > Vlad,
> >
> > rule_tree_queue is the only implementation of MpseMatch, which is a call
> > back provided so that your MPSE can report matches to the detection engine.
> > It is not a side-effect and it is not intended to be overridden.
> >
> > If you break in rule_tree_queue, you will see that KTriePrefixMatch is
> > calling rule_tree_queue on the match via the match callback which was set
> > in fp_partial (called by fp_full).
> >
> > There is scant documentation on this so we will improve that.
> >
> > Also, that demo is a little off. The README says it is high performance,
> > but lowmem is decidedly low performance. It is also throws an unrelated
> > 119:43 for no good reason so ignore that. 1:1 is the one you are after.
> >
> > Hope that helps.
> > Russ
> >
> >
> > ------------------------------
> > *From:* Snort-devel <snort-devel-bounces () lists snort org> on behalf of
> > Vlad Ulmeanu via Snort-devel <snort-devel () lists snort org>
> > *Sent:* Sunday, March 10, 2024 4:57 AM
> > *To:* snort-devel () lists snort org <snort-devel () lists snort org>
> > *Subject:* [Snort-devel] Multi Pattern Search Engine Plugin
> >
> > Hi all,
> >
> > I'm trying to plug in my Multi Pattern Search Engine
> > <https://github.com/vlad-ulmeanu01/ExpoSizeStringSearch> into snort3 and
> > run some benchmarks. I have run into some problems
> > <https://stackoverflow.com/questions/78121441/snort3-where-is-the-default-implementation-for-mpsematch> with
> > the setup: tried to rewrite the lowmem
> > <https://github.com/snort3/snort3_extra/tree/master/src/search_engines/lowmem> example
> > in snort3_extra <https://github.com/snort3/snort3_extra>, but there is a
> > side effect occurring in lowmem's _search
> > <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/lowmem.cc#L65C9-L65C16>
> > function (that triggers another "allow
> > <https://github.com/snort3/snort3_demo/blob/3fdada8224f8ec5ecea4649fdad144edec7a9c9e/tests/search_engines/ac_bnfa/expected#L2>"
> > in the snort3_demo
> > <https://github.com/snort3/snort3_demo/tree/master/tests/search_engines/ac_bnfa>
> >  example
> > <https://github.com/snort3/snort3_demo/tree/master/tests/search_engines/ac_bnfa>)
> > when calling match
> > <https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/sfksearch.cc#L579>
> >  (MpseMatch
> > <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/search_engines/search_common.h#L39>
> >  -> rule_tree_queue
> > <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L865> (I
> > suppose this is the default implementation of MpseMatch that lowmem ends
> > up using) -> MpseStash::push
> > <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L773>
> > -> MpseStash::process
> > <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L832>
> >  -> rule_tree_match
> > <https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L375>).
> > Unfortunately, things get quite complicated, and I couldn't pinpoint the
> > reason for the side effect.
> >
> > How can I deal with this side effect? I assume that I should call match
> > with a non-nullptr argument for tree, but I don't really understand its
> > meaning. Also, where can I find a good documentation source for
> > snort3_extra? The best I could find is this
> > <https://fossies.org/dox/snort3_extra-3.1.78.0/classLowmemMpse.html>.
> >
> > Thank you,
> > Vlad Ulmeanu
_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!