Re: Multi Pattern Search Engine Plugin

["Russ Combs \(rucombs\) via Snort-devel" <snort-devel () lists snort org>]

Hey Vlad,

Sounds like you are making progress.

lowmem is caseless which helps reduce memory. The exact match is checked during signature evaluation unless the content 
is nocase.

ac_bnfa and ac_full are also caseless. The hyperscan MPSE is case sensitive. Your algorithm can be either.

I'm not able to reproduce the match off the end of the buffer. Is it possible that your input includes a null 
terminator with a length of 25? If you want to send a pcap and config I'll take a look.

This list or a github issue are the best way to get support.

Russ

________________________________
From: Vlad Ulmeanu <vlad.ulmeanu01 () gmail com>
Sent: Wednesday, April 10, 2024 3:38 PM
To: Russ Combs (rucombs) <rucombs () cisco com>
Cc: snort-devel () lists snort org <snort-devel () lists snort org>
Subject: Re: [Snort-devel] Multi Pattern Search Engine Plugin

Hi, back with some bugs:

* lowmem seems to treat every pattern as if they have `nocase == true`. All calls to `KTriePrefixMatch` pass `Tnocase` 
as the useful parameter. Is this intended?

* For the following text (in `uint8_t` format):

```
(n = 24)
T = 0 0 0 0 243 127 95 75 189 112 255 71 180 46 93 169 167 197 0 248 21 0 0 0
```

Tested against the following dictionary<https://pastebin.com/raET1dJR> (originally `63` entries, only `50` are distinct 
(considering the pattern array and the `nocase`, `negated` tags)):

My 
solution<https://github.com/vlad-ulmeanu01/ExpoSizeStringSearch/blob/main/snort_benchmark/snort3_extra/src/search_engines/bruteforce/bruteforce.cc>
 finds only one match: `pat = 0 0 0 0`, with the first index that doesn't match being `4` (counting from `0`).

Lowmem<https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/sfksearch.cc#L579>
 finds two matches:

```
LOWMEM index 4
LOWMEM: user 0x55a5a90e9540, pattern len 4, index 4, nocase 1, negative 0, n 20. P nocase = 0 0 0 0
LOWMEM index 25
LOWMEM: user 0x55a5a90e9540, pattern len 4, index 25, nocase 1, negative 0, n 17. P nocase = 0 0 0 0
search finished: matches = 2
```

The first one is the same as mine. The second one matches the same pattern, but ends outside of `T`, as if it also 
compares an extra `\0`? I do not know why the value of `n` is so high for the second match, I believe it should have 
been `-1`. `index + (remaining) n` should be equal to the original value of `n`, `24`.

Is this behaviour correct? How can it be explained?

Also, is there any faster way to communicate?

Thank you.

În vin., 15 mar. 2024 la 14:11, Vlad Ulmeanu <vlad.ulmeanu01 () gmail com<mailto:vlad.ulmeanu01 () gmail com>> a scris:
Thank you very much for your answer!

I called build_tree for each pattern, and passed the current pattern's associated tree object to the match function. 
The unit test passes now.

I have some follow-up questions:
* What are the literal, multi_match and flags variables from the 
PatternDescriptor<https://github.com/snort3/snort3/blob/bd6cbf1bbd3dcad9cd09261786b664d819357d94/src/framework/mpse.h#L65>
 supposed to do? lowmem ignores them. Does literal mean one byte characters? Is the order in which patterns must be 
tried to be matched conditioned by multi_match?

* Locally, snort spawns 18 instances of 
LowmemMpse<https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/lowmem.cc#L42>.
 It calls add_pattern(s), then prep_patterns and get_pattern_count on all of them, but only calls _search on a single 
one. Is this related to the first paragraph in 
here<https://github.com/snort3/snort3/blob/master/src/detection/dev_notes.txt> (creating an MPSE instance for each 
...)? No further _search calls are made if the first one doesn't return a match.

* Am I supposed to interact with int *current_state from _search?

Thank you again!

În joi, 14 mar. 2024 la 13:48, Russ Combs (rucombs) <rucombs () cisco com<mailto:rucombs () cisco com>> a scris:
Vlad,

rule_tree_queue is the only implementation of MpseMatch, which is a call back provided so that your MPSE can report 
matches to the detection engine. It is not a side-effect and it is not intended to be overridden.

If you break in rule_tree_queue, you will see that KTriePrefixMatch is calling rule_tree_queue on the match via the 
match callback which was set in fp_partial (called by fp_full).

There is scant documentation on this so we will improve that.

Also, that demo is a little off. The README says it is high performance, but lowmem is decidedly low performance. It is 
also throws an unrelated 119:43 for no good reason so ignore that. 1:1 is the one you are after.

Hope that helps.
Russ


________________________________
From: Snort-devel <snort-devel-bounces () lists snort org<mailto:snort-devel-bounces () lists snort org>> on behalf of 
Vlad Ulmeanu via Snort-devel <snort-devel () lists snort org<mailto:snort-devel () lists snort org>>
Sent: Sunday, March 10, 2024 4:57 AM
To: snort-devel () lists snort org<mailto:snort-devel () lists snort org> <snort-devel () lists snort 
org<mailto:snort-devel () lists snort org>>
Subject: [Snort-devel] Multi Pattern Search Engine Plugin

Hi all,

I'm trying to plug in my Multi Pattern Search Engine<https://github.com/vlad-ulmeanu01/ExpoSizeStringSearch> into 
snort3 and run some benchmarks. I have run into some 
problems<https://stackoverflow.com/questions/78121441/snort3-where-is-the-default-implementation-for-mpsematch> with 
the setup: tried to rewrite the lowmem<https://github.com/snort3/snort3_extra/tree/master/src/search_engines/lowmem> 
example in snort3_extra<https://github.com/snort3/snort3_extra>, but there is a side effect occurring in lowmem's 
_search<https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/lowmem.cc#L65C9-L65C16>
  function (that triggers another 
"allow<https://github.com/snort3/snort3_demo/blob/3fdada8224f8ec5ecea4649fdad144edec7a9c9e/tests/search_engines/ac_bnfa/expected#L2>"
 in the snort3_demo<https://github.com/snort3/snort3_demo/tree/master/tests/search_engines/ac_bnfa> 
example<https://github.com/snort3/snort3_demo/tree/master/tests/search_engines/ac_bnfa>) when calling 
match<https://github.com/snort3/snort3_extra/blob/b81e2e4f9296d9ae724e8d1b409371a3715fc2cc/src/search_engines/lowmem/sfksearch.cc#L579>
 
(MpseMatch<https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/search_engines/search_common.h#L39>
 -> 
rule_tree_queue<https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L865>
 (I suppose this is the default implementation of MpseMatch that lowmem ends up using) -> 
MpseStash::push<https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L773>
  -> 
MpseStash::process<https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L832>
 -> 
rule_tree_match<https://github.com/snort3/snort3/blob/be0977a3a8a98632e5cd1238c1d0da6dc2693b5f/src/detection/fp_detect.cc#L375>).
 Unfortunately, things get quite complicated, and I couldn't pinpoint the reason for the side effect.

How can I deal with this side effect? I assume that I should call match with a non-nullptr argument for tree, but I 
don't really understand its meaning. Also, where can I find a good documentation source for snort3_extra? The best I 
could find is this<https://fossies.org/dox/snort3_extra-3.1.78.0/classLowmemMpse.html>.

Thank you,
Vlad Ulmeanu

_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!