Re: Snort3: segfault after "Inspector found in the trash is still use"

[Yehor Velykozhon via Snort-devel <snort-devel () lists snort org>]

Hello, Meridoff!

Can you still provide core dump and binary (not stripped will be the best)?
In order to share it, you can upload them to the GoogleDisk (or any other cloud storage) and send a link.

As I can see, previously was proposed to use snort with sanitizers, has it given you any additional information?
Also, what OS you use? Information about the version and process architecture can help as well.

Thanks, Yehor.

From: Snort-devel <snort-devel-bounces () lists snort org> on behalf of Meridoff via Snort-devel <snort-devel () lists 
snort org>
Reply to: Meridoff <oagvozd () gmail com>
Date: Friday, 15 October 2021 at 17:35
To: "Oleksii Shumeiko -X (oshumeik - SOFTSERVE INC at Cisco)" <oshumeik () cisco com>, "snort-devel () lists snort org" 
<snort-devel () lists snort org>
Subject: Re: [Snort-devel] Snort3: segfault after "Inspector found in the trash is still use"


CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you 
recognize the sender and know the content is safe.
Sure, will do.

Also, may be it is not connected to sigsegv , concerning the warning of snort "Inspector found in the trash is still 
use" I've found the minimal config, that lead to this warning.

It is strange, but when I remove 1 of the inspectors or for example perf_monitor or binder - this warning is absent.
Also this warning appears when I use more 1 processing threads (snort -z=2 for example).

My test:
1) Start:  /usr/bin/snort -M -c /tmp/config and then kill snort: kill -TERM 15565
2) See snort.log:
srv:/home/usr$ tail -f /var/log/snort.log
Aug 11 17:04:28 srv snort[15564]: Finished /tmp/config:
Aug 11 17:04:28 srv snort[15564]: --------------------------------------------------
Aug 11 17:04:28 srv snort[15564]: afpacket DAQ configured to passive.
Aug 11 17:04:28 srv snort[15564]: initializing daemon mode
Aug 11 17:04:28 srv snort[15564]: child process is 15565
Aug 11 17:04:28 srv snort[15565]: Commencing packet processing
Aug 11 17:04:28 srv snort[15565]: ++ [0] eth0
Aug 11 17:04:28 srv snort[15565]: ++ [1] eth1
Aug 11 17:04:28 srv snort[15565]: Chroot directory = /
Aug 11 17:04:28 srv snort[15565]: Writing PID "15565" to file "/var/run/snortpid/snort.pid"

......STOPPING:
Aug 11 17:04:34 srv snort[15565]: process
Aug 11 17:04:34 srv snort[15565]:                   signals: 1
Aug 11 17:04:34 srv snort[15565]: --------------------------------------------------
Aug 11 17:04:34 srv snort[15565]: == end of dumping stats
Aug 11 17:04:34 srv snort[15565]: --------------------------------------------------
Aug 11 17:04:34 srv snort[15565]: timing
Aug 11 17:04:34 srv snort[15565]:                   runtime: 00:00:06
Aug 11 17:04:34 srv snort[15565]:                   seconds: 6.159726
Aug 11 17:04:34 srv snort[15565]: o")~   Snort exiting
Aug 11 17:04:34 srv snort[15565]: Inspector found in the trash is still in use: 'sip'.

3) My config is attached.



пт, 15 окт. 2021 г. в 15:02, Oleksii Shumeiko -X (oshumeik - SOFTSERVE INC at Cisco) <oshumeik () cisco 
com<mailto:oshumeik () cisco com>>:
Thank you for cooperation!
Since this bug may have a broad interest, could you add snort-devel in the email-thread, please.

Adding snort-devel () lists snort org<mailto:snort-devel () lists snort org>.

Regards,
Alexey


On 13 Oct 2021, at 20:24, Meridoff <oagvozd () gmail com<mailto:oagvozd () gmail com>> wrote:

Thanks, maybe I'll try sanitizer.
Core File is big (more 1GB), I'd be happy to share, but never shared such big files. May be some free web-service for 
this , which you can use..

In any case, I've analyzed backtrace and founded such calls chain (in the order of calling):

Dce2Smb2SessionTracker::~Dce2Smb2SessionTracker()
|
V
DetectionEngine::get_current_packet()
|
V
get_switcher()

So in code we can see calling GET_CURRENT_PACKET (defined in dce_smb2.h)
in the destructor of Dce2Smb2SessionTracker.

I'll try to add a debug message to this function and then reproduce the bug.

вт, 12 окт. 2021 г. в 11:49, Oleksii Shumeiko -X (oshumeik - SOFTSERVE INC at Cisco) <oshumeik () cisco 
com<mailto:oshumeik () cisco com>>:
Hi

Another way to locate the bug is to run Snort with sanitizers enabled:
./configure_cmake.sh --enable-address-sanitizer --enable-thread-sanitizer

Thanks


On 11 Oct 2021, at 14:25, Oleksii Shumeiko -X (oshumeik - SOFTSERVE INC at Cisco) via Snort-devel <snort-devel () lists 
snort org<mailto:snort-devel () lists snort org>> wrote:

Hi, Meridoff

I've asked for a complete list of modules to see what is inspectors are used.

Anyway, core file would be very helpful, if you can share it.
Also, maybe you can run snort with -v option (verbose output) to see what the configuration is.
How many processing threads (-z option) you setup for your snort?

Regards,
Alexey



On 11 Oct 2021, at 13:04, Meridoff <oagvozd () gmail com<mailto:oagvozd () gmail com>> wrote:

Sorry, I've not correctly run snort) All errors absent (I 've run snort without ENV settled).
My modules are:
ack
active
alert_csv
alert_fast
alert_full
alert_json
alert_sfsocket
alert_shmem
alert_syslog
alert_talos
alert_unixsock
alerts
appid
appids
arp
arp_spoof
asn1
attribute_table
auth
back_orifice
base64_decode
ber_data
ber_skip
binder
bufferlen
byte_extract
byte_jump
byte_math
byte_test
cip
--
and so on..
stream
stream_file
stream_icmp
stream_ip
stream_reassemble
stream_size
stream_tcp
stream_udp
stream_user
suppress
tag
target
tcp
tcp_connector
telnet
tos
trace
ttl
udp
unified2
vlan
window
wizard

--list-plugins is OK too.



пн, 11 окт. 2021 г. в 12:41, Meridoff <oagvozd () gmail com<mailto:oagvozd () gmail com>>:
Hi, I've run --list-modules and --list-plugins:

snort3 --list-modules

and found errors:

ERROR: can't init bootstrap: [string "..."]:25: module 'ffi' not found:
        no field package.preload['ffi']
        no file './ffi.lua'
        no file '/usr/share/luajit-2.1.2/ffi.lua'
        no file '/usr/share/lua/5.1/ffi.lua'
        no file '/usr/share/lua/5.1/ffi/init.lua'
        no file '/usr/share/lua/5.1/ffi.lua'
        no file '/usr/share/lua/5.1/ffi/init.lua'
        no file '/usr/lib/lua/5.1/ffi.lua'
        no file '/usr/lib/lua/5.1/ffi/init.lua'
        no file './ffi.so'
        no file '/usr/lib/lua/5.1/ffi.so'
        no file '/usr/lib/lua/5.1/ffi.so'
        no file '/usr/lib/lua/5.1/loadall.so'

ERROR: can't init bootstrap: [string "..."]:25: module 'ffi' not found:
        no field package.preload['ffi']
        no file './ffi.lua'
        no file '/usr/share/luajit-2.1.2/ffi.lua'
        no file '/usr/share/lua/5.1/ffi.lua'
        no file '/usr/share/lua/5.1/ffi/init.lua'
        no file '/usr/share/lua/5.1/ffi.lua'
        no file '/usr/share/lua/5.1/ffi/init.lua'
        no file '/usr/lib/lua/5.1/ffi.lua'
        no file '/usr/lib/lua/5.1/ffi/init.lua'
        no file './ffi.so'
        no file '/usr/lib/lua/5.1/ffi.so'
        no file '/usr/lib/lua/5.1/ffi.so'
        no file '/usr/lib/lua/5.1/loadall.so'
Is it possible that it can lead to my crash? Of casue I'l try to fix my installation.


ср, 6 окт. 2021 г. в 17:09, Oleksii Shumeiko -X (oshumeik - SOFTSERVE INC at Cisco) <oshumeik () cisco 
com<mailto:oshumeik () cisco com>>:
Hi, Meridoff

It looks like, some inspector didn't delete all its instances from the bin, or did it incorrectly, or without respect 
to execution threads (like thread local instances).

Can you run the following commands and share their output, please:
> snort --list-modules
> snort --list-plugins

Also, can you provide the core file if it is available?

Regards,
Alexey


On 5 Oct 2021, at 19:22, Meridoff via Snort-devel <snort-devel () lists snort org<mailto:snort-devel () lists snort 
org>> wrote:

Hello, I have a snort 3.1.8.0 with config with inspector file, where a lot of (10000) rules for blocking files by SHA 
hashes.
All works fine.
But, when I've stopped snort, such messages occured:

Oct 4 15:17:00 srv snort[4850]: ** caught term signal
...
Oct 4 15:17:01 srv snort[4850]: o")~ Snort exiting
...
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'smtp'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'appid'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'port_scan'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'so_proxy'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'binder'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'ftp_client'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'file_id'.
Oct 4 15:17:02 srv snort[4850]: Inspector found in the trash is still in use: 'file_log'.

I mean "Inspector found in the trash is still use" - I haven't seen such messages before.

After this SEGFAULT occured :
Oct 4 15:17:02 srv kernel: [22911.382854] snort3[4850]: segfault at 128 ip 00000000004faa59 sp 00007ffcd023e2b8 error 4 
in snort3[446000+287000]
Oct 4 15:17:02 srv kernel: [22911.382859] Code: ff 48 89 df ff 15 47 2a 35 00 48 83 c4 10 5b c3 90 64 48 8b 04 25 68 b7 
fe ff c3 66 0f 1f 44 00 00 64 48 8b 04 25 68 b7 fe ff <48> 8b 80 28 01 00 00 c3 90 66 66 2e 0f 1f 84 00 00 00 00 00 0f 
1f

I've looked to binary code and saw that it's happened in get_switcher() function..

Can not found why, cause this function called from many-many places and in term stage too..

May be It's possible to fix it. Though I can not replay this bug. It happened only 1 time for now.

PS: please remove my previous bug-report(wrong theme: "snort2 ...") with the same text but invalid theme ("snort2" 
instead of snort3)

Thanks.

_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org<mailto:Snort-devel () lists snort org>
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org<http://blog.snort.org/> for the latest news about Snort!


_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org<mailto:Snort-devel () lists snort org>
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org<http://blog.snort.org/> for the latest news about Snort!


_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!