Snort mailing list archives

Previous By Date Next
Previous By Thread Next

Memory leak in snort 2.9 and FreeBSD 11?

From: elof () sentor se
Date: Fri, 2 Mar 2018 16:35:24 +0100 (CET)

Critical issue.
After I upgraded a few FreeBSD 10.3 machines to 11.1, snort has begun eating memory until it crashes. 
This seem to be happening on all upgraded machines, all the time.


I suspect there's a memory leak somewhere.


Example of 40 minutes after I start snort. I run:

while true
do
  ps faxuw | egrep "^USER|/[s]nort "
  echo "---"
  top | grep -B3 ^Swap
  echo "---"
  sleep 120
done

Here you see it start to consume RAM:
                    ####
USER      PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED      TIME COMMAND
snort    7337  98.4  1.4  356096 232376  -  Rs   14:40     0:01.35 snort
---
Mem: 550M Active, 174M Inact, 1585M Wired, 13G Free
ARC: 711M Total, 153M MFU, 545M MRU, 1600K Anon, 4623K Header, 7465K Other
     593M Compressed, 1647M Uncompressed, 2.78:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  54.3  7.2 3002112 1199900  -  Rs   14:40     1:04.85 snort
---
Mem: 1499M Active, 191M Inact, 1670M Wired, 12G Free
ARC: 763M Total, 178M MFU, 572M MRU, 1308K Anon, 4860K Header, 7441K Other
     646M Compressed, 1779M Uncompressed, 2.75:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  54.3 12.9 5644032 2155388  -  Ss   14:40     2:07.16 snort
---
Mem: 2427M Active, 191M Inact, 1682M Wired, 11G Free
ARC: 777M Total, 178M MFU, 585M MRU, 1344K Anon, 4935K Header, 7513K Other
     661M Compressed, 1815M Uncompressed, 2.75:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  54.5 18.7 8275712 3114844  -  Rs   14:40     3:09.86 snort
---
Mem: 3357M Active, 192M Inact, 1768M Wired, 10G Free
ARC: 821M Total, 194M MFU, 614M MRU, 556K Anon, 5195K Header, 7513K Other
     711M Compressed, 1942M Uncompressed, 2.73:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  54.2 24.3 10862336 4053456  -  Rs   14:40     4:11.28 snort
---
Mem: 4270M Active, 194M Inact, 1778M Wired, 9646M Free
ARC: 890M Total, 299M MFU, 578M MRU, 400K Anon, 5243K Header, 7442K Other
     726M Compressed, 1978M Uncompressed, 2.73:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  56.2 29.9 13461248 4998904  -  Ss   14:40     5:13.96 snort
---
Mem: 5188M Active, 195M Inact, 1798M Wired, 8708M Free
ARC: 826M Total, 261M MFU, 551M MRU, 528K Anon, 5300K Header, 7410K Other
     741M Compressed, 2015M Uncompressed, 2.72:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  53.1 35.5 16033536 5929068  -  Rs   14:40     6:15.56 snort
---
Mem: 6091M Active, 195M Inact, 1823M Wired, 7779M Free
ARC: 870M Total, 255M MFU, 602M MRU, 276K Anon, 5391K Header, 7521K Other
     755M Compressed, 2051M Uncompressed, 2.72:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  51.2 41.1 18605824 6867124  -  Ss   14:40     7:16.95 snort
---
Mem: 7002M Active, 195M Inact, 1848M Wired, 6843M Free
ARC: 885M Total, 221M MFU, 651M MRU, 288K Anon, 5454K Header, 7515K Other
     769M Compressed, 2087M Uncompressed, 2.71:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    7337  54.0 46.8 21212928 7810464  -  Rs   14:40     8:19.24 snort
---
Mem: 7924M Active, 195M Inact, 1943M Wired, 5826M Free
ARC: 936M Total, 218M MFU, 703M MRU, 952K Anon, 5766K Header, 7829K Other
     824M Compressed, 2222M Uncompressed, 2.70:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 53.0 52.5 23834368 8762624 - Ss 14:40 9:21.14 snort 
---
Mem: 8849M Active, 195M Inact, 1954M Wired, 4891M Free
ARC: 951M Total, 218M MFU, 718M MRU, 920K Anon, 5827K Header, 7814K Other
     838M Compressed, 2259M Uncompressed, 2.69:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 50.9 58.2 26472192 9721948 - Rs 14:40 10:23.33 snort 
---
Mem: 9782M Active, 195M Inact, 1971M Wired, 3941M Free
ARC: 965M Total, 220M MFU, 731M MRU, 920K Anon, 5882K Header, 7822K Other
     853M Compressed, 2295M Uncompressed, 2.69:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 50.3 63.9 29105920 10676928 - Rs 14:40 11:26.69 snort 
---
Mem: 10G Active, 204M Inact, 2116M Wired, 2852M Free
ARC: 1062M Total, 274M MFU, 774M MRU, 1052K Anon, 6252K Header, 7728K Other 
     939M Compressed, 2510M Uncompressed, 2.67:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 55.5 69.6 31735552 11617828 - Rs 14:40 12:28.81 snort 
---
Mem: 11G Active, 224M Inact, 2132M Wired, 1921M Free
ARC: 1092M Total, 300M MFU, 777M MRU, 1072K Anon, 6285K Header, 7591K Other 
     954M Compressed, 2547M Uncompressed, 2.67:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 55.1 75.1 34324224 12545340 - Rs 14:40 13:30.78 snort 
---
Mem: 12G Active, 230M Inact, 2222M Wired, 931M Free
ARC: 1117M Total, 325M MFU, 777M MRU, 1204K Anon, 6542K Header, 7580K Other 
     1010M Compressed, 2686M Uncompressed, 2.66:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 55.2 80.4 36824832 13427160 - Ss 14:40 14:32.93 snort 
---
Mem: 12G Active, 263M Inact, 1428M Laundry, 1929M Wired, 367M Free
ARC: 1156M Total, 325M MFU, 816M MRU, 1068K Anon, 6614K Header, 7613K Other 
     1025M Compressed, 2722M Uncompressed, 2.66:1 Ratio
Swap: 4096M Total, 4096M Free
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 54.6 82.6 39397120 13800500 - Rs 14:40 15:35.14 snort 
---
Mem: 12G Active, 110M Inact, 1784M Laundry, 1937M Wired, 191M Free
ARC: 1159M Total, 323M MFU, 822M MRU, 936K Anon, 6648K Header, 7488K Other
     1039M Compressed, 2758M Uncompressed, 2.65:1 Ratio
Swap: 4096M Total, 685M Used, 3411M Free, 16% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 53.8 82.6 41801472 13800476 - Ss 14:40 16:35.38 snort 
---
Mem: 11G Active, 107M Inact, 1846M Laundry, 1990M Wired, 182M Free
ARC: 1208M Total, 338M MFU, 854M MRU, 1552K Anon, 6866K Header, 7564K Other 
     1087M Compressed, 2878M Uncompressed, 2.65:1 Ratio
Swap: 4096M Total, 1508M Used, 2587M Free, 36% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 54.5 82.2 44414720 13735076 - Rs 14:40 17:38.62 snort 
---
Mem: 12G Active, 34M Inact, 1766M Laundry, 2014M Wired, 292M Free
ARC: 1221M Total, 337M MFU, 868M MRU, 1436K Anon, 6944K Header, 7616K Other 
     1102M Compressed, 2914M Uncompressed, 2.64:1 Ratio
Swap: 4096M Total, 2551M Used, 1544M Free, 62% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 7337 54.6 82.1 46968576 13714884 - Ss 14:40 18:42.19 snort 
---
Mem: 11G Active, 64M Inact, 1884M Laundry, 2058M Wired, 259M Free
ARC: 1252M Total, 352M MFU, 885M MRU, 672K Anon, 7150K Header, 7525K Other
     1153M Compressed, 3043M Uncompressed, 2.64:1 Ratio
Swap: 4096M Total, 3457M Used, 639M Free, 84% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort     7337   2.2  0.0       0     16  -  R<Es 14:40    19:49.65 snort
---
Mem: 12G Active, 5252K Inact, 786M Laundry, 1937M Wired, 461M Free
ARC: 1319M Total, 384M MFU, 920M MRU, 967K Anon, 7275K Header, 7534K Other
     1181M Compressed, 3112M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 4096M Used, K Free, 100% Inuse
---
After maxing out at 82.2% or RAM for a copuple of minutes, the process is automatically killed by the system: 

Mar  2 15:17:48 chobetsu-10 kernel: swap_pager: out of swap space
Mar  2 15:17:48 chobetsu-10 kernel: swap_pager_getswapspace(11): failed
Mar 2 15:20:18 chobetsu-10 kernel: pid 7337 (snort), uid 100, was killed: out of swap space 





...the while-loop continues...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
<no snort process started>
---
Mem: 272M Active, 12M Inact, 2079M Wired, 13G Free
ARC: 1376M Total, 516M MFU, 844M MRU, 1552K Anon, 7347K Header, 7524K Other 
     1197M Compressed, 3150M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 225M Used, 3871M Free, 5% Inuse
---




I startup snort again.
...the while-loop continues...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    14277  57.3  5.1 2055936 856740  -  Rs   15:23     0:43.21 snort
---
Mem: 1090M Active, 56M Inact, 2136M Wired, 12G Free
ARC: 1330M Total, 435M MFU, 879M MRU, 1432K Anon, 7487K Header, 7824K Other 
     1213M Compressed, 3193M Uncompressed, 2.63:1 Ratio
Swap: 4096M Total, 190M Used, 3906M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    14277  57.9 11.0 4714240 1829656  -  Ss   15:23     1:49.20 snort
---
Mem: 2046M Active, 97M Inact, 2366M Wired, 11G Free
ARC: 1401M Total, 438M MFU, 945M MRU, 1296K Anon, 8041K Header, 8700K Other 
     1314M Compressed, 3438M Uncompressed, 2.62:1 Ratio
Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND 
snort    14277  54.3 16.8 7362304 2798096  -  Ss   15:23     2:54.43 snort
---
Mem: 2985M Active, 98M Inact, 2378M Wired, 10G Free
ARC: 1450M Total, 438M MFU, 994M MRU, 1468K Anon, 8120K Header, 8754K Other 
     1329M Compressed, 3475M Uncompressed, 2.62:1 Ratio
Swap: 4096M Total, 172M Used, 3924M Free, 4% Inuse
---
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND snort 14277 60.4 22.6 10053376 3776916 - Ss 15:23 4:00.53 snort 

...and so on until pid 14277 gets out of swap.



I'm running:
FreeBSD 11.1-RELEASE-p4 amd64
Snort      Version 2.9.11.1 (Build 268)
           Using libpcap version 1.8.1
           Using PCRE version: 8.40 2017-01-11
           Using ZLIB version: 1.2.11



Has anyone else observed this?

Any tips on how I can help debug this further?

A SIGHUP don't reveal anything about what subsystem is eating memory.
I've tried setting two memcaps to a really low value, to see if the process stop increasing in size: 
  preprocessor stream5_global: ......... memcap 128257751
  preprocessor http_inspect: global .... memcap 85505167
No luck. The snort process grows to >80% of system RAM and then dies.



Some info about the snort conf:

Running in IDS mode
Detection: Search-Method = AC-Full-Q
           Split Any/Any group = enabled
           Search-Method-Optimizations = enabled
           Maximum pattern length = 20
+-----------------------[detection-filter-config]------------------------------
           memory-cap : 1048576 bytes
+-----------------------[rate-filter-config]-----------------------------------
           memory-cap : 1048576 bytes
+-----------------------[event-filter-config]----------------------------------
           memory-cap : 1048576 bytes
Rule application order: pass->activation->dynamic->drop->alert->log->sdrop->reject
pcap DAQ configured to passive.
chroot
Set gid to 100
Set uid to 100
The same snort version was running just fine on FreeBSD 10.3 before the upgrade. 


/Elof

_______________________________________________
Snort-users mailing list
Snort-users () lists snort org
Go to this URL to change user options or unsubscribe:
https://lists.snort.org/mailman/listinfo/snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!

Please follow these rules: https://snort.org/faq/what-is-the-mailing-list-etiquette
Previous By Date Next
Previous By Thread Next

Current thread: