Re: Segfault building Snort3 b245 with --enable-tcmalloc flag

[Michael Altizer via Snort-devel <snort-devel () lists snort org>]

Good to hear that switching to a newer version of gperftools solved the 
problem, although that's still a little troubling.  I don't have exact 
numbers, but at least in one of the more contrived tests we were doing, 
it looked like a 7-12% reduction in processing time.

On 06/15/2018 06:19 AM, Noah Dietrich wrote:
> Thank you Michael for the deep-dive analysis of this issue. It looks 
> that installing gperftools 2.7 from github fixes the problem, although 
> I haven't done more than just run snort with the -V flag. The steps it 
> took for this are below.
> Has anyone done an analysis of the performance benefits of using 
> tcmalloc?  I'd assume that you're getting more speed for the trade-off 
> of increased memory usage, but it would be interesting if someone has 
> some results to look at.
>
> Thanks,
> Noah
>
>
> *# remove gperftools 2.5 from system:*
> sudo apt-get remove libgoogle-perftools-dev
> sudo apt-get autoremove
>
> *$ install gperftools 2.7 from github from source:*
> cd ~
> wget 
> https://github.com/gperftools/gperftools/releases/download/gperftools-2.7/gperftools-2.7.tar.gz
> tar xzvf gperftools-2.7.tar.gz
> cd gperftools-2.7/
> ./configure
> make
> sudo make install
>
> *# get, build, and install Snort3*
> cd ~/snort_src/
> git clone https://github.com/snort3/snort3.git
> cd snort3/
> ./configure_cmake.sh --prefix=/opt/snort3a4 --enable-tcmalloc
> cd build/
> make
> sudo make install
>
> *# Verify libraries linked and that snort3 runs:*
> noah@snort18x:~$ ldd /opt/snort3a4/bin/snort
> ...
> libtcmalloc.so.4 => /usr/local/lib/libtcmalloc.so.4 (0x00007fc7ce334000)
> ...
>
> noah@snort18x~$ /opt/snort3a4/bin/snort -V
>
>    ,,_     -*> Snort++ <*-
>   o"  )~   Version 3.0.0 (Build 245) from 2.9.11
>    ''''    By Martin Roesch & The Snort Team
> http://snort.org/contact#team
>            Copyright (C) 2014-2018 Cisco and/or its affiliates. All 
> rights reserved.
>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>            Using DAQ version 2.2.2
>            Using LuaJIT version 2.1.0-beta3
>            Using OpenSSL 1.1.0g  2 Nov 2017
>            Using libpcap version 1.8.1
>            Using PCRE version 8.39 2016-06-14
>            Using ZLIB version 1.2.11
>            Using FlatBuffers 1.9.0
>            Using Hyperscan version 4.7.0 2018-06-13
>            Using LZMA version 5.2.2
>
>
> On Thu, Jun 14, 2018 at 6:16 AM, Michael Altizer via Snort-devel 
> <snort-devel () lists snort org <mailto:snort-devel () lists snort org>> wrote:
>
>     I can reproduce it on my Ubuntu 18.04 system, but it doesn't look
>     like a bug in Snort.  From combing through the backtrace and
>     stepping through the code, it looks like a messy bug in the
>     interaction between tcmalloc, libhwloc, and libltdl.  libhwloc
>     uses libltdl to iterate over and load all of its plugins. 
>     Somewhere in the middle of "canonicalizing" the paths of the
>     plugins it is evaluating, libltdl's loop gets horribly confused
>     and starts writing over itself forever until get gets a SEGV.  In
>     stepping through it in a debugger, it appears that the malloc it
>     calls to create the new character array to write into returns an
>     address 16 bytes into the original path.  I didn't dig into the
>     assembly, but it might actually be a compiler issue where it
>     optimized libltdl around assumptions using builtin malloc and then
>     it gets hosed by tcmalloc doing the actual allocation.
>     Alternatively, something in what it's doing is buggily stomping on
>     tcmalloc's internal metadata, allowing it to give out the "path"
>     memory again even while it's being used and read.  Or, who knows,
>     maybe there's a bug in the gperftools 2.5 that Ubuntu is packing
>     (we tested with 2.7)?  Not that it's a great solution, but I can
>     make the crash go away by uninstalling the hwloc-plugins package.
>
>     On 06/13/2018 12:50 AM, Noah Dietrich wrote:
> >     Hello,
> >
> >     thanks for tracking that down.  I have verified that the segfault
> >     issue also occurs when building on Ubuntu 16 x64.
> >
> >     Changing the command to:
> >     *      ./configure_cmake.sh --prefix=/opt/snort --enable-tcmalloc
> >     --disable-docs*
> >     does not generate any errors and allows for successful build and
> >     installation.
> >
> >     however, running snort3 when*-enable-tcmalloc *is included as
> >     above generates a segfault;
> >        noah@snort18x:~/snort_src/snort3/build$*/opt/snort/bin/snort -V*
> >        Segmentation fault (core dumped)
> >
> >     Please let me know if any further info is needed.
> >     thanks
> >     Noah
> >
> >
> >
> >
> >     On Tue, Jun 12, 2018 at 8:32 PM, Y M via Snort-devel
> >     <snort-devel () lists snort org
> >     <mailto:snort-devel () lists snort org>> wrote:
> >
> >         After looking at the errors, which I received too, the build
> >         fails at generating the text documentation at the target
> >         prefix. Using the --disable-text-docs alone did not stop the
> >         documentation generation. However, disabling all types of
> >         documentation generation resulted in a successful build.
> >
> >         # ./configure_cmake.sh --prefix=/usr/local/snort
> >         --enable-tcmalloc --disable-text-docs --disable-html-docs
> >         --disable-pdf-docs --disable-docs
> >
> >         ...
> >         [ 99%] Built target preprocessor_states
> >         Scanning dependencies of target snort2lua
> >         [100%] Building CXX object
> >         tools/snort2lua/CMakeFiles/snort2lua.dir/init_state.cc.o
> >         [100%] Building CXX object
> >         tools/snort2lua/CMakeFiles/snort2lua.dir/snort2lua.cc.o
> >         [100%] Linking CXX executable snort
> >         [100%] Linking CXX executable snort2lua
> >         [100%] Built target snort2lua
> >         [100%] Built target snort
> >
> >         This doesn't particularly answers your question, but
> >         hopefully can help you move along.
> >
> >         Thanks.
> >         YM
> >
> >         ------------------------------------------------------------------------
> >         *From:* Snort-devel <snort-devel-bounces () lists snort org
> >         <mailto:snort-devel-bounces () lists snort org>> on behalf of Y
> >         M via Snort-devel <snort-devel () lists snort org
> >         <mailto:snort-devel () lists snort org>>
> >         *Sent:* Tuesday, June 12, 2018 8:25 PM
> >         *To:* snort-devel () lists snort org
> >         <mailto:snort-devel () lists snort org>
> >         *Subject:* Re: [Snort-devel] Segfault building Snort3 b245
> >         with --enable-tcmalloc flag
> >         Interesting. Snort build 245 with tcmalloc over CetnOS 7 and
> >         FreeBSD 11 was successful. There have been recent changes in
> >         Ubuntu 18 that are making several apps too unhappy. I will do
> >         some testing on my side and report back if I come up with
> >         anything useful.
> >
> >         YM
> >
> >         ------------------------------------------------------------------------
> >         *From:* Snort-devel <snort-devel-bounces () lists snort org
> >         <mailto:snort-devel-bounces () lists snort org>> on behalf of
> >         Noah Dietrich <noah_dietrich () 86penny org
> >         <mailto:noah_dietrich () 86penny org>>
> >         *Sent:* Tuesday, June 12, 2018 5:26 PM
> >         *To:* snort-devel () lists snort org
> >         <mailto:snort-devel () lists snort org>
> >         *Subject:* [Snort-devel] Segfault building Snort3 b245 with
> >         --enable-tcmalloc flag
> >         Hello,
> >
> >         when building Snort 3 245 with the --enable-tcmalloc flag,
> >         snort throws a segfault.
> >
> >         *Environment: *git clone of the latest Snort3 (b245) on
> >         Ubuntu 18 x64 in a VMware virtual Machine.
> >
> >         *Commands:*
> >         cd ~/snort_src
> >         git clone https://github.com/snort3/snort3.git
> >         <https://github.com/snort3/snort3.git>
> >         cd ./snort3
> >         ./configure_cmake.sh --prefix=/usr/local --enable-tcmalloc
> >         cd build
> >         make
> >
> >         *Error:*
> >         ...
> >         Scanning dependencies of target all_built_sources
> >         [ 97%] Documents: building commands.txt with
> >         Segmentation fault (core dumped)
> >         doc/CMakeFiles/all_built_sources.dir/build.make:80: recipe
> >         for target 'doc/commands.txt' failed
> >         make[2]: *** [doc/commands.txt] Error 139
> >         make[2]: *** Deleting file 'doc/commands.txt'
> >         CMakeFiles/Makefile2:6091: recipe for target
> >         'doc/CMakeFiles/all_built_sources.dir/all' failed
> >         make[1]: *** [doc/CMakeFiles/all_built_sources.dir/all] Error 2
> >         Makefile:151: recipe for target 'all' failed
> >         make: *** [all] Error 2
> >
> >
> >
> >         When you remove the --enable-tcmalloc flag, snort compiles
> >         and runs without error:
> >
> >            ,,_     -*> Snort++ <*-
> >           o"  )~   Version 3.0.0 (Build 245) from 2.9.11
> >            ''''    By Martin Roesch & The Snort Team
> >         http://snort.org/contact#team
> >                    Copyright (C) 2014-2018 Cisco and/or its
> >         affiliates. All rights reserved.
> >                    Copyright (C) 1998-2013 Sourcefire, Inc., et al.
> >                    Using DAQ version 2.2.2
> >                    Using LuaJIT version 2.1.0-beta3
> >                    Using OpenSSL 1.1.0g  2 Nov 2017
> >                    Using libpcap version 1.8.1
> >                    Using PCRE version 8.39 2016-06-14
> >                    Using ZLIB version 1.2.11
> >                    Using FlatBuffers 1.9.0
> >                    Using Hyperscan version 4.7.0 2018-06-12
> >                    Using LZMA version 5.2.2
> >
> >
> >         please let me know if you need any other information.
> >         thank you,
> >         Noah
> >
> >
> >
> >         _______________________________________________
> >         Snort-devel mailing list
> >         Snort-devel () lists snort org <mailto:Snort-devel () lists snort org>
> >         https://lists.snort.org/mailman/listinfo/snort-devel
> >         <https://lists.snort.org/mailman/listinfo/snort-devel>
> >
> >         Please visit http://blog.snort.org for the latest news about
> >         Snort!
> >
> >
> >
> >
> >     _______________________________________________
> >     Snort-devel mailing list
> >     Snort-devel () lists snort org <mailto:Snort-devel () lists snort org>
> >     https://lists.snort.org/mailman/listinfo/snort-devel
> >     <https://lists.snort.org/mailman/listinfo/snort-devel>
> >
> >     Please visithttp://blog.snort.org  for the latest news about Snort!
>
>
>
>     _______________________________________________
>     Snort-devel mailing list
>     Snort-devel () lists snort org <mailto:Snort-devel () lists snort org>
>     https://lists.snort.org/mailman/listinfo/snort-devel
>     <https://lists.snort.org/mailman/listinfo/snort-devel>
>
>     Please visit http://blog.snort.org for the latest news about Snort!

_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!