Re: Fuzz-test.sh crashes that can't be reproduced

[Nora Sandler <nsandler () securityinnovation com>]

Hi Peter,

I did take a look at a few different core dumps. I don't *think* it's running out of memory - looking at the 
backtraces, it sometimes fails when deallocating, sometimes when allocating. And the tool's memory usage seems to stay 
pretty constant.

Here's a backtrace where it fails on allocation:

Core file '/cores/core.10792' (x86_64) was loaded.
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type 
information available.
* thread #1: tid = 0x0000, 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160, stop reason = signal 
SIGSTOP
  * frame #0: 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160
    frame #1: 0x00007fffbb152131 libsystem_malloc.dylib`malloc_zone_check + 58
    frame #2: 0x00007fffbb151e8a libsystem_malloc.dylib`internal_check + 17
    frame #3: 0x00007fffbb14231d libsystem_malloc.dylib`malloc_zone_malloc + 86
    frame #4: 0x00007fffbb1412b0 libsystem_malloc.dylib`malloc + 24
    frame #5: 0x000000010da0e746 libglib-2.0.0.dylib`g_malloc + 24
    frame #6: 0x000000010fbda70d libwireshark.0.dylib`wmem_strict_alloc(private_data=0x00007fbc0ed02bf0, size=40) + 29 
at wmem_allocator_strict.c:93 [opt]
    frame #7: 0x000000010fbd8d58 libwireshark.0.dylib`wmem_alloc0 [inlined] wmem_alloc(allocator=<unavailable>) + 27 at 
wmem_core.c:58 [opt]
    frame #8: 0x000000010fbd8d3d libwireshark.0.dylib`wmem_alloc0(allocator=<unavailable>, size=40) + 13 at 
wmem_core.c:66 [opt]
    frame #9: 0x000000010f195030 libwireshark.0.dylib`proto_register_erf [inlined] 
erf_meta_tag_info_new(allocator=<unavailable>, tag=0x00000001109f4bb0) + 13 at packet-erf.c:902 [opt]
    frame #10: 0x000000010f195023 libwireshark.0.dylib`proto_register_erf [inlined] 
init_tag_fields(hfri_table=0x00007fbc115694c0, ett_table=0x00007fbc11569590, tag=0x00000001109f4bb0) + 5 at 
packet-erf.c:1008 [opt]
    frame #11: 0x000000010f19501e libwireshark.0.dylib`proto_register_erf [inlined] init_meta_tags + 148 at 
packet-erf.c:1088 [opt]
    frame #12: 0x000000010f194f8a libwireshark.0.dylib`proto_register_erf + 362 at packet-erf.c:3269 [opt]
    frame #13: 0x000000010fb3ac2d libwireshark.0.dylib`register_all_protocols(cb=0x0000000000000000, 
cb_data=0x0000000000000000) + 33757 at register.c:2243 [opt]
    frame #14: 0x000000010eea4cd4 
libwireshark.0.dylib`proto_init(register_all_protocols_func=(libwireshark.0.dylib`register_all_protocols at 
register.c:1465), register_all_handoffs_func=(libwireshark.0.dylib`register_all_protocol_handoffs at register.c:5689), 
cb=0x0000000000000000, client_data=0x0000000000000000) + 404 at proto.c:534 [opt]
    frame #15: 0x000000010ee87abf 
libwireshark.0.dylib`epan_init(register_all_protocols_func=(libwireshark.0.dylib`register_all_protocols at 
register.c:1465), register_all_handoffs_func=(libwireshark.0.dylib`register_all_protocol_handoffs at register.c:5689), 
cb=0x0000000000000000, client_data=0x0000000000000000) + 239 at epan.c:166 [opt]
    frame #16: 0x000000010d67b352 tshark`main(argc=<unavailable>, argv=<unavailable>) + 994 at tshark.c:902 [opt]
    frame #17: 0x00007fffbafc0255 libdyld.dylib`start + 1
    frame #18: 0x00007fffbafc0255 libdyld.dylib`start + 1

And one where it's failing on free:

Core file '/cores/core.10458' (x86_64) was loaded.
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type 
information available.
* thread #1: tid = 0x0000, 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160, stop reason = signal 
SIGSTOP
  * frame #0: 0x00007fffbb14e6ab libsystem_malloc.dylib`szone_check_all + 3160
    frame #1: 0x00007fffbb152131 libsystem_malloc.dylib`malloc_zone_check + 58
    frame #2: 0x00007fffbb151e8a libsystem_malloc.dylib`internal_check + 17
    frame #3: 0x00007fffbb143feb libsystem_malloc.dylib`free + 358
    frame #4: 0x00000001057a88ff libwireshark.0.dylib`wmem_strict_free_all [inlined] 
wmem_strict_free(private_data=<unavailable>) + 47 at wmem_allocator_strict.c:139 [opt]
    frame #5: 0x00000001057a88dc libwireshark.0.dylib`wmem_strict_free_all(private_data=0x00007fa42650d670) + 12 at 
wmem_allocator_strict.c:194 [opt]
    frame #6: 0x00000001057a6efa libwireshark.0.dylib`wmem_destroy_allocator [inlined] 
wmem_free_all_real(allocator=0x00007fa426527eb0, final=1) + 26 at wmem_core.c:118 [opt]
    frame #7: 0x00000001057a6ee9 libwireshark.0.dylib`wmem_destroy_allocator(allocator=0x00007fa426527eb0) + 9 at 
wmem_core.c:137 [opt]
    frame #8: 0x00000001057a9cdf libwireshark.0.dylib`wmem_cleanup_scopes + 95 at wmem_scopes.c:159 [opt]
    frame #9: 0x00000001049edf4b tshark`main(argc=<unavailable>, argv=<unavailable>) + 12251 at tshark.c:2087 [opt]
    frame #10: 0x00007fffbafc0255 libdyld.dylib`start + 1
    frame #11: 0x00007fffbafc0255 libdyld.dylib`start + 1

Thanks,
Nora

> Hey Nora,
>
> On Wed, Dec 07, 2016 at 04:07:45PM -0800, Nora Sandler wrote:
> > Hi list!
> >
> > I'm trying to do some fuzzing with fuzz-test.sh, and I'm seeing some strange behavior that I hope someone can help 
> > me figure out. For every pcap I try, I get a crash pretty quickly, usually in less than 10 passes. But then I can't 
> > reproduce the crash using test-captures.sh and the fuzzed output file. 
> >
> > I'm seeing this behavior using my own pcaps as well as captures from 
> > https://wiki.wireshark.org/SampleCaptures#Sample_Captures. I'm running the latest code from the master branch on OS 
> > X 10.12. It seems like it's heap corruption related since I stop getting crashes if I comment out the following 
> > lines test-common.sh: 
> >
> > export MallocCheckHeapStart=1000
> > export MallocCheckHeapEach=1000
> >
> > Here's some sample output:
> >
> > Fuzzing:
> >
> > $ ./tools/fuzz-test.sh -b ./build/run/ ~/Downloads/dhcp.pcap
> > expr(15011,0x7fff9ab623c0) malloc: protecting edges
> > expr(15011,0x7fff9ab623c0) malloc: enabling scribbling to detect mods to free blocks
> > expr(15011,0x7fff9ab623c0) malloc: checks heap after 1000th operation and each 1000 operations
> > expr(15011,0x7fff9ab623c0) malloc: will abort on heap corruption
> > [...]
> > mv(15166,0x7fff9ab623c0) malloc: will abort on heap corruption
> > ./tools/fuzz-test.sh: line 203: 15155 Segmentation fault: 11  (core dumped) "$RUNNER" $COMMON_ARGS $ARGS 
> > $TMP_DIR/$TMP_FILE > /dev/null 2>> $TMP_DIR/$ERR_FILE.$SUBSHELL_PID
> > [...]
> >
> > ERROR
> > Processing failed. Capture info follows:
> >
> >  Input file: /Users/me/Downloads/dhcp.pcap
> >  Output file: /tmp/fuzz-2016-12-07-15007.pcap
> >  Pass: 6
> > [...]
> > stderr follows:
> >
> > cat(15181,0x7fff9ab623c0) malloc: protecting edges
> > cat(15181,0x7fff9ab623c0) malloc: enabling scribbling to detect mods to free blocks
> > cat(15181,0x7fff9ab623c0) malloc: checks heap after 1000th operation and each 1000 operations
> > cat(15181,0x7fff9ab623c0) malloc: will abort on heap corruption
> > Input file: /Users/me/Downloads/dhcp.pcap
> >
> > Build host information:
> > Darwin my-machine.local 16.1.0 Darwin Kernel Version 16.1.0: Thu Oct 13 21:26:57 PDT 2016; 
> > root:xnu-3789.21.3~60/RELEASE_X86_64 x86_64
> > [...]
> > Return value:  0
> >
> > Dissector bug:  0
> >
> > Valgrind error count:  0
> > [...]
> >
> > And trying to reproduce the crash:
> >
> > $ ./tools/test-captures.sh -b ./build/run /tmp/fuzz-2016-12-07-15007.pcap
> > Testing file /tmp/fuzz-2016-12-07-15007.pcap...
> > - with tree... sh(15256,0x7fff9ab623c0) malloc: protecting edges
> > [...]
> > OK
> > [...]
> > - without tree... sh(15262,0x7fff9ab623c0) malloc: protecting edges
> > [...]
> > OK
> > - without tree but with a read filter... sh(15268,0x7fff9ab623c0) malloc: protecting edges
> > [...]
> > OK
> >
> > Is this an actual memory corruption bug in wireshark? A problem with the fuzzing script? Or am I doing something 
> > wrong? I'd appreciate any suggestions you have.
> >
> > Thanks,
> > Nora Sandler
>
> Is it possible that the tool consumes a lot of memory, eventually
> running out of memory and failing on allocations? Can you try to obtain
> a coredump for post-mortem analysis?
> -- 
> Kind regards,
> Peter Wu
> https://lekensteyn.nl

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe