Re: New packet list - out of memory?

["Anders Broman" <anders.broman () ericsson com>]

-----Original Message-----
From: wireshark-dev-bounces () wireshark org
[mailto:wireshark-dev-bounces () wireshark org] On Behalf Of Bill Meier
Sent: den 6 oktober 2009 00:49
To: Developer support list for Wireshark
Subject: Re: [Wireshark-dev] New packet list - out of memory?

Bill Meier wrote:
> > A data point: A few days ago while working on bug #2375 I noticed
that 
> > on Windows that if I defined the env logicals to not use se_alloc & 
> > etc that *much* less memory was required to load a capture. (20% of 
> > that used with se_alloc & etc ? I don't quite remember).
>
> Details:
>
> Env: Windows Vista
>
> Actions:
>
> 1. Load Wireshark
>    Determine "Private Working Set", "Working Set", "Commit Size"
>
> 2. Load a 40 Meg capture file.
>    Determine ....  again
>
> The above was done for:
>
> a. Wireshark with new_packet_list:
>
> b. Wireshark w/new_packet_list:
>     set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
>     set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
>
> c. Wireshark without new packet list
>
> d. Wireshark without new packet list
>     set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
>     set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
>
> Measurements
>
> Note: The Working Set was always about 20 Meg larger than the
>    Private Working Set(PWS) so only the PWS is shown.
>
> (All values in Megs)
>                                        EP/SE Chunks        No Chunks
>                                        PWS  Commit        PWS  Commit
>                                        ---- -------     ----- -------
> Wireshark: current_packet_list
>      After startup:                     42      62         42     52
>      After 40M CF load:                180     399        195    205
>
>
> Wireshark: new_packet_list:
>      After startup:                     42      62         42     52
>      After 40M CF load:                136     564        165    177
>
>
> A conclusion:
>
> Commit Size (which I believe is the amount of VM address space used
>    by the process and which is presumably the limiting factor for
>    memory usage).
>       - Best for new_packet_list/no_chunks;
>       - worst (quite) for new_packet_list/chunks;
>
> Comments welcome...
>
> (I did these tests somewhat quickly so I could have missed
something...)
> Bill

Hi,
I expanded on Bills measurements using the file that caused out of
memory for the new packet list. (Thanks Bill I didn't know about the
commit size option in the task manager).
 
146 MB 661417 pkts (TCP reassembly)

Version 1.2.2 (SVN Rev 29910)
PWS       Commit                
 42 016      61 876     
704 656   1 639 256              
Loading time ~57s
 
Old packet list SVN 30353
PWS       Commit                
 43 060      63 384     
492 456   1 427 480              
Loading time 47.752 - 48.485 (2 Loading time measures)

Old packet list SVN 30353
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
PWS       Commit                
 43 060      63 384     
559 744     579 968              
Loading time 48.797

Old packet list SVN 30353
set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
 43 208      53 312
559 612     569 632
Loading time 55.162

Old packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
PWS       Commit                
 43 288      64 188     
492 104     567 060              
Loading time 47.736 - 48.391 (2 Loading time measures)

 

(This file can't be loaded using 512)
New packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
PWS       Commit                
 43 492      64 512     
332 400     471 264              
Loading time 55.895 - 57.018 (4 Loading time measures)

New packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
PWS       Commit                
 43 660      64 736     
468 496     493 660              
Loading time 53.666

New packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
PWS       Commit                
 43 660      64 736     
468 428     481 516              
Loading time 1:01.215 - 1:02.682 (2 Loading time measures) 

  Compared with 1.2.2 memory usage has improved quite a bit. Loading
time
  has improved with the old packet list but not so much with the new
packet list!?

Conclusions:
- se alloced memory is a problem as the allocation "waists" memory.
  This is not a big problem for ep memory as it's given back to the pool
after each packet.
- Using the old packet list the file loads faster !?
  This was not the case some way along the line...
- Surprisingly enough the file loads faster not using chunks for
se_alloc ???
  Whereas it is faster using ep memory.

Increasing EMEM_ALLOCS_PER_CHUNK looks like a good thing - drawbacks?
Should the use of se_alloc be discouraged/removed?

Regards
Anders

________________________________________________________________________
___
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
 
mailto:wireshark-dev-request () wireshark org?subject=unsubscribe
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe