Re: I have a 5.8GB capture and it is taking an enormous amount of time to load

[Richard Sharpe <realrichardsharpe () gmail com>]

On Mon, Dec 10, 2018 at 8:11 PM Guy Harris <guy () alum mit edu> wrote:
> On Dec 10, 2018, at 5:16 PM, Richard Sharpe <realrichardsharpe () gmail com> wrote:
>
> > It has taken 20 minutes or more to load this monster (looks like about
> > 5M packets)
>
> "It has taken 20 minutes or more..." as in "it took 20 minutes or more to load it" or as in "it has, so far, taken 20 
> minutes or more to load it, and it's still loading it"?

The latter.

> > and I dread making searches through it.
> >
> > The Wireshark process is only consuming about 1.7GB of memory at the
> > moment and 17% of CPU on my laptop with NVMe and oodles of memory and
> > a 4-core Xeon.
>
> "At the moment" as in "after it finished loading, it's only consuming..." or "while it's loading, it's consuming..."?

While it's loading it's consuming ...

> The dissection process is currently not multi-threaded, so extra cores don't help.  Making it multi-threaded is 
> possible, but it would involve a lot of work to parallelize an in-principle-sequential process, including finding 
> dependencies (e.g., a case where dissecting packet N involves looking up something in a table that might be modified 
> by the dissection of packet M, for M < N, in which case the dissection of packet N has to block waiting for packet M 
> to be dissected).
>
> Just out of curiosity, what happens if you read the file with TShark without the -2 flag, sending the output to the 
> null device, and then with the -2 flag, again sending the output to the null device?  What's the difference in the 
> memory consumption of TShark in those cases, and how does it differ from the memory consumption of Wireshark for the 
> same file?  (That could indicate how much memory is used by the frame table - which shouldn't be constructed by 
> TShark without -2, but should be constructed by TShark with -2 and by Wireshark - and how much is used by the GUI 
> display widget showing the packet list, which will only be in Wireshark.)

I will give that a try.

However, I have found a way to make it more tractable. Since there are
two TCP streams (connections) in the capture, I am using tshark to
split the capture into those two streams, one of which I am interested
in (I don't know which at this stage). They should be around 2.9GB
each.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)(传说杜康是酒的发明者)
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe