Re: pcap_next_ex() and pcap_dump() performance decreases over time...

[barcaroller <barcaroller () gmail com>]

On 2015-08-23 01:42:30 +0000, Guy Harris said:

> So probably real-time.
>
> That means that, *if* there are other processes on the machine, some of 
> the time *could* be due to other processes having all the cores, but, 
> with 8 cores, that's unlikely unless your test programs are lighting up 
> all 8 cores.

That's correct; my test programs are the only programs running.


> pcap_dump() is pretty simple - just a couple of fwrite() calls.  
> fwrite() is probably just copying stuff into the FILE *'s buffer and, 
> eventually, write()ing it out.
>
> I don't know whether you could use strace to see how long whatever 
> write() calls it makes take, but, with plenty of free disk space and 
> files only getting to 100MB, I don't see what would cause write() calls 
> to take up to 2 seconds.

It has puzzled me too.  Keep in mind that I am making millions of these 
pcap calls; only one of them could be taking 2 seconds.


> So, during phase 1, it captures packets and writes them out with 
> pcap_dump(), and, during phase 2, it reads from the file it just wrote?
>
> When capturing packets, are you using libpcap()?  If so, are you using 
> pcap_next_ex() both when capturing-and-writing and when reading?  If 
> so, does the slowdown in pcap_next_ex() happen in both phases?

I actually read the packets from a pcap file (not part of the test), 
store the packets in memory (e.g. 50,000 packets), then write them in a 
loop to disk (e.g. 50,000 x 1,000) using multiple pcap files (none 
greater than 100MB); this is phase 1.  I then read all packets (e.g. 
50,000 x 1,000) in chunks and destroy each chunk before reading the 
next; this is phase 2.


> I.e. Linux, and on a machine not starved for resources.

Right.  But something in the process is making pcap_next_ex() and 
pcap_dump() slower with time.


_______________________________________________
tcpdump-workers mailing list
tcpdump-workers () lists tcpdump org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers