tcpdump mailing list archives
Re: advice for heavy traffic capturing
From: "Fulvio Risso" <fulvio.risso () polito it>
Date: Sun, 15 Aug 2004 09:55:11 +0200
Hi Darren.
-----Original Message----- From: Darren Reed [mailto:darrenr () reed wattle id au] Sent: sabato 14 agosto 2004 21.23 To: Fulvio Risso Cc: tcpdump workers Subject: Re: [tcpdump-workers] advice for heavy traffic capturing Hi Fulvio,What have you found that makes you say this ? The simplicity in cpu cycle cost ?1. simplicity 2. swappable buffers are very helpful if you plan to makestatistics, notonly packet capture. For instance, let's think about a system (like a NetFlow probeor somethinglike that) that collects statistics, then it returns data tothe user everyN minutes. If you have two buffers you can put statistics inthe first one,while you can read data from the second one, and swap buffers every N minutes. If you have a ring buffer and your application wants to readdata, you haveto stop collecting stats, lock the ring, copy its content inanother buffer,unlock the ring, read data from the second buffer, and restart computing statistics.Is it really that hard with a ring ?
Yes, it is harder. For instance, let's imagine you collected statistics based on the IP source address (e.g. byte count). You need something like a 10MB buffer (uhmmm, probably much more) to store data. Now, let's imagine you want to plot a snapshot of your stats every minute. If you have a ring buffer, you should lock the entire buffer in order to copy its content and make it available to the application. You have to make sure that no other threads are going to update data in the meanwhile, otherwise your statistics are broken (they refers to different time intervals). With a swappable buffer, you can avoid this problem by simply swapping the buffers.
At least on Unix I think the way to go would be to just make sure that the input side of the ring processing (transferring packets from NICs) was not able to overtake the reading by the user process. Then, with successive calls to uiomove(), data is copied from the ring out to the user process. That is if the number of buffers in the ring is static.
This is correct if you plan to do packet capture. This is way we said, in 2001, "ring buffer is better than swappable buffer". Then, we decided to implement statistical capabilities (which have never been released), and we discovered that the swappable buffer is not a good choice from that point of view.
Or did you try doing this with Windows and found other problems? Like the copy to user space is so expensive that it is quicker to copy into a new kernel buffer, unlock the ring, and then copy to userspace? btw, do you have any comments on how stable WinPCap2p3 is/was? We had some systems running this but found Windows 2000 to be extremely unreliable (systems would lockup and require a hard reset at least once every 2 weeks.)
Yes, I remember some memory leaks and such in WinPcap 2.3. They should have gone now. You should probably try the 3.1 beta (the final is hopefully planned for the beginning of September; its more than two months that we do not get any bug report). Cheers, fulvio - This is the tcpdump-workers list. Visit https://lists.sandelman.ca/ to unsubscribe.
Current thread:
- Re: advice for heavy traffic capturing, (continued)
- Re: advice for heavy traffic capturing Fulvio Risso (Aug 09)
- Re: advice for heavy traffic capturing Darren Reed (Aug 09)
- Re: advice for heavy traffic capturing Fulvio Risso (Aug 09)
- Re: advice for heavy traffic capturing Darren Reed (Aug 09)
- Re: advice for heavy traffic capturing Fulvio Risso (Aug 09)
- Re: advice for heavy traffic capturing Loris Degioanni (Aug 09)
- Re: advice for heavy traffic capturing Darren Reed (Aug 10)
- Re: advice for heavy traffic capturing Loris Degioanni (Aug 10)
- Re: advice for heavy traffic capturing Motonori Shindo (Aug 12)
- Re: advice for heavy traffic capturing Darren Reed (Aug 14)
- Re: advice for heavy traffic capturing Fulvio Risso (Aug 15)
- Re: advice for heavy traffic capturing Darren Reed (Aug 16)
- Re: advice for heavy traffic capturing Fulvio Risso (Aug 16)
- Re: advice for heavy traffic capturing Guy Harris (Aug 08)