tcpdump mailing list archives
Re: libpcap delivering the same packet more than once under high load?
From: Jim Lloyd <jlloyd () silvertailsystems com>
Date: Tue, 14 Sep 2010 10:36:19 -0700
On Tue, Sep 14, 2010 at 5:48 AM, Alexander Dupuy <alex.dupuy () mac com> wrote:
Jim Lloyd writes:These duplicate packets cannot be unique packets that were retransmitted between the two machines on the layer 1 GigE link, because if there was a significant increase in retransmission duplicates on the link, they would have competed for the fixed 1Gbps channel capacity. But the data is consistent with libpcap delivering the same packets more than once.How are you capturing these packets (i.e. where is your libpcap application running, and what is it monitoring)?
For this testing I am running the application on the same machine that is running apache httpd.
If you are using a switch mirror or SPAN port to capture the traffic, there are many configurations that will capture packets multiple times, e.g. both on receive from one port and transmit on another.
Yes, we've see that in production environments. One of our customers has their SPAN configured so that we see duplicates of every packet. But that isn't the case with this test run in our lab.
If your traffic is crossing multiple (V)LANs which are being monitored, you can also see packets that are not exact duplicates (different MAC addresses, IP TTL/hop count & checksum) but which are effectively duplicates at the TCP layer.
The client test generator application (a custom app written using libcurl) runs on a different machine, but the two machines are attached to the same switch. No VLANs are involved.
If your libpcap application is running on the sender or receiver, and monitoring the host's interface that is used for the test traffic, you won't have this problem (although depending on your hardware you may see transmitted traffic that is not what actually goes out on the wire - e.g. TCP segmentation offload will split/fragment packets but libpcap will not see that).
This all makes sense, and shouldn't apply in the given test.
Unless you are using an optical tap or port replicator or something like that to perform the traffic capture, it is generally the case that what you see is not 100% identical to what is actually going across the wire.
Some other non-libpcap-related reasons why what you see is not exactly what was on the wire are in the kernel - on some multiprocessor systems (Linux at least, perhaps this is also true for *BSD) you may get packets in a different order than the one in which they were actually received (you can even see this in the libpcap packet receive timestamps).
I've been assuming that some reordering of packets may occur even in this idealized lab setup with two machines connected to the same switch. Our app handles reordering fine, and will not count reordered packets as duplicates.
All of these are not the fault of libpcap - although it is possible to add functionality to libpcap (or your application) to eliminate duplicate packets and/or reorder packets based on receive timestamps - we have done the former in our applications.
Yes, the application is written to expect duplicates and eliminate them when they occur. I'm not surprised to see some duplicates. I even expect to see a higher percentage of duplicates when I saturate a link. I just don't expect to see total traffic close to 3x the physical capacity of the link. I am of course still open to the possibility that the cause of the unexpected behavior is something outside of libpcap, but I am at a loss for a good explanation. Thanks for your ideas. Please let me know if you think of others. Jim - This is the tcpdump-workers list. Visit https://cod.sandelman.ca/ to unsubscribe.
Current thread:
- libpcap delivering the same packet more than once under high load? Jim Lloyd (Sep 10)
- <Possible follow-ups>
- Re: libpcap delivering the same packet more than once under high load? Alexander Dupuy (Sep 14)
- Re: libpcap delivering the same packet more than once under high load? Jim Lloyd (Sep 14)