Re: Memory-mapped capture and thinking the packet's

[Eloy Paris <peloy () chapus net>]

Hi Guy,

On 07/10/2009 05:07 PM, Guy Harris wrote:

> pcap_next() and pcap_next_ex() rely on the packet data pointer handed to
> their pcap_dispatch() callback still pointing to the same packet data
> after the callback returns.
>
> If the packet data is being read into a buffer with
> read()/getmsg()/recvfrom()/etc., that works.
>
> If, however, the packet data is being fetched from a memory-mapped
> buffer shared with the kernel, so that the pointer points into that
> buffer, it doesn't necessarily work - when the callback returns, libpcap
> might release that entry in the buffer, allowing the kernel to reuse it
> and put new packet data there.
>
> A problem that was probably the result of this was reported in a thread
> titled "Buffer overwrites with pcap_next_ex"; I attempted to fix it by,
> in the Linux memory-mapped capture implementation, not releasing the
> packet until the *next* packet was fetched.
>
> Unfortunately, *that* broke select()/poll(), as reported in a thread
> titled "select() regression in libpcap-devel?"; select() thinks there's
> an unconsumed packet, and always reports that reading a packet is
> available, so I had to back that fix out.
>
> The pcap_next() and pcap_next_ex() problem could be fixed by having
> those routines, when capturing in memory-mapped mode, use a callback
> that copies the packet to a separate buffer and returns a pointer to
> *that* buffer. This means you do one more copy, but if that bothers you,
> you need to stop using pcap_next()/pcap_next_ex() and just use
> pcap_dispatch()/pcap_loop() and live with having to deal with the
> callback architecture.
>
> Unfortunately, it wouldn't fix any applications that use
> pcap_dispatch()/pcap_loop() and *still* assume that the packet data
> won't be changed until the next packet is read. I don't know whether any
> such applications exist; given that there is no guarantee that the data
> for packet N is still available after packet N+1 is read (and, in fact,
> there's a good chance that it *isn't* available - a certainty when, for
> example, capturing on Linux in non-memory-mapped mode), I *suspect* few
> applications make that assumption.

My application Network Expect ran into a regression and after spending a 
couple of hours troubleshooting this I narrowed the problem down to 
packet data getting overwritten after returning from pcap_next() and 
before the next call to pcap_next(). I then found the thread titled 
"Buffer overwrites with pcap_next_ex", which you mention above and is 
the exact same problem I ran into, and then I recalled seeing a recent 
message from you on this topic (the message above).

So it seems like the only option I have to fix the regression is to 
convert the pcap_next() call to pcap_dispatch()/pcap_loop() semantics. I 
don't think that copying the packet to a safe place as soon as 
pcap_next() returns is good enough since, while the exposure window is 
narrower, there's still a chance to have packet data overwritten by the 
kernel.

This problem is really nasty, by the way, and I'm surprised I haven't 
seen more reports of this problem -- the regression in Network Expect 
ranged from weird, incorrect results to segfaults caused by starting to 
dissect a packet to later have the entire packet changed while in the 
middle of the dissection. Really hard to troubleshoot too.

Under these conditions pcap_next() and pcap_next_ex() are completely 
unusable. Perhaps we should document that pcap_next() and pcap_next_ex() 
should not be used when libpcap uses mmap()?

Along the same lines, is there a way to disable libpcap's use of mmap() 
at run-time even if it's available? That would be a better workaround 
than to re-write my application...

Cheers,

Eloy Paris.-
netexpect.org
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.