Serious Performance Problems with Sun NIT/DLPI

[mcn () EnGarde com (Mike Neuman)]

While this doesn't have a direct security implication, it does affect
many of us because of our use of network sniffers as security tools. Here's
the post I made this morning:

----
In preparation for the commercial release of our IP-Watcher network
monitoring and control program (see http://nad.infostructure.com/watcher.html),
we've been doing a great deal of performance testing on a variety of network
packet filters. The results of which are interesting, although they only
seem to support my theory that STREAMS *REALLY* stinks for realtime (or
at least time-critical) applications.

Our product incorporates the BPF filtering routines from LBL's libpcap. We
wanted to compare the performance of:

SunOS:
NIT with user-space BPF filtering (e.g. NIT returns the packet, and our
                                   application filters it "by hand")
vs.
NIT with in-kernel NIT filtering  (nit_pf)

Solaris:
DLPI with user-space BPF filtering (e.g. DLPI returns the packet, and our
                                    applications filteres it "by hand")
vs.
DLPI with in-kernel DLPI filtering (pfmod)

Pfmod and nit_pf should be much faster than user-space filtering as the
overhead of copying data out of the kernel is elminated for filtered packets.

Our test worked as follows:

There are three machines (A, B, C) on an ethernet with no other traffic.
A starts two connections to B - one to chargen, one to sink. A then repeatedly 
sends 1500 byte packets to sink, and reads 1500 bytes from chargen.

The sniffing program on C is configured to filter everything but the sink 
traffic. C starts a timer and after 100 non-filtered packets have been 
received, it stops the timer and prints out the difference.

More technical information:
NIT with user-space BPF filtering has nit & nit_buf pushed on the stream.
NIT with in-kernel NIT filtering has nit & nit_pf & nit_buf pushed on the 
stream. (in that order)

DLPI with user-space BPF filtering has dlpi & bufmod pushed on the stream.
DLPI with in-kernel DLPI filtering has dlpi & pfmod & bufmod pushed on the 
stream. (in that order)

All 4 tests have the buffer timeouts set to 0.

We ran each individual tests 4 times in a row (to annul cache effects and to
get an even sample), followed immediately by the alternative test run 4 times
in a row. This test sequence was run at least 10 times on each platform.

The results:

DLPI w/ user-space BPF filtering: 216ms average
DLPI w/ in-kernel DLPI filtering: 385ms average

NIT w/ user-space BPF filtering:  296ms average
NIT w/ in-kernel NIT filtering:   386ms average

Summary statements:

NIT with user-space BPF filtering is 30% faster than NIT with NIT filtering
DLPI with user-space BPF filtering is 78% faster than DLPI with DLPI filtering


Summary:

As I said at the beginning of this: STREAMS is bad for realtime. I wonder
how badly Solaris suffers with its network STREAMS implementation versus
SunOS with its mbuf-et-al implementation for high speed/low latency traffic.
I know I've heard Sun people say Solaris networks faster, but I can't see
how this can be the case with results like these.

Why not just implement the filtering stuff in libraries--it apparently is 
faster to copy it out of the kernel than filter it in-kernel? :-)

Or, if my dreams were to come true, include the Berkeley Packet Filter
with Solaris/SunOS rather than NIT/DLPI. It's faster, easier to use, more
powerful, and benefits from the source being freely distributed and refined 
over the past few years.

-Mike
mcn () EnGarde com
En Garde Systems
Computer Security Software and Consulting