Re: AT&T MPLS / BIB Routers

[Jim Gettys <jg () freedesktop org>]

On 02/16/2011 06:16 PM, Mikeal Clark wrote:
> I'm building up to 3000-4000ms latency with these BIB routers.  We never
> had this issue on the old point to points using Cisco gear.

Bufferbloat is getting more and more common, as memory has gotten 
cheaper, and braindead people claim that 0 packet loss is a good idea 
and only test for bandwidth.

So push back on AT&T, and start routinely testing latency under 
saturating load on each and every piece of hardware you bring in the door.

You don't see the queues build unless the gear is the bottleneck of a 
path, so it's easy to have the problem move on you: ergo test everything 
independently.

For example, as soon as your broadband bandwidth exceeds that of your 
wireless link in your house, your suffering will move from the broadband 
link to your 802.11 hop.  There your home router (and your laptop both) 
will have even more bufferbloat than most broadband connections, and you 
suffer even worse.

Ironically, the cleaner your 802.11 environment is, the worse you'll 
suffer, as some packet loss will keep the insanely huge buffers from 
filling.  Most current OS hardware seems to put queues of around 256 
packets in the driver, and there may be additional buffering above that.
                        - Jim

> On Wed, Feb 16, 2011 at 5:09 PM, Jim Gettys <jg () freedesktop org
> <mailto:jg () freedesktop org>> wrote:
>
>     On 02/16/2011 05:44 PM, Mikeal Clark wrote:
>
>         We just put in a AT&T MPLS and are having a pretty negative
>         experience with
>         the "Business in a Box" routers they are using for our smaller
>         sites.  We
>         are seeing extremely high latency under load.  Anyone have any
>         experience
>         with these devices that could shed some light on this?  Are they
>         really this
>         bad?
>
>
>     There is excessive buffering in all sorts of devices all over the
>     Internet. This causes high latency under load (along with higher
>     packet losses, and lots of other problems.
>
>     It's what I've been blogging about on http://gettys.wordpress.com.
>     These buffers fill; and they are so large they have defeated TCP
>     congestion avoidance to boot, with horrifying consequences.
>
>     So far, I've found this problem (almost) everywhere I've looked:
>             o ICSI has good data that bufferbloat is endemic in DSL,
>     Cable, and FIOS.  Delays are often measured in seconds (rather than
>     milliseconds).
>             o some corporate and ISP networks run without AQM, in
>     circumstances that they should.
>             o Windows, Mac OSX and Linux all have bufferbloat in their
>     network stacks, at a minimum on recent network device drivers, and
>     often elsewhere.
>             o Every home router I've tested is horrifyingly bad.
>             o 3g networks & 802.11 have this in spades.
>
>     Why should AT&T's MPLS be any different?
>
>     My next topic will be "transient" bufferbloat, having to do with
>     defeating slowstart.
>
>     Come start helping fix this: please join us at bufferbloat.net
>     <http://bufferbloat.net>, as we
>     try to get people to fix it.  Already there are some experimental
>     patches for the Linux Intel wireless driver.
>                             Jim Gettys
>                             Bell Labs