Re: MTU to CDN's

[Mikael Abrahamsson <swmike () swm pp se>]

On Thu, 18 Jan 2018, Michael Crapse wrote:

> I don't mind letting the client premises routers break down 9000 byte
> packets. My ISP controls end to end connectivity. 80% of people even let
> our techs change settings on their computer, this would allow me to give
> ~5% increase in speeds, and less network congestion for end users for a one
> time $60 service many people would want. It's also where the internet
> should be heading... Not to beat a dead horse(re:ipv6 ) but why hasn't the
> entire internet just moved to 9000(or 9600 L2) byte MTU? It was created for
> the jump to gigabit... That's 4 orders of magnitude ago. The internet
> backbone shouldn't be shuffling around 1500byte packets at 1tbps. That
> means if you want to layer 3 that data, you need a router capable of more
> than half a billion packets/s forwarding capacity. On the other hand, with
> even just a 9000 byte MTU, TCP/IP overhead is reduced 6 fold, and
> forwarding capacity needs just 100 or so mpps capacity. Routers that
> forward at that rate are found for less than $2k.

As usual, there are 5-10 (or more) factors playing into this. Some, in 
random order:

1. IEEE hasn't standardised > 1500 byte ethernet packets
2. DSL/WIFI chips typically don't support > ~2300 because reasons.
3. Because 2, most SoC ethernet chips don't either
4. There is no standardised way to understand/probe the L2 MTU to your 
next hop (ARP/ND and probing if the value actually works)
5. PMTUD doesn't always work.
6. PLPMTUD hasn't been implemented neither in protocols nor hosts 
generally.
7. Some implementations have been optimized to work on packets < 2000 
bytes and actually has less performance than if they have to support 
larger packets (they will allocate 2k buffer memory per packet), 9k is 
ill-fitting across 2^X values
8. Because of all above reasons, mixed-MTU LAN doesn't work, and it's 
going to be mixed-MTU unless you control all devices (which is typically 
not the case outside of the datacenter).
9. The PPS problem in hosts and routers was solved by hardware offloading 
to NICs and forwarding NPUs/ASICs with very high lookup speeds where PPS 
no longer was a big problem.

On the value to choose for "large MTU", 9000 for edge and 9180 for core is 
what I advocate, after non-trivial amount of looking into this. All major 
core routing platforms work with 9180 (with JunOS only supporting this 
after 2015 or something). So if we'd want to standardise on MTU that all 
devices should support, then it's 9180, but we'd typically use 9000 in RA 
to send to devices.

If we want a higher MTU to be deployable across the Internet, we need to 
make it incrementally deployable. Some key things to achieve that:

1. Get something like 
https://tools.ietf.org/html/draft-van-beijnum-multi-mtu-05 implemented.
2. Go to the IETF and get a document published that advises all protocols 
to support PLMTUD (RFC4821)

1 to enable mixed-MTU lans.
2 to enable large MTU hosts to actually be able to communicate when PMTUD 
doesn't work.

With this in place (wait ~10 years), larger MTU is now incrementally 
deployable which means it'll be deployable on the Internet, and IEEE might 
actually accept to standardise > 1500 byte packets for ethernet.

--
Mikael Abrahamsson    email: swmike () swm pp se