nanog mailing list archives
Re: Lossy cogent p2p experiences?
From: Mark Tinka <mark@tinka.africa>
Date: Wed, 6 Sep 2023 18:15:11 +0200
On 9/6/23 11:20, Benny Lyne Amorsen wrote:
TCP looks quite different in 2023 than it did in 1998. It should handle packet reordering quite gracefully; in the best case the NIC will reassemble the out-of-order TCP packets into a 64k packet and the OS will never even know they were reordered. Unfortunately current equipment does not seem to offer per-packet load balancing, so we cannot test how well it works.
I ran per-packet load balancing on a Juniper LAG between 2015 - 2016. Let's just say I won't be doing that again.
It balanced beautifully, but OoO packets made customers' lives impossible. So we went back to adaptive load balancing.
It is possible that per-packet load balancing will work a lot better today than it did in 1998, especially if the equipment does buffering before load balancing and the links happen to be fairly short and not very diverse. Switching back to per-packet would solve quite a lot of problems, including elephant flows and bad hashing. I would love to hear about recent studies.
2016 is not 1998, and certainly not 2023... but I've not heard about any improvements in Internet-based applications being better at handling OoO packets.
Open to new info.100Gbps ports has given us some breathing room, as have larger buffers on Arista switches to move bandwidth management down to the user-facing port and not the upsteam router. Clever Trio + Express chips have also enabled reasonably even traffic distribution with per-flow load balancing.
We shall revisit the per-flow vs. per-packet problem when 100Gbps starts to become as rampant as 10Gbps did.
Mark.
Current thread:
- Re: Lossy cogent p2p experiences?, (continued)
- Re: Lossy cogent p2p experiences? Masataka Ohta (Sep 07)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 07)
- Re: Lossy cogent p2p experiences? Saku Ytti (Sep 07)
- Re: Lossy cogent p2p experiences? Benny Lyne Amorsen (Sep 09)
- Re: Lossy cogent p2p experiences? Randy Bush (Sep 09)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 09)
- Re: Lossy cogent p2p experiences? Dave Cohen (Sep 09)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 09)
- Re: Lossy cogent p2p experiences? Saku Ytti (Sep 10)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 06)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 06)
- Re: Lossy cogent p2p experiences? Masataka Ohta (Sep 06)
- Re: Lossy cogent p2p experiences? Nick Hilliard (Sep 03)
- Re: Lossy cogent p2p experiences? Masataka Ohta (Sep 03)
- Re: Lossy cogent p2p experiences? Nick Hilliard (Sep 03)
- Re: Lossy cogent p2p experiences? Masataka Ohta (Sep 03)
- Re: Lossy cogent p2p experiences? Nick Hilliard (Sep 02)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 01)
- Re: Lossy cogent p2p experiences? David Hubbard (Sep 01)
- Re: Lossy cogent p2p experiences? David Hubbard (Sep 11)
- Re: Lossy cogent p2p experiences? Mark Tinka (Sep 01)