Re: Validating multi-path in production?

[Martijn Schmidt via NANOG <nanog () nanog org>]

If your ECMP hashing algorithm considers L4 data I can recommend giving the TCP mode of the standard Linux MTR package 
a try. While the destination port remains a constant (iirc it defaults to port TCP/80) each iteration will use a 
different TCP source port, thereby introducing sufficient entropy to see if you get packetloss on a given amount of 
links in an ECMP heavy forwarding path. This always worked wonders for me back in the day when hunting down a broken 
port in a pair of 5x10G LACP bundles, e.g. 10 different possible paths, or when trying to find the rotten switching 
fabric in a chassis from a vendor with less than stellar debugging capabilities. Do keep in mind you need to keep the 
MTR running for a longer period of time to get a statistically significant amount of data to conclude anything from the 
percentages, e.g. let's say 10 minutes is better than 1 minute.

Best regards,
Martijn
________________________________
From: NANOG <nanog-bounces+martijnschmidt=i3d.net () nanog org> on behalf of Adam Thompson <athompson () merlin mb ca>
Sent: 14 November 2021 17:20
To: James Bensley <jwbensley+nanog () gmail com>; nanog <nanog () nanog org>
Subject: Re: Validating multi-path in production?

The problem I'm looking to solve is the logical opposite, I think: I want to demonstrate that no links are 
malfunctioning in such a way that packets on a certain path are getting silently dropped.  Which has some "proving a 
negative" aspects to it, unfortunately.
I think the only way I can demonstrate it is to determine that every single multi-path/hashed-member link is working, 
which is... hard.  Especially if I need to deal with the combinatoric explosion - I *think* I can skip that part.
-Adam

Get Outlook for 
Android<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=04%7C01%7Cmartijnschmidt%40i3d.net%7C8dc123f885474dfa474308d9a78aa33d%7Ce01bd386fa514210a2a429e5ab6f7ab1%7C0%7C0%7C637725036175018395%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=34%2FyM2QUlvE0fi%2F4V7tFFvJYTEyfdWfKG%2FjtbL4IIV8%3D&reserved=0>
________________________________
From: James Bensley <jwbensley+nanog () gmail com>
Sent: Sunday, November 14, 2021 5:29:25 AM
To: Adam Thompson <athompson () merlin mb ca>; nanog <nanog () nanog org>
Subject: Re: Validating multi-path in production?

On Fri, 12 Nov 2021 at 16:54, Adam Thompson <athompson () merlin mb ca<mailto:athompson () merlin mb ca>> wrote:
The best I've come up with so far is to have two test systems (typically VMs) that use adjacent IP addresses and 
adjacent MAC addresses, and test both inbound and outbound to/from those, blindly trusting/hoping that hashing 
algorithms will probably exercise both paths.

If the goal is to test that traffic *is* being distributed across multiple links based on traffic headers, then you can 
definable roll your own. I think the problem is orchestrating it (feeding your topology data into the tool, running the 
tool, getting the results out, and interpreting the results etc).

A coupe of public examples:
https://github.com/facebookarchive/UdpPinger<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookarchive%2FUdpPinger&data=04%7C01%7Cmartijnschmidt%40i3d.net%7C8dc123f885474dfa474308d9a78aa33d%7Ce01bd386fa514210a2a429e5ab6f7ab1%7C0%7C0%7C637725036175028351%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=29lU8yLVrzWEA5%2BeD6SmzmeSV%2FDc44YwvUMlXWB28sM%3D&reserved=0>
https://www.youtube.com/watch?v=PN-4JKjCAT0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPN-4JKjCAT0&data=04%7C01%7Cmartijnschmidt%40i3d.net%7C8dc123f885474dfa474308d9a78aa33d%7Ce01bd386fa514210a2a429e5ab6f7ab1%7C0%7C0%7C637725036175028351%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FmRle7f9%2F2QPO8b9AWyqQC9FP9CYcaGpzI%2BqKjwSs90%3D&reserved=0>

If you do roll your own, you need to taylor the tests to your topology and your equipment. For example, you can have 
two VMs as you mentioned, each at opposite ends of the network. Then, if your network uses a 5-tuple for ECMP inside 
the core for example, you could send many flows between the two VMs, rotating the sauce port for example, to ensure all 
links in a LAG or all ECMP paths are used.

It's tricky to know the hashing algo for every type of device you have in your network, and for each traffic type for 
each device type, if you have a multi vendor network. Also, if your network carries a mix of IPv4, IPv6, PPP, MPLS L3 
VPNs, MPLS L2 VPNs, GRE, GTP, IPSEC, etc. The number of permutations of tests you need to run and the result sets you 
need to parse, grows very rapidly.

Cheers,
James.