nanog mailing list archives
Re: Thousands of hosts on a gigabit LAN, maybe not
From: Dave Taht <dave.taht () gmail com>
Date: Fri, 8 May 2015 12:26:16 -0700
On Fri, May 8, 2015 at 11:53 AM, John Levine <johnl () iecc com> wrote:
Some people I know (yes really) are building a system that will have several thousand little computers in some racks.
Very cool-ly crazy.
Each of the computers runs Linux and has a gigabit ethernet interface. It occurs to me that it is unlikely that I can buy an ethernet switch with thousands of ports, and even if I could, would I want a Linux system to have 10,000 entries or more in its ARP table.
Agreed. :) You don't really want 10,000 entries in a routing FIB table either, but I was seriously encouraged by the work going on in linux 4.0 and 4.1 to improve those lookups. https://netdev01.org/docs/duyck-fib-trie.pdf I'd love to know the actual scalability of some modern routing protocols (isis, babel, ospfv3, olsrv2, rpl) with that many nodes too....
Most of the traffic will be from one node to another, with considerably less to the outside. Physical distance shouldn't be a problem since everything's in the same room, maybe the same rack.
That is an awful lot of ports to fit in a rack (48 ports, 36 2U slots in the rack (and is that too high?) = 1728 ports) A thought is you could make it meshier using multiple interfaces per tiny linux box? Put, say 3-6 interfaces and have a very few switches interconnecting given clusters (and multiple paths to each switch). That would reduce your arp table (and fib table) by a lot at the cost of adding hops...
What's the rule of thumb for number of hosts per switch, cascaded switches vs. routers, and whatever else one needs to design a dense network like this? TIA
max per vlan 4096. Still a lot. Another approach might be max density on a switch (48?) per cluster, routed (not switched) 10GigE to another 10GigE+ switch. I'd love to know the rule of thumbs here also, I imagine some rules must exist for those in the VM or VXLAN worlds.
R's, John
-- Dave Täht Open Networking needs **Open Source Hardware** https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
Current thread:
- Thousands of hosts on a gigabit LAN, maybe not John Levine (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Chuck Church (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Christopher Morrow (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Benson Schliesser (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Dave Taht (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not John Levine (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Rafael Possamai (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Brandon Martin (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Niels Bakker (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Brandon Martin (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Niels Bakker (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Miles Fidelman (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Miles Fidelman (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Blake Hudson (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Sameer Khosla (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Brian R (May 08)
(Thread continues...)