nanog mailing list archives
Re: Thousands of hosts on a gigabit LAN, maybe not
From: Rafael Possamai <rafael () gav ufsc br>
Date: Fri, 8 May 2015 14:26:54 -0500
- The more switches a packet has to go through, the higher the latency, so your response times may deteriorate if you cascade too many switches. Legend says up to 4 is a good number, any further you risk creating a big mess. - The more switches you add, the higher your bandwidth utilized by broadcasts in the same subnet. http://en.wikipedia.org/wiki/Broadcast_radiation - If you have only one connection between each switch, each switch is going to be limited to that rate (1gbps in this case), possibly creating a bottleneck depending on your application and how exactly it behaves. Consider aggregating uplinks. - Bundling too many Ethernet cables will cause interference (cross-talk), so keep that in mind. I'd purchase F/S/FTP cables and the like. Here I am going off on a tangent: if your friends want to build a "super computer" then there's a way to calculate the most "efficient" number of nodes given your constraints (e.g. linear optimization). This could save you time, money and headaches. An example: maximize the number of TFLOPS while minimizing number of nodes (i.e. number of switch ports). Just a quick thought. On Fri, May 8, 2015 at 1:53 PM, John Levine <johnl () iecc com> wrote:
Some people I know (yes really) are building a system that will have several thousand little computers in some racks. Each of the computers runs Linux and has a gigabit ethernet interface. It occurs to me that it is unlikely that I can buy an ethernet switch with thousands of ports, and even if I could, would I want a Linux system to have 10,000 entries or more in its ARP table. Most of the traffic will be from one node to another, with considerably less to the outside. Physical distance shouldn't be a problem since everything's in the same room, maybe the same rack. What's the rule of thumb for number of hosts per switch, cascaded switches vs. routers, and whatever else one needs to design a dense network like this? TIA R's, John
Current thread:
- Thousands of hosts on a gigabit LAN, maybe not John Levine (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Chuck Church (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Christopher Morrow (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Benson Schliesser (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Dave Taht (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not John Levine (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Rafael Possamai (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Brandon Martin (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Niels Bakker (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Brandon Martin (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Niels Bakker (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Miles Fidelman (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Miles Fidelman (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Blake Hudson (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Sameer Khosla (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Brian R (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not John R. Levine (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not charles (May 08)
(Thread continues...)