nanog mailing list archives
Re: Thousands of hosts on a gigabit LAN, maybe not
From: Miles Fidelman <mfidelman () meetinghouse net>
Date: Fri, 08 May 2015 15:53:28 -0400
John Levine wrote:
Some people I know (yes really) are building a system that will have several thousand little computers in some racks. Each of the computers runs Linux and has a gigabit ethernet interface. It occurs to me that it is unlikely that I can buy an ethernet switch with thousands of ports, and even if I could, would I want a Linux system to have 10,000 entries or more in its ARP table. Most of the traffic will be from one node to another, with considerably less to the outside. Physical distance shouldn't be a problem since everything's in the same room, maybe the same rack. What's the rule of thumb for number of hosts per switch, cascaded switches vs. routers, and whatever else one needs to design a dense network like this? TIA
It's become fairly commonplace to build supercomputers out of clusters of 100s, or 1000s of commodity PCs, see, for example:
www.rocksclusters.org http://www.rocksclusters.org/presentations/tutorial/tutorial-1.pdf orhttp://www.dodlive.mil/files/2010/12/CondorSupercomputerbrochure_101117_kb-3.pdf (a cluster of 1760 playstations at AFRL Rome Labs)
Interestingly, all the documentation I can find is heavy on the software layers used to cluster resources - but there's little about hardware configuration other than pretty pictures of racks with lots of CPUs and lots of wires.
If the people you know are trying to do something similar - it might be worth some nosing around the Rocks community, or some phone calls. I expect that interconnect architecture and latency might be a bit of an issue for this sort of application.
Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Current thread:
- Thousands of hosts on a gigabit LAN, maybe not John Levine (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Chuck Church (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Christopher Morrow (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Benson Schliesser (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Dave Taht (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not John Levine (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Rafael Possamai (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Brandon Martin (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Niels Bakker (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Brandon Martin (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Niels Bakker (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Miles Fidelman (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Miles Fidelman (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Blake Hudson (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Sameer Khosla (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Brian R (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not John R. Levine (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not charles (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not Phil Bedard (May 08)
- RE: Thousands of hosts on a gigabit LAN, maybe not charles (May 08)
- Re: Thousands of hosts on a gigabit LAN, maybe not Baldur Norddahl (May 09)
- Re: Thousands of hosts on a gigabit LAN, maybe not charles (May 09)
- RE: Thousands of hosts on a gigabit LAN, maybe not charles (May 08)