Re: Thousands of hosts on a gigabit LAN, maybe not

[Miles Fidelman <mfidelman () meetinghouse net>]

Forgot to mention - you might also want to check out Beowulf clusters - 
there's an email list at http://www.beowulf.org/ - probably some useful 
info in the list archives, maybe a good place to post your query.

Miles

Miles Fidelman wrote:
> John Levine wrote:
> > Some people I know (yes really) are building a system that will have
> > several thousand little computers in some racks.  Each of the
> > computers runs Linux and has a gigabit ethernet interface.  It occurs
> > to me that it is unlikely that I can buy an ethernet switch with
> > thousands of ports, and even if I could, would I want a Linux system
> > to have 10,000 entries or more in its ARP table.
> >
> > Most of the traffic will be from one node to another, with
> > considerably less to the outside.  Physical distance shouldn't be a
> > problem since everything's in the same room, maybe the same rack.
> >
> > What's the rule of thumb for number of hosts per switch, cascaded
> > switches vs. routers, and whatever else one needs to design a dense
> > network like this?  TIA
>
> It's become fairly commonplace to build supercomputers out of clusters 
> of 100s, or 1000s of commodity PCs, see, for example:
> www.rocksclusters.org
> http://www.rocksclusters.org/presentations/tutorial/tutorial-1.pdf
> or
> http://www.dodlive.mil/files/2010/12/CondorSupercomputerbrochure_101117_kb-3.pdf 
> (a cluster of 1760 playstations at AFRL Rome Labs)
>
> Interestingly, all the documentation I can find is heavy on the 
> software layers used to cluster resources - but there's little about 
> hardware configuration other than pretty pictures of racks with lots 
> of CPUs and lots of wires.
>
> If the people you know are trying to do something similar - it might 
> be worth some nosing around the Rocks community, or some phone calls.  
> I expect that interconnect architecture and latency might be a bit of 
> an issue for this sort of application.
>
> Miles Fidelman


--
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra