nanog mailing list archives

Scaling from home broom closet to multisite "home" data center/WAN network on a budget


From: charles () thefnf org
Date: Tue, 14 Oct 2014 17:17:20 -0500

Hi everybody,

It's been a long time since I've kicked up a new thread here on ye ol nanog.

Recently I've been putting some serious thought into home "budget" data centers. What started out as a little router/switch/virt server lab by me/myself/I in 2008, has turned into a multisite (7 points of presence spread across two states), multi rack (223U, 10 racks), multi carrier (time warner, suddenlink, att dsl, att uverse, blended bandwidth) affair today.

I've realized that my friends and I may need to approach things a bit more..... professionally/seriously/circumspectly (is that a word?).

We run OSPF (and soon i/eBGP including out to amprnet) between all the sites, have a multi vendor (cisco/dell/pfsense/mikrotik/linux) l3/l2 network. All the gear has been sourced from flea markets/ebay etc.

Anyone else on here have a similar situation? How do you deal with it? (More details/questions after the jump)

I mean my little lab is not quite as .... *ahem* involved as mr morris home data center
http://smorris.uber-geek.net/lab.htm

but it's also just a whee bit bigger then the typical /r/homelab posts.

https://commons.thefnf.org/index.php/FNF_Lab << needs a bit more work to reflect current reality, but you can get the general idea)

So.... how to home/small (multisite) business datacenter/network on budgetz? Not so much the software side of things (networking/operating system etc), but more the actual layer 1 stuff (and of course software to manage all that).

Things like:

Power

* Any good resources for wiring optimization/layout/dos/dont's? (like say how to hook up surge protector -> ups -> pdu -> gear ?)

* Network UPS tools is what we are considering "betting on" for managing our multi vendor (APC/Dell/Cyclades) PDU/UPS setup across all sites. (The joys of sourcing from flea markets). Does anything better exist? Any issues with running this across a WAN? Any tips for tying it into LDAP? (We have a large amount of lab gear that we wish to make available to the public for use by reservation, we don't want them being able to turn off our production gear) :)


Temperature / Environmental monitoring

* I've found the temper USB sensors to be perfect for per rack temperature statistics. I believe they will do humidity as well. Any other good environmental sensors for cheap? I know about PacketFlux which is cool for an all in one solution. (I'd love to use netbotz or something, but those are pricy).

Cooling

* BTU calculation resources?
* Cooling sizing?
* How hot can gear really run?
* Is per rack cooling/airflow via floor fans necessary?

For now we all just have room AC cooling the gear. So far all seems well, curious if folks have done stuff like pipe AC output directly into a cabinet etc?

Noise dampening

Any good room partitions or other solutions for noise cancelling? We've got combination of closed cabs and open racks, so a rack based solution won't quite cut it.


Also what about insurance?

Probably a thousand other questions/comments/ideas/suggestions, but I think this should kick off a great discussion!

It's funny how it kind of all just combines/grows and next thing you know, you've got yourself a whole little internet as it were. :)



Current thread: