Re: Layer 2 vs. Layer 3 to TOR

[Malte von dem Hagen <mvh () hosteurope de>]

Hej,

Am 12.11.2009 21:04 Uhr schrieb Raj Singh:
> We are actually looking at going Layer 3 all the way to the top of rack and
> make each rack its own /24.

what a waste of IPs and unnecessary loss of flexibility!

> This provides us flexibility when doing maintenance (spanning-tree).

If you use a simple setup for aggregation, you do not need xSTP. Even including
redundancy, RTG (big C: flex-link) will be sufficient. Spanning the L2 over more
than one rack is dirty when you do L3 on the TORs, because you need to build a
Virtual Chassis or VPLS tunnels (not sure if EX4200 does that as of today).

> Also, troubleshooting during outages is much easier by using
> common tools like ping and trace routes.

Oh, c'mon. Yes, Layer 2 is a wild jungle compared to clean routing, but tracing
isn't that magic there. You have LLDP, mac-address-tables, arp-tables...

> I want to make sure this is something other people are doing out there and
> want to know if anyone ran into any issues with this setup.

From the design POV, it is a clean and nice concept to do L3 on the
TOR-switches, but in real life, it's not working very well. Everytime I played
with such, with every vendor I've seen, there is just always the same conclusion:

Let routers route and let switches switch.

Switches which are supposed to do routing never scale, provide almost always
immature implementations of common L3 features and run into capacity problems
just too fast (too small tables for firewall roules, route entries, no full IPv6
capabilities, sometimes expensive licenses needed for stuff like IS-IS...).

I understand the wish to keep broadcast domains small and network paths
deterministic and clean, but the switches you can buy today for
not-too-much-money aren't ready yet.

So my hint is: Look at model #4 from the mentioned NANOG presentation.

My 2 Euro-Cents,

.m

Attachment: signature.asc
Description: OpenPGP digital signature