AW: AW: AW: /27 the new /24

[Jürgen Jaritsch <jj () anexia at>]

Hi Mike,

> but the boxes that have been there for 10 years have more than paid for themselves (unless they're a shitty business).

No question about that! But why should they throw them away if they can still print $$$ with these boxes? They have to 
change nothing till the global routing table reaches at least 768k ... so let's say this will happen in 12-18 months. 
They have enough time to prepare, migrate, etc ... and while all the side stories are happening they are still able to 
print $$$ with the "old shit".

> What I was saying is that my little business with meager means (and revenues) can afford a box to do it.

This is definitely a question about sizing. Replacing a box with ~200 connected customers (only at this box!) is way 
more complex and this is nothing unrealistic.

> If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or 
> whatever. 

4500x => no MPLS features

QFX-5100 => very nice box (I'm a big fan) but complicate (and expensive!) licensing. 

Extreme x670 => nice box too - we also use this. But it's simply too small and the BGP configuration on these boxes is 
horrible. It's also not possible to provide Ethernet over MPLS with LACP BPDU forwarding ... too less features. Nice 
for aggregation and POP interconnect. 

All three models are new and shiny but they can't replace a 6500/7600. Too less port density and too less features 
(people are still using SDH. You need SDH in an 6500/7600? Simply install the required line card ...). If you really 
plan to replace a 6509 or even a 6513 you have to go with something like Juniper MX480/960 (I'm in love ... :D) or 
Cisco Nexus 7k/9k. 

One thing that will more and more happen: physical separation. There will be boxes with 10G/40G/100G only and boxes 
with 100M/1G only. Why? It's easier for vendors to remove old compatibility requirements (like electrical interfaces). 
So what we did in the past 3 years (replacing old boxes with new boxes with 1G/10G interfaces) was useless - we'll get 
our "old shit" back in place and bring them up and running. Of course: the "old shit" will be reduced to do aggregation 
layer or to something like "multihop instance" to transport the customers access port to the "real big and powerful 
router". Solving this with Layer2 extensions (like VLANs) is not practicable because you'll ran into other problems 
(like STP instances, etc). Probably it makes sense to solve it with Layer2VPN (Ethernet over MPLS, etc) to transport 
the physical interface to a virtual interface.

Lots of things to think about :(.


> Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you whatever 
> the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Power is no argument. You get power starting at 0,10 Eur /kWh. Another 0,10 Eur / kWh for cooling and we talk about 
0,20 Eur / kWh => Cisco 6513 (configured with 11 line cards + 2x SUP) with 2x 6kW PSU uses 3,8kW. 3,8kW * 24 hours * 30 
days = 2.736 kWh per month. 2.736 * 0,20 Eur = 547,2 Eur per month for power consumption + cooling. If you have a good 
sales engineer you earn the revenue for this "side cost" with 1 customer :). Realistic calculation is: 10 customers are 
required to earn the money for the footprint.


> Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no 
> money or old big new big if money. 

Think the other way around: companies are earning Mio (or even Bil??) with the old equipment and everything is up and 
running. Only sometimes there is a small hick up because (of course!) also the "old shit" gets stuck from time to time 
and crashes. They did everything the right way (especially Level3 ...) from the commercial POV. 

> BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly 
> reduced time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with 
> the EdgeRouters yet. 

Never tried the earlier versions - my last tests happened in the end of 2014. I think we're talking a little bit about 
different sizes: you're talking about the CCRs and EdgeRouters (which are nice of course - no question about that!) and 
I'm talking about customer access devices (not CEP!) at carrier grade networks. Boxes I'm talking about have  at least 
a few hundred ports.

I think it's very important what UBNT and MT does: they bring fresh wind at the customer/semi-pro market and they show 
up that you (as a vendor) could get in touch with customers and optimize your equipment with customers feedback.


best regards


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: JJaritsch () anexia-it com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Ursprüngliche Nachricht-----
Von: NANOG [mailto:nanog-bounces () nanog org] Im Auftrag von Mike Hammett
Gesendet: Samstag, 03. Oktober 2015 02:52
Cc: NANOG <nanog () nanog org>
Betreff: Re: AW: AW: /27 the new /24

I don't expect carriers to be running UBNT\Mikrotik, but the boxes that have been there for 10 years have more than 
paid for themselves (unless they're a shitty business). It's time to rip and replace with whatever is appropriate for 
that site. No, I obviously don't think I'm going to change anyone's opinion on the matter (at least not anyone that 
matters in one of these networks). What I was saying is that my little business with meager means (and revenues) can 
afford a box to do it. They can too. 



I don't doubt their situation sucks... but either you fix it or you don't. Time and the rest of the Internet won't wait 
for them. 


If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or 
whatever. Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you 
whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no money 
or old big new big if money. 


BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly reduced 
time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with the 
EdgeRouters yet. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message -----

From: "Jürgen Jaritsch" <jj () anexia at> 
To: "Mike Hammett" <nanog () ics-il net> 
Cc: "NANOG" <nanog () nanog org> 
Sent: Friday, October 2, 2015 6:11:47 PM 
Subject: AW: AW: /27 the new /24 

Hi Mike, 

sorry, this was probably sent to quick ... let me please explain my POV of your statement: 

I want to concentrate my detailed answer only to the backbone situation which is often handled by the 6500/7600 - I 
guess all of us know that the 6500/7600 has a ton of additional features ... 


6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone capacities built with platforms that only had 
n*100M interfaces another 3-5 years before. Their only invest in these 3-5 years was to add the Gig line cards, install 
some software updates and add new fibre optics (GBICs). Chassis, cabling, management interfaces etc could be remain in 
the cabinet - they only had to replace ONE line card (let's say for a few thousand bucks) and with this invest they 
were able to scale up their capacities. Of course: at some point they also had to replace the SUPs, PSUs, FANs, etc. 
But the invest in the surrounding stuff is nothing compared with completely new machines. 

So what all these companies did was buying a machine with an basic configuration and since 10(!) years they are able to 
expand this machines with (more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources the platform can provide. Most of the 
carriers (and of course also the bigger ISPs) had a real chance to evaluate a new model/vendor to ran future networks 
(with possibly also a very good scale-up path and scaling- and upgrade-options). Most of the before mentioned are 
already in an migration process (let's take a look at Seabone ... they are migration from Cisco to a mix of Juniper and 
Huawei). 

Summary: there are strict limitations within the Cisco 6500/7600 platform and these limitations forces the big players 
to move this boxes out (or move them into other parts of their network). The limitation with 1Mio routes is not a 
secret and the admins of these boxes decide what they want to use (e.g. 768k routes for IPv4 unicast and 256k routes 
for MPLS+VRF, etc). If the global routing table reaches the 768k mark (I guess this will happen in the next 
12-18months) most of the boxes will crash again (as it happened in Aug 2014). 


Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at 
the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it 
could be: the BGP daemon uses only one core of a 36(?) core CPU. Same problem in the past with the EoIP daemon (not 
sure if they fixed it on the CCRs - they fixed it on x86). 

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff (also Ubiquiti). But with this boxes 
you're not able to ran a stable enterprise class carrier network with >99,5% uptime. And that’s thei MAIN reason why 
"the old shit" is still online :). 

Hopefully my words explained my hard "you know nothing" blabla ? 

Best regards 


Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch () anexia-it com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces () nanog org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 21:33 
Cc: NANOG <nanog () nanog org> 
Betreff: Re: AW: /27 the new /24 

Hrm. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message ----- 

From: "Jürgen Jaritsch" <jj () anexia at> 
To: "Mike Hammett" <nanog () ics-il net>, "NANOG" <nanog () nanog org> 
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about 
carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch () anexia-it com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces () nanog org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG <nanog () nanog org> 
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and 
has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your 
business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, 
heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the 
moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in 
every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires 
$500/month in power... and is worse. 

Stop using old shit. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message ----- 

From: "William Herrin" <bill () herrin us> 
To: "Mike Hammett" <nanog () ics-il net> 
Cc: "NANOG" <nanog () nanog org> 
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog () ics-il net> wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 




-- 
William Herrin ................ herrin () dirtside com bill () herrin us 
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>