Re: The Making of a Router

[Eugeniu Patrascu <eugen () imacandi net>]

On Fri, Dec 27, 2013 at 10:00 PM, Baldur Norddahl <baldur.norddahl () gmail com
> wrote:

> On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands <fohdeesha () gmail com> wrote:
>
> > On Dec 27, 2013 10:08 AM, "Baldur Norddahl" <baldur.norddahl () gmail com>
> > wrote:
> >
> > > We are an upstart and just buying the fancy Juniper switch times two
> > would burn half of my seed capital.
> >
> > Then you didn't ask for nearly enough capital.
>
> Another told Nick Cameo that if he can afford a 10G link, he can afford
> Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial
> fee and less than $4k / month with unlimited traffic. The Juniper gear is
> $100k up front for two routers able to handle the 10G links.
What you should understand is not the fact that a 10G interface is
expensive, but what you can do with that interface tends to get very
expensive.
If you want to move traffic from one interface to another, you can achieve
this today with two physical interfaces on a Linux box. How many PPS ?
Well, that's another story. You then want shaping, Q-in-Q and other stuff
which consume a lot of resources even on dedicated hardware.


> What I get from you guys is that in your opinion it is not possible to set
> up a small ISP without spending a ton on Juniper or Cisco. I am not buying
> that. Even if I did not have a clear limit on my capital, I would be
> looking at avoiding paying that kind of money, because in the end the money
> comes out of my own pocket.
You can build your ISP without getting big routers but you need to cut back
a little bit on your expectations about what you can in terms of features:
- Do pool NAT for your users if they accept this. You can easily squeeze a
lot of users on a single IP address. Downside is that if one of them does
something bad, that IP might get blackholed on some providers and the rest
will suffer. Also, you might want to take into consideration regulatory
requirements like to know what users used what port to what destination for
a certain number of months (in Europe regulations vary, but the smallest
period is 6 months).
- If you give them VoIP/IPTV then assign a VLAN for VOIP and another for
IPTV and run it to all your users to their STBs and make use of IGMP
snooping for Multicast traffic on all your switches
- You can run full table BGP with Quagga on Linux (it worked for me when
the DFZ was at around 270k prefixes, I assume it will work with 480k
prefixes today) - also, do you really need full tables ?. Your IGP, if you
don't run anything fancy should be a few tens of routes, that can be
achieved with modest L3 switches that do 64/128 routes in hardware.


> Everybody have critical services running on servers. DHCP, DNS, Radius and
> so on are all on servers and you will be down if these services are down.
> What is with the knee jerk reaction for suggesting that the BGP daemon
> could also be run on a server? There seems to be many advantages of doing
> it this way, and not all of them are related to cost.
For the sake of a good night sleep, you would want to separate all the
services on different physical machines for redundancy/availability and
load sharing.

Once you grow, you can move to more powerful and dedicated hardware for
your networking needs.

Eugeniu