Re: Changing ISP?

[Samuel Young <syoung () LASIERRA EDU>]

We are looking at using our Cable TV provider in addition AT&T.  The Cable
provider uses their own cabling and uplinks via Fiber to Level3 and MCI, so
it does not intersect with our phone provider.

Sam Young
CIO
La Sierra University.

-----Original Message-----
From: Valdis Kletnieks [mailto:Valdis.Kletnieks () VT EDU]
Sent: Wednesday, October 04, 2006 9:30 AM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: Re: [SECURITY] Changing ISP?

On Wed, 04 Oct 2006 10:16:03 EDT, John Kaftan said:

> 1.    Should we leave MCI and suffer the pain of changing ISPs and
> receiving a new set of Public IPs?

If you're doing that, you may as well look at biting the bullet and
dual-homing.

> 2.    Can anyone else relate their recent experiences with MCI positive or
> negative?

Can't comment on that, sorry.

> 3.    Also, is it sufficient to use a single ISP for redundancy if they
> give us separate local loops, via separate ILECs, into opposite ends of
the
> campus, to separate COs?

A few comments on diversity here.  Surprisingly enough, for most of the
issues,
it doesn't actually matter whether it's a single ISP or 2.

0) Sometimes, putting all your eggs in one basket is good enough, if it's
a sufficiently good basket.  The vast majority of our connectivity to the
outside world is via a single OC-12 - it's reliable, and getting a
redundant OC-12 would be quite expensive.

1) What are your chances of *really* getting them from separate ILECs into
separate COs, on *truly* diverse paths?  I've been in Utica - there aren't
THAT
many ways out of town.  (Checks Google Maps quickly).  It looks like you
have
railroad tracks southeast of you - how many places can a cable get across
the
tracks?  Similarly, if you have 2 COs in Utica, are both in the same ILEC
(they
almost certainly are - to get another CO in another ILEC, somebody will
likely
be stringing a *LOT* of cable on poles)? If both COs are fed out of Rome,
how
do the cables get across I-90?

2) If you get several miles of cable from a CO in another ILEC, what methods
do you plan to use to ensure that cable isn't fate-sharing with the cable
to your local CO?

(If you think I'm kidding, get brave, and go down to where the I-790 bridge
crosses the river just north of Utica.  I *guarantee* you'll find an amazing
number of conduits glued to the underside of the bridge.  Just don't take
pictures, or the DHS guys will have to get medieval on you.. ;)

3) Even assuming you get the ISP to fess up to what the *current* routing
of the cable is (and getting *two* providers to be detailed enough to make
sure they're not sharing conduit or even lambdas is a major challenge), how
do you ensure that they *stay* diverse?  It's not at all unusual for a
company to reprovision your DS-3 onto a different cable in order to free
up lambdas for somebody else's pipe.  Unknown to you, the path that used
to cross the river on one bridge now is routed across the river
on the *other* bridge, in the same conduit as your other path...

(If you think I'm picking on you, I'm not.  Go back and read the NANOG
archives - even the "big players" that are housed at 60 Hudson in Manhattan
have trouble finding 2 diverse paths off the island.  45,000 square feet
of routers - and most of them need to find ways out of Manhattan, and
there's
a limited number of bridges and tunnels to carry your fiber, unless you want
to get *real* ambitious and bury your own cable across the floor of the
Hudson - at least one company did end up doing that...)

http://www.carrierhotels.com/properties/telx/

And the guys at One Wilshire on the left coast have just as big issues.
30 floors, 656K square feet.  The Meet-Me room is quite possibly the biggest
routing swamp on the entire planet - but there's still only 3 points of
entry into the building....
http://www.carrierhotels.com/properties/onewilshire/index.shtml

4)
http://maps.google.com/?ie=UTF8&z=16&ll=43.09821,-75.268793&spn=0.011892,0.0
22831&t=h&om=1
I admit not knowing exactly how your infrastructure is laid out, but I'm
going
to take a wild guess and say there aren't all *that* many ways to bring a
cable
onto campus and hook it into your campus net in a useful manner (barring
digging a big ditch to bury cable). And even if there are 2 or more ways
onto
campus, is self-inflicted backhoe fade due to construction on campus an
issue?

5) You may want to review your current reliability stats, and figure out how
often an outage was due to backhoe fade, hardware failure, or some
chucklehead
at the ISP NOC fat-fingering an IOS configuration (or a chucklehead at
*your*
end of the cable doing the same - happens to the best of us). <grumbling
about
a busticated ACL on a Foundry switch adding 4 hours to troubleshooting an
unrelated 15 hour mess yesterday>...

6) "sufficient" depends on your paranoia level, the chances of one link
having
an outage, the chances of the ISP having an upstream issue that kills *both*
of your links (you can have 53 redundant links, if your ISP's peering with
AS701
comes unglued, you're going to have some severe reachability issues), and
a lot of other issues.

You really need to ask questions like "How much uptime do we *want*?", "How
much downtime can we *really* tolerate?" and "How much are we willing to
spend
to improve the situation?" and "What level of survivability do various
services need to have?".   And keep in mind that the answers when talking
to the outside world may be *drastically* different than for on-campus - it
may be unacceptable to have more than 10-15 minutes of unscheduled outage
for your main mail hub to your on-campus users, but if mail to/from AOL
is delayed for an hour it's not a big deal.

> 4.    Can anyone speak to setting up redundancy with separate ISPs and
> BGP?

You'll need somebody who understands BGP.  However, if you're a fairly small
shop and don't have any *major* routing issues, it shouldn't be too hard.
(Our routing swamp is a tad more complicated, as it includes Internet2 and
Lambda connections,  but most of our stuff goes outbound via
network.virginia,
and we backhaul *some* (but not all) net.virginia sites for Internet2.
BGP communities are your friend. :)

> We have not talked about cost with any ISPs but I imagine it would be much
> more affordable using a single ISP.   

Actually, it won't be that much different - the single biggest chunk of your
costs will likely be the 2 pipes.  If you have 2 DS3's, it's going to cost
you $X/mo whether it's from one ISP or 2.

> We are planning owning our next set of IP addresses.  I'm told there might
> be a chance that we could keep our current set.  We'll see.

Note that you can (realistically) only keep your current allocation if one
of your upstreams remains MCI.