Re: NANOG Digest, Vol 48, Issue 89

[Rafael Cresci <rafael () cresci org>]

On 25/01/2012, at 20:51, nanog-request () nanog org wrote:

> Message: 4
> Date: Wed, 25 Jan 2012 15:11:42 -0500 (EST)
> From: Jay Ashworth <jra () baylink com>
> To: NANOG <nanog () nanog org>
> Subject: Equinix Miami 1 condemnation
> Message-ID:
>       <22786388.6606.1327522302701.JavaMail.root () benjamin baylink com>
> Content-Type: text/plain; charset=utf-8
>
> Last week, we saw some traffic about the Lightfiber problems because EqM1 
> is apparently in a building that's been condemned by the city or county
> of Miami.
>
> That struck me curious, so I wanted to look into it further.
>
> Amazingly, there doesn't seem to be any coverage of the incident, even in
> technical circles.  Does anyone know anything they're permitted to tell
> about how a building which contained a datacenter managed to get itself 
> condemned?
> This was the Metro Mall facility, operated by Switch and Data until
> Equinix's acquisition of S&D a few years ago.  It is now known as
> Equinix-Miami 1.  1 NE 1st Street.
And...

> The issue here has nothing to do with any hurricanes in the past.  My
> opinion is that they lacked proper documentation and planning.  Having a
> bit of insider knowledge here and knowing where this fiber came from
> originally (ACSI) and how FiberLight came to acquire it, I'm fairly sure
> I'm on the money.

I have no NDA on this so I can talk better in details.

Equinix Miami 1 (formerly Switch & Data, formerly another company name I don't recall now) was the temporary building 
that the NAP was built on, before moving to the current location that Terremark is (if not wrong, they called it the 
"T-Rex project" and the NOTA exchange started there in that place). Thus, (nearly) every major fiber strands that come 
from the submarine cable landing stations in SFl come into EQX MIA1 first, physically, before going into Terremark at 
9th Street.

The building itself (Metromall) is not a datacenter by design, it's a normal commercial building. Some handfuls of 
jewelry stores, coffee shop, travel agency, a hair dresser, in the lobby and ground and basement levels. The loading 
dock of the building is on the same alley as Equinix Miami 2 (36 NE 2nd Street/Telesource 
building/Telx/Colohouse/Fibermedia/Savvis) loading dock entrance. Global Crossing's datacenter also was in the 
Metromall at another floor before going into Terremark's 6th floor. Equinix/S&D was on the 5th floor (whole floor). 
They were the biggest lease there, but they had no word on the building administration or maintenance, only on their 
own floor, that was pretty empty (no more than two or three dozen of customers). S&D considered Miami more of a 
strategic network PoP than a real customer datacenter (as does Savvis, that has a full floor in the 36 NE 2nd St 
building, but no customers, and doesn't announce it as a datacenter/colocation PoP even having all that square footage 
and huge generators), they had only a small number of customers and the two facilities (MIA1 and MIA2) were 
acquisitions of former small/broken telco companies. They were not built by S&D, less yet by Equinix.

The building failed the 40-year inspection by the city of Miami. The landlord/owner of the building decided not to fix 
the numerous issues that were presented by the inspectors, and then decided to just let it go, close everything, expel 
all stores (mostly were on monthly leases) with little to no notice, and maybe sell the building or the ruins to 
someone who wanted it to fix or to demolish.

Equinix (or better, Switch and Data) was in negotiations with a company I worked for as an outsourced partner, to build 
a pipe between both buildings, when they faced a long delay on approval by the landlord, When they pressed, they got 
the real reason why he was holding it. Last June Equinix started to evacuate the building preemptively, moving all 
customers to Miami 2 (it's in the same block, just two corners or a backdoor away) and paying their moving costs 
(materials, manpower, telco circuit movings) up to a ceiling of $10k (in service credits, not cash, power install was 
covered by Equinix). They did some preparation work on Miami 2 for receiving these customers - not many - and started 
that, with an ETA to move everybody out by October 2011.

The "secret" ingredient was more of a "let's not annouce it so no one tries to signup to Miami 1 and then take 
advantage of service credits", until the unavoidable was unavoidable and until Miami 2 was ready for deployment of 
these customers. Only the customers, partners and the Digital Realty Trust management (Miami 2 is a DRL 
building)/security were aware of that while the move was not completed or at least mass moved. Some competitors got to 
the knowledge after the customers were notified (in June 2011) and some customers leaked that information by accident 
or intentionally at that time.

AFAIK, Fiberlight spliced all Fiber on the manhole during 3 nights in July/August 2011, to reroute them out of MIA1. 
What happened these couple days ago was something completely different and probably the result of something they forgot 
to do on that manhole, or pure stupidity of doing maintenance/upgrades on both sides of the ring at the same time.

-- rc