Re: 365 Datacenters Tampa AC Failure

[Nick Olsen <nick () 141networks com>]

James, Any word on how redundant systems failed at the same time? Or why
the secondary was untested and not known-working?

It seems like HVAC maintenance was done by the night crew which was
quietly eliminated at some point.

On Tue, Jun 13, 2023 at 12:02 AM James Ashton <james () gitflorida com> wrote:

> Hello all,
>  From our side (365 Data Centers) We saw the issue start at a little after
> 6:30 PM yesterday. We had a component failure and we saw a rise in
> temperature. We brought additional capacity online while we were working
> with our vendors on getting replacement hardware.
> The hardware has now been replaced and the system is back online.
> This unit and others in this suite are on the block for replacement this
> year as we add cooling capacity to our various spaces within the Franklin
> Exchange building.
>
> On a side note, this was not related to the issue mentioned from earlier
> in the month. This was a straight component failure and replacement.
>
> Any customers impacted can reach out to the 365 Customer Service Center
> for details and documentation.
>
> Thank you,
> James Ashton
> VP Network Engineering
> 365 Data Centers
>
>
>
> ------------------------------
> *From: *"NANOG mailing list" <nanog () nanog org>
> *To: *"George Herbert" <george.herbert () gmail com>
> *Cc: *"NANOG mailing list" <nanog () nanog org>
> *Sent: *Monday, June 12, 2023 8:03:10 PM
> *Subject: *Re: 365 Datacenters Tampa AC Failure
>
> Issue started a little after 2am, I was hard down till about 11:30am
> (servers did a high temp shutdown)
>
> On Jun 12, 2023 8:01 PM, Michael Spears <michael () spears io> wrote:
>
> Yep there's issues over there. They had some compressors go down. Should
> be getting back to normal now... Hasn't been a good month for them in
> regards to cooling..
>
> On Jun 12, 2023 7:15 PM, George Herbert <george.herbert () gmail com> wrote:
>
> Oof.  Get ready to replace all spinning media you may have there.
>
> -George
>
> Sent from my iPhone
>
> > On Jun 12, 2023, at 4:06 PM, Nick Olsen <nick () 141networks com> wrote:
> >
> >
> > Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently
> seeing floor temps of ~105F as reported by equipment. Started yesterday at
> ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any
> advisory notices as of yet.