nanog mailing list archives

Re: Best TAC Services from Equipment Vendors


From: Crist Clark <cjc+nanog () pumpky net>
Date: Tue, 12 Mar 2024 22:24:59 -0700

I've been reading the "${VENDOR}'s support has really gotten worse lately"
threads for pretty much every vendor for the past twenty years. That's not
to say they've all been wrong. But it reminds me of those quotes you'll see
about how "these kids today are awful and society is going to pot" and then
the big reveal is that it was written in the 1950s, or 1920s, or just
before the peak of Rome, or something like that. The general tendency for
people to view the past as the good ol' days.

My most memorable Cisco TAC disaster story. Taking away "configure" from
TAC wouldn't have saved us. The guy simply reloaded the switch without
asking. The core switch for a building with hundreds of end users. In the
middle of the day. The building with most of the C-level execs. Our
management was pi-i-i-issed. That got escalated pretty high, pretty
quickly. And quick policy change that we did not give TAC keyboard control.
This was about ten years ago.

On Tue, Mar 12, 2024 at 7:47 AM Lyden, John C <lyden () rowan edu> wrote:

when a TAC engineer wanted to bounce our Voice VLAN SVI in the middle of
an *airport* production day.
I about turned over my desk trying to wrest the remote control session
back from him before he hit enter
on the shut. Since then, I have had to go through a not insignificant
evaluation period of TAC engineers
before I let them take control of a remote session, and it is now simply
pure instinct to log SSH sessions.

Picture it, Cisco TAC, on a troubleshooting call, runs 'no ip routing' and
hits enter before our engineer could scream "NO" at 11:30AM on a core L3 on
a college campus.

RCA afterwards:

1. "Always log all terminals (we prefer SecureCRT) from Windows bastion
host to OneDrive or Google Drive"
2. New CiscoTAC TACACS login created allowing Enable but Denying
"configure" as a command. When you troubleshoot, you log in as CiscoTAC.

The CiscoTAC tacacs profile description in Clearpass makes it clear why
it's there. I left the curse words out.

-J

John C. Lyden
Associate Director, Network Operations
Division of Information Resources & Technology
Rowan University




Current thread: