Re: puck not responding

[Jared Mauch <jared () puck nether net>]

> On Feb 28, 2024, at 1:30 AM, Daniel Marks via NANOG <nanog () nanog org> wrote:
>
> We’re getting rocked by storms here in Michigan, could be related.

[ brief version of what happened from what I can tell reconstructing things]

I was alerted ~4am US/E yesterday about the issue.  This machine has been generously hosted by my previous employer for 
quite some time, funnily enough it was 7 years ago almost to the day since I started my current employment.

The IPMI was not responsive and the machine was located in 350 Cermak, on a floor that was not impacted with the 
heat/cold event.

I have been meaning to move things off and on, but never quite had the motivation to tackle the task.  Yesterday forced 
my hand.

Once I confirmed that we could get the machine out of the colocation facility (thank you again NTT) I drove from 
Michigan to Chicago, got lunch and picked up the machine and headed back to the colocation that I have in Michigan at 
the 123Net/DetroitIX site.

Once I had a console on it, I determined that this old machine had a few things that had been gradually updated and 
upgraded over time, not all the filesystem options were set correctly and after some tune2fs options were set and fstab 
updated to ensure everything is migrated fully from ext2 -> ext4 the system was able to be booted without issues.

Afterwards I’ve determined that there is still a hardware related problem, so I am now going to move it to new hardware 
later today schedule permitting as I want to go onsite and make sure that the I/O is performant.

Feb 28 22:09:05 kernel: Memory: 32816872K/33544380K available (20480K kernel code, 3276K rwdata, 14748K rodata, 4588K 
init, 4892K bss, 727248K reserved, 0K cma-reserved)
Feb 29 00:20:07 kernel: Memory: 16326408K/16767164K available (20480K kernel code, 3276K rwdata, 14748K rodata, 4588K 
init, 4892K bss, 440496K reserved, 0K cma-reserved)

Not quite a great thing when nobody is onsite and the machine requires being power cycled and the amount of memory 
changes.

If you are seeing any other issues, do let me know, I did move the IPv4 space but have renumbered for v6, so if you use 
my free secondary dns service, and your own vanity name, you will need to update your AAAA records.

If you are seeing any reachability issues let me know, there should be ROA and other objects in place for things.

Sorry everyone got this email, feel bad it’s like when warren asked the list some personal details :-)

- Jared


(Even more details: changing disk images from qcow -> qcow2 and other things like ext2 -> ext3/4 over all the years as 
the machine has gone from Linux -> FreeBSD -> Linux again and other things is always a fun way to keep bringing your 
legacy around with you, it’s good overall)