My server maintenance has been on autopilot for a bit now. Every week or so I log in, run updates, and check in on this and that. For my last birthday I had a virtual party, and I used this machine to temporarily host some photos, which wound up requiring some temporary tweaks to my nginx configuration. By and large, I haven't changed or implemented anything major in a while.
It turns out I had become complacent for too long — Ubuntu 16.04 LTS has just lost its support! I found this out from an apt message during an upgrade telling me that I should subscribe to something for continued security updates.
Enable UA Infra: ESM to receive additional future security updates.
(It took a bit of Googling to connect the dots on what this meant. Why can't they just send you a straightforward “You should upgrade your distro” message?) I backed up the machine, stopped all services, and tried to run a dist-upgrade. I immediately ran into an error, something about a missing package. I tried the first fix my google search suggested, and that borked too. That's when I panicked about the possibility of losing incoming emails and restarted the server.
Faced with the possibility of trying dist-upgrade again, perhaps late at night, and struggling with it for an unknown amount of time, I decided to change my approach. I bought a new Linode with a newer LTS image and started migrating. My thought process was that I can get it up and running while the old machine keeps doing its job, so even if I have a lot of work to do, I'll minimize the downtime. I can even add the new machine as a subdomain on my DNS record to test out the web capabilities.
It turns out this is also what the ISPmail tutorial at workaround.org recommends. (It's so great to see Christoph is maintaining it and adding new versions!) Working through it again was much less of a headache than trying to debug a broken upgrade process. The former was familiar and reassuring in all the ways the latter was unfamiliar and scary. I got to make a few configuration changes I had never gotten around to on my previous machine: adding Ubuntu's unattended-upgrades package, improving my fail2ban filters, and more like that. Having the old server live while I worked on the new one gave me the time to taking a couple of days and double check certain things. Then I just updated my DNS records to point to the new machine when I was ready to cut over. After everything was done, I had an image of my old server on a hard drive in case I ever needed to refer to something, and I had a fresher, leaner server running everything I needed.
Overall I'm really glad I did it this way. I'd even recommend it to other hobbyists like me. On some level, setting up a server is a skill like any other. It's worth practicing. Most of us don't just set up servers all the time, though, because we really want to do other things with them. But then there are times when a problem comes up that sends you back into some part of your system setup that you don't recognize, and you think …
"What did I do this for?"
"Is that a default or something I changed?"
"Man, I wish I had taken better notes on this piece!"
Personally, I think periodically rolling up my sleeves and setting up a new system is well worth the time. I feel much more confident in being able to keep doing this. I think I'll try this again when it's time for the next upgrade.