A Monkey Off the Back

We have six single node Proxmox hosts at work. For the past several weeks I have been updating them from version 5 to 6.

I wrote a comprehensive shell script to automate much of the task. The script includes the following features:

Check the user is not performing the update remotely.
Remind the user to perform a clean and inspect.
Disable autostart on boot for all guests.
Remind the user to schedule Nagios downtime for the host and guests.
Prompt the user to perform the final Debian 9/Proxmox 5 updates.

Remind the user to save time by copying all new packages to /var/cache/apt/archives.
Launch the Proxmox pve5to6 script.
Automatically shutdown all guests.
Perform local vzdump backups.
Prompt the user to copy the vzdump backups to the local backup server.

Disable certain apt hooks that delay the updates.
Update the apt sources to Buster.
Perform an apt download-only to avoid potential network issues.
Perform the update.
Restore the apt hooks.

Restore autostart on boot for all guests.
Start all guests.
Remind the user to test the web browser interface.
Remind the user to verify all guest systems launched as expected.

And other nominal tweaks.

I updated one system per week, observing and adjusting for differences each time. A conservative “let the dust settle” approach. I fine tuned the shell script after updating each host.

The last host has the company mail server. At 250 GB of data I did not want the updating on that host to go afoul. In a worst case scenario restoring the mail server from backups would take hours.

Unlike the previous five hosts, because the mail server was involved I scheduled the maintenance window for a Saturday, outside business hours. I did not look forward to that but there are worst things in life to complain about.

Overall the updates went well.

After each update and reboot all systems were stable. Nobody complained. Not a single peep. That is the best reward.

Posted: July 28, 2020 Category: Usability Tagged: Virtual Machines

Next: GRUB Security Exploit

Previous: Writing Notes to Self