Migrating a Business to Linux — 12
One aspect of our migration that caused some frowns is updating kernels. I prefer to update systems from a terminal window and command line, but co-workers are pointy-clicky folks. When the Ubuntu update notifier pops open, they presume to update using the GUI notifier.
All fine and good except for a bug with updating GRUB on encrypted partitions. The bug affects grub.cfg, introducing a bogus cryptomount -u. The result is upon booting, the system shows a Press any key to continue message and then fails to prompt for a LUKS passphrase. The boot process drops to an initramfs shell.
Related, the initrd is not compiled correctly.
I asked co-workers to let me update the systems until I found a solution. Not palatable long-term. At all.
I already had written a custom /usr/local/sbin/update-grub script to remove the bogus cryptomount -u entries. As I configure systems with /usr/local/binpreceding /usr/bin in the $PATH environment variable, the script is always launched rather than the default. This resolved the Press any key to continue message.
On next to resolving the initrd problem.
I had already created /etc/initramfs-tools/hooks/cryptsetup, which contained the following:
CRYPTSETUP=y
export CRYPTSETUP=y
I also had added the same export in /etc/environment:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
CRYPTSETUP=y
Yet when updating the system, somehow the initrd failed to compile correctly.
My next attempt was to add a custom script called /usr/local/sbin/fix-boot and run that script on shutdown/reboot with a systemd fix-boot.service unit file. Basically the script is a wrapper to the update-grub script and update-initramfs command. The script runs those commands only if there was a kernel update. The script explicitly contains export CRYPTSETUP=y.
Running the script manually always worked great. The script often failed to launch on shutdown or reboot.
I never could get the fix-boot.service unit file to launch with certainty. About half the time the system never bothered running the unit file, almost as though the system was using some kind of shortcut to rebooting or shutting down. I don’t know whether this is an Ubuntu glitch or systemd. Hugely frustrating. To me systemd remains a fuster cluck. The whole self-centered approach of running everything in parallel is silly. There are times when users want specific tasks launched sequentially or in a specific order. The systemd design does not even support running rc.local last in the boot sequence. I searched many discussions about running scripts at reboot/shutdown and none of the discussions helped fully. After several days of futility I abandoned this approach.
My next attempt was to modify the behavior of how Ubuntu updates. Initially I thought a file in /etc/apt/apt.conf.d would suffice. Specifically, adding a custom file in /etc/apt/apt.conf.d to run the script using APT::Update::Post-Invoke. Yet I suspected that too would fail, which was the case. Basically then my script ran during the apt update phase, which was useless for my use case. The apt command does not support APT::Upgrade::Post-Invoke.
Traveling down the road I read of another idea. I decided to try running my fix-boot script only when updating kernels. That meant launching my script from within /etc/kernel/postinst.d.
The idea looked good and I had partial success. Yet I needed the script to run only once and the script was being launched with each individual kernel package.
I looked at the existing scripts in /etc/kernel/postinst.d. A quick test revealed the zz-update-grub script was indeed using my custom /usr/local/sbin/update-grub rather than the stock script. Looking at /etc/kernel/postinst.d/initramfs-tools caused the proverbial light bulb in my head to glow a little. I modified the upstream script by adding export CRYPTSETUP=y at the beginning of the script.
That seemed to resolve the problem of the initrd compiling correctly.
That missing environment variable taunted me. Digging deeper, I found a related bug report. A patch was applied to cryptsetup 1.7.2. Ubuntu MATE 16.04 is using 1.6.6-5.
I had a file in /etc/initramfs-tools/hooks/cryptsetup containing the variable export. The file was not being sourced correctly. According to the bug report, exporting the variable in /etc/initramfs-tools/initramfs.conf would resolve the problem.
I restored the original /etc/kernel/postinst.d/initramfs-tools and modified /etc/initramfs-tools/initramfs.conf.
So far so good. Hopefully this approach allows co-workers to update systems and forget about the reboot bug.
In the end a simple solution. I traveled deep into the rabbit hole to find that simple solution. This is an example why Linux suffers on the desktop.
Posted: Category: Usability Tagged: General, Migrate, Ubuntu
Next: Migrating a Business to Linux — 13
Previous: Subtle Tracking