Safely Halting or Rebooting
The idea of a safe shutdown shell script is hardly far-fetched. Browsing the web indicates many people have looked for a way to prevent or delay a system halt or reboot during backups or other critical processes.
For many years I have used a safe-shutdown shell script. I created the script to ensure a system is rebooted or halted in a safe manner. That is, only when specific criteria are satisfied. If any condition fails then the script exits and /sbin/shutdown [-r|-h] is not executed. Some existing processes do not cause the script to exit but instead displays a progress indicator and the script passively waits for the process to complete.
The script was born when I built a dedicated Home Theater PC (HTPC), some seven years ago or so. I installed XBMC for playback. After configuring the remote control unit, I did not want to inadvertently press the Power button and have the system power off in the middle of watching a video. The Power button is near the Pause and Stop buttons.
From there the script evolved and matured. For several years I ran my office desktop as a pseudo server. I used the safe-shutdown script to ensure the system was not rebooted or halted with any connected network clients.
When I built a dedicated LAN server I tweaked the script for that new purpose. Through a cron job I power down the server at night when no clients are connected. Occasionally I burn some midnight oil and this script ensures the server remains available until I quit for the night.
The safe-shutdown script now includes several ways to check for connected clients, ensures currently running backups and cron jobs complete and are not interrupted, as well as safely powering down headless VirtualBox virtual machines.
The script supports logging, which helps as a debugging tool.
I wrote support wrapper scripts to pass the appropriate parameter to the /usr/local/sbin/safe-shutdown script.
- /usr/local/sbin/safe-halt
- /usr/local/sbin/safe-hibernate
- /usr/local/sbin/safe-reboot
- /usr/local/sbin/safe-suspend
All of this works well with Slackware or any other /etc/inittab system.
My modifications of /etc/inittab and the login manager do not cover all possible ways to reboot or halt a system. They do nicely cover most use cases.
Enter stage-right with something called systemd.
A cornerstone design goal of systemd is to launch or halt services and tasks in parallel. This is part of the heart of systemd.
Faster boot and halt/reboot times are reasonable goals but in my world not the alpha and omega of computers. My experience with several distros using systemd is boot and halt/reboot times are not faster than an init system with shell scripts. In my use case boot times are slower than shell scripts, while halt/reboot is a tad faster.
My safe-shutdown script is sequential. The script is fast enough not to cause any waiting pain during a reboot or halt. Thus running everything in parallel is not important to me. I much prefer booting, reboots, and halts occur in a known order.
Since my first introduction to systemd with Fedora and CentOS 7 a few summers ago, I have been unable to find a way to run the safe-shutdown script using the same triggers I used in non systemd systems. Fundamentally, running the safe-shutdown script required modifying /etc/inittab and the login manager. That is not possible with systemd.
Nowadays critical programs are hard-coded to support systemd when systemd exists on a system. Remapping keyboard shortcuts and login managers has proven a challenge. The only way to invoke my safe-shutdown script is through systemd. Yet systemd itself is hard-coded with presumptions, such as using Ctrl+Alt+Del.
On non systemd systems I am able to remap the following:
- Login manager -> safe-halt, safe-reboot
- Desktop session manager -> safe-halt, safe-reboot
- Ctrl+Alt+Del -> safe-reboot
- Ctrl+Alt+End -> safe-halt
- Power Button -> safe-halt
I have been more than content with this configuration. Returning to my original motivation to prevent inadvertent halts on my HTPC, I never once experienced such an event. The safe-shutdown script works as intended. Moreso now that I use the script with my dedicated server.
Since my introduction to systemd I have been watching for ideas or tips to resolve this problem. I have spent more than a few search sessions looking for a way to run my safe-shutdown script first, before systemd tries to reboot or halt.
All of this might have been simpler if systemd used scripts for the /sbin commands or C code that passed parameters to systemctl. Using sym links is an awkward design. “Mission creep” has been a common complaint about systemd.
There is a mechanism called systemd-inhibit. This feature is basically a wrapper to other processes. The feature does not allow me to do what I want. I want only to run my safe-shutdown script. I do not want systemd attempting to halt or reboot at all unless my safe-shutdown script exits successfully. I want systemd to remain completely ignorant of this safe-shutdown process.
Recently I discovered a package known as molly-guard.
I was intrigued by this package because the function is similar to my safe-shutdown script. The molly-guard package has been around many years. Perhaps in my own way I reinvented the wheel, but my safe-shutdown script works the way I want.
I noticed molly-guard was available for Ubuntu MATE. I wondered how molly-guard supported systemd, if at all.
Apparently the trick is to move the /sbin commands to an alternate directory, replace those commands with sym links to molly-guard, and the molly-guard script launches the moved /sbin commands when no errors occur.
I do not need to install molly-guard. I only need a way to intercept systemd reboot and halt triggers to always run my safe-shutdown script. That way systemd never gets involved until satisfying safe halt and reboot conditions. Seems I could use a similar approach as that used by molly-guard.
There is a challenge. I am not fond of replacing system files. Doing so is always a sledge hammer approach. Any systemd update means reconfiguring a system because a systemd package update likely will clobber the replacement files. Likely I would need a repair script to ensure any systemd system I use is configured the way I want and not with the systemd commands. Perhaps a cron job could be added to ensure my configurations remain intact.
My safe-shutdown script calls /sbin/shutdown [-r|-h]. On a systemd system that is a sym link, /sbin/shutdown -> /bin/systemctl. I do not have to modify my safe-shutdown script at all with special systemd commands. I have tested this many times from the console command line.
Seems I need to replace the following systemd sym links:
- /sbin/halt -> /bin/systemctl
- /sbin/poweroff -> /bin/systemctl
- /sbin/reboot -> /bin/systemctl
- /sbin/runlevel -> /bin/systemctl
- /sbin/shutdown -> /bin/systemctl
- /sbin/telinit -> /bin/systemctl
Possibly the following commands too:
- /usr/sbin/pm-hibernate -> /usr/lib/pm-utils/bin/pm-action
- /usr/sbin/pm-powersave
- /usr/sbin/pm-suspend -> /usr/lib/pm-utils/bin/pm-action
- /usr/sbin/pm-suspend-hybrid -> /usr/lib/pm-utils/bin/pm-action
In my non systemd systems I never renamed or moved any of the /sbin commands. Leaving those commands as-is provided me an low level admin method to bypass my safe-shutdown script and force an immediate reboot or halt. All I needed was to modify /etc/inittab and the login manager. I would not be able to do that in systemd because of the sym links to /bin/systemctl. I would have to replace the sym links.
One of my early attempts at resolving this problem was adding sym links of /sbin/halt and sbin/reboot in my /usr/local/sbin directory. My $PATH environment variable gives priority to /usr/local/sbin over /usr/sbin and /sbin. The challenge is the hard-coding within programs. The sym link trick works only from the command line.
Because systemd runs as much as possible in parallel, other past attempts at resolving this problem failed because my /usr/local partition would get unmounted. With hard-coding the /sbin commands directly to my /usr/local scripts, I should not see that happen. My safe-shutdown script should always be executed first. That was my theory.
I created a new sym link /sbin/reboot -> /usr/local/sbin/safe-reboot. I tested the following.
- Rebooting (Restart) from the MATE desktop.
- Rebooting from the Light DM login manager.
- Toggling to a console and pressing Ctrl+Alt+Del.
None of the tests succeeded. My guess is all of the options are hard-coded directly to systemd rather than to the /sbin/reboot command.
To use an old expression, this sucks.
While I use Slackware with my LAN server and not any systemd distro, I still want to use my safe-shutdown script on systemd workstations. I tend to tinker a lot as well as run regular backups to the server. I do not want those tasks interrupted.
At this point I remain stumped to properly execute my safe-shutdown script on systemd. An obvious option is stop using systemd. That idea is a strong contender for future decisions.
Posted: Category: Usability Tagged: General
Next: Chromebooks
Previous: Isolating Untrusted Virtual Machines — 2