DNF Automatic with healthchecks.io Integration

I've started getting lazier and lazier when it comes to doing SysAdmin work, but don't misunderstand me here.  That just means I want to automate myself out of the picture (mostly) and just get alerts if things aren't quite right.

One particular workflow I'd like to automate is patching my servers.

This used to be something I enjoyed doing manually.  SSHing into the target server, running apt/dnf/yum/pacman/whatever and watching the updates roll by was almost therapeutic.

Not so much anymore.  I just want it to happen automatically and report back in if something went afoul.

This is where dnf-automatic comes in.

Alternative CLI to dnf upgrade with specific facilities to make it suitable to be executed automatically and regularly from systemd timers, cron jobs and similar.

Quick Note: I do realize that automatic updates could be a source of pain, and I speak a bit more to that point later in this post, but I'm willing to take the chance with my personal systems.

Make sure you understand the risks if you take this post and run with it...and have you tested your backups lately?

Installing

I'm assuming you are on a Rocky8 server (or whatever CentOS8/RHEL8 flavor), so installing dnf-automatic should be a quick dnf away.

dnf install dnf-automatic

Configuring

Now that you have dnf-automatic installed, open /etc/dnf/automatic.conf with your favorite ( *cough  vim cough* ) editor and update to suit your needs.

The main items I was concerned with was upgrade_type (which I kept as default), apply_updates (which I changed to yes), and emit_via (which I changed to motd). Everything else I left as-is.

Here is a quick example of what my config file looked like after my changes (and removing all comments).

[commands]
upgrade_type = default
random_sleep = 0
network_online_timeout = 60
download_updates = yes
apply_updates = yes

[emitters]
emit_via = motd

[email]
email_from = root@example.com
email_to = root
email_host = localhost

[command]

[command_email]
email_from = root@example.com
email_to = root

[base]
debuglevel = 1

Adding in Healthchecks

I first started down the path of trying to shoehorn the health checks into the config file above under the section [command] but gave up pretty quickly after attempting, and failing, to understand what that section actually does.

My next attempt relied on the assumption that there would be some fanciness built-in to SystemD that I could leverage here, and after some quick DuckDuckGoin', I had a possible answer.  ExecStartPre and ExecStopPost.

At this point, I am going to assume you already have a https://healthchecks.io/ system in some way shape or form up and running, have a project defined with a new check, and the ping URL of said check.

Now, we want to add in our new settings to dnf-automatic.service without mucking up system files that may get updated, modified, or otherwise changed during updates, so we want to leverage the override abilities in SystemD.

Run systemctl edit dnf-automatic.service.  This should create a fancy new file in /etc/systemd/system/dnf-automatic.service.d/override.conf that will contain the settings we want to add or override to the already defined service.  You should have a new editor opened with a blank file to which you can add the following.  Make sure to update %%YOUR_PING_URL%% with the ping URL for the check you are creating/using in your HealthChecks.

[Service]
ExecStartPre=curl -m 10 --retry 5 %%YOUR_PING_URL%%/start
ExecStopPost=curl -m 10 --retry 5 %%YOUR_PING_URL%%/${EXIT_STATUS}

Item	Reason
ExecStartPre	Run this command before the item defined in ExecStart. We use this to inform our HealthChecks system the service is being started.
ExecStopPost	Run this command when the service exits. We use this to signal HealthChecks that the service exited and include its exit status

A quick note on the setting for ExecStopPost.  This references the variable ${EXIT_STATUS}.  More information on how ExecStopPost and ${EXIT_STATUS} works can be found here and here.  Also, there are some cases that this variable will be a string instead of a numeric exit code...and that may throw a wrench into things.

In most cases though, it will be set to a number, so I'm calling it 'good enough' for now.  I could probably get fancy with a BASH one-liner in there that attempts to catch those cases, but I'll cross that bridge if I get there.

Now that we have all the bits in place, we can enable the dnf-automatic.timer, which is the bit that will fire off the dnf-automatic.service at a regular interval.  

systemctl enable dnf-automatic.timer

You can also fire off the dnf-automatic.service manually to trigger a run.  Assuming all goes well, your HealtChecks should get a ping or two.

systemctl start dnf-automatic.service

Quick Warning

All this being said, I'm still early days on deploying this solution, so I could have something wrong here.  If that is the case, I will update this post accordingly, but as of right now it seems to be working as intended.

OH!  And you know how sometimes you hear of someone who interrupted system updates and now has a broken system?  I assume this is a possibility on my systems as well and have taken steps to lessen the chance this happens.  It all boils down to schedules.

If, for example, you have a weekly backup job that your hypervisor kicks off (that also shuts down your systems first to take a cold snapshot), you may want to adjust the SystemD timers accordingly so that they are far enough away from that as possible.  Really, I would hope that the system running the updates would inhibit its ability to shutdown during said updates...but I'm nothing if not paranoid.

Bonus Points

Just dug into how to do it with APT on Ubuntu 20.04.

In Ubuntu Land, it's called Unattended Upgrades and requires the installation of the package unattended-upgrades.

apt install unattended-upgrades

Then edit /etc/apt/apt.conf.d/50unattended-upgrades.  You will need to update Unattended-Upgrade::Allowed-Origins if you want anything more than just security updates.  Mine ended up looking like this:

Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
        "${distro_id}ESMApps:${distro_codename}-apps-security";
        "${distro_id}ESM:${distro_codename}-infra-security";
        "${distro_id}:${distro_codename}-updates";
        "${distro_id}:${distro_codename}-proposed";
        "${distro_id}:${distro_codename}-backports";
        "Zabbix:focal";
};

You may also want to take a gander at the setting for Unattended-Upgrade::Automatic-Reboot.  I don't want my systems rebooting on me so I left it as false/commented out.

Next, I edited /etc/apt/apt.conf.d/20auto-upgrades and updated it to:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";

As far as I could tell, the SystemD service doing the legwork here was apt-daily-upgrade.service so I added the same two lines to the override file via systemctl edit apt-daily-upgrade.service.

[Service]
ExecStartPre=curl -m 10 --retry 5 %%YOUR_PING_URL%%/start
ExecStopPost=curl -m 10 --retry 5 %%YOUR_PING_URL%%/${EXIT_STATUS}

You can read more about Automatic Updates from the Ubuntu Docs on the subject