How to Get Notified When a systemd Service Fails on Linux

You get notified when a systemd service fails by adding an OnFailure= directive to the unit, pointing it at a small notifier service that posts the failing unit's name and exit status to a webhook or push notification the moment systemd marks the unit failed. systemd is the init system and service manager on most modern Linux distributions, and it already tracks every service's failure state internally — the gap is that nothing reads that state and tells a human. Once the OnFailure= unit is wired up, a crashed service reaches your phone within seconds instead of showing up as a stale failed line in systemctl status three days later.

Why Do Failed systemd Services Go Unnoticed?

By default, systemd does exactly one thing when a service fails: it flips the unit's state to failed and writes the exit reason to the journal. Nobody is watching either of those places in real time. On a small VPS running a handful of services — a backup script, a queue worker, a scheduled sync — the only way you find out something died is by manually running systemctl --failed, which most people do after they've already noticed something is broken downstream.

That is the same blind spot as a cron job that silently stops running: the absence of output looks identical to the absence of a problem. The fix is the same shape too — make the failure state push itself to you instead of waiting for you to poll it.

What systemd Already Tracks for You

• Exit code and signal for the last run of every unit
• Whether a unit is in active, failed, or activating state
• Restart counts, if Restart= is configured
• A timestamped journal entry for every state transition

How Does systemd's OnFailure= Directive Work?

OnFailure= is a unit directive that names one or more other units to start whenever the unit it's attached to enters the failed state. It runs automatically, with no polling and no cron job checking in on a timer — systemd itself triggers it as part of the state transition. That makes it the correct primitive for failure alerting, rather than a script that periodically greps systemctl --failed.

Add it to the service you want to watch:

[Unit]
Description=Nightly backup job
OnFailure=notify-failure@%n.service

[Service]
ExecStart=/usr/local/bin/run-backup.sh

Then define the notifier as a templated unit, so the same one handles every service that references it:

[Unit]
Description=Send failure alert for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure.sh %i

%i is the failed unit's name, passed in from %n on the caller. Your notifier script receives it as an argument, so one small script covers every service on the box.

Adding OnFailure= Without Editing a Vendor Unit File

You don't have to edit a package-managed unit directly — that copy gets overwritten on the next update. Use a drop-in instead:

sudo systemctl edit nginx.service

That opens an override file at /etc/systemd/system/nginx.service.d/override.conf. Add just the two lines you need:

[Unit]
OnFailure=notify-failure@%n.service

systemd merges the drop-in with the original unit at load time, so the alert survives package upgrades and you never touch the vendor file. The same drop-in pattern works for any unit already installed on the box, not just ones you wrote yourself.

How Do You Send the Alert Somewhere You'll Actually See It?

A shell script that fires on failure is only useful if what it sends reaches you outside the terminal. The most direct path is a single HTTP call to a webhook from inside notify-failure.sh:

#!/bin/bash
unit="$1"
curl -sS -X POST https://api.pingwire.dev/v1/messages \
  -H "Authorization: Bearer $PINGWIRE_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"channel\":\"ops\",\"text\":\"systemd unit failed: $unit\"}"

That is the same one-HTTP-call pattern covered in sending a push notification from a bash script — the only thing that changes here is what triggers the script. systemd calling it on a state transition is more reliable than a cron job polling for the same thing, because it fires exactly once, exactly when the failure happens, with no gap between the crash and the check. Pingwire's developer API takes that POST and delivers it as a real-time push to a phone or desktop, with the failing unit name in the body so you don't have to SSH in just to find out what broke.

OnFailure= vs Other Ways to Catch a systemd Failure

OnFailure= is not the only option, and it is not always the right one on its own. Here is how the common approaches compare:

Method	Fires immediately	Setup effort	Catches a hung process that never "fails"
OnFailure= unit	Yes	Low — two small unit files	No
Cron job polling systemctl --failed	No — up to one poll interval late	Low, but adds a second moving part	No
Journal log scraping	Depends on scrape interval	Higher — needs a log shipper	No
External heartbeat / dead man's switch	Yes, on missed check-in	Low — one curl call per run	Yes

What About Services That Fail Silently Without Crashing?

OnFailure= only fires when systemd itself decides the unit failed — a non-zero exit code, a crashed process, a timeout you configured. It does nothing for a service that is still technically running but has stopped doing useful work, or a scheduled job that simply never starts because the timer unit itself misfired. For that class of failure you need the opposite signal: something the job checks in with on success, where the absence of a check-in within a window is itself the alert. That is heartbeat monitoring, and it pairs well with OnFailure= — the same idea already covered for catching a failed GitHub Actions workflow applies here: crash alerts and silence alerts are two different problems, and a service that matters usually needs both.

Isn't This Overkill for a Small VPS or Side Project?

It's a fair question if you're running one server with three services on it. The honest answer: for a single unit you don't care about, skip it. But the setup cost is genuinely small — two unit files and one script, reused across every service on the box — and the alternative isn't "no risk," it's "you find out from a user, a bounced payment, or a missed backup instead of from systemd itself." The failure detection was already happening inside systemd for free; OnFailure= just routes it somewhere you'll see it before the consequence does.

The other common objection is maintenance: one more moving part someone has to remember exists. That's a real cost, which is why the templated notify-failure@.service pattern matters — you write and debug the notifier once, then add a single OnFailure= line to any unit you want covered. There is no per-service script to maintain, and no separate daemon to keep running; systemd itself does the triggering, using a mechanism that's already part of the base install on every current distribution.

Set Up Your First systemd Failure Alert

1. Add OnFailure=notify-failure@%n.service to the [Unit] section of the service you want to watch.
2. Create /etc/systemd/system/notify-failure@.service as a oneshot unit that runs your notifier script with %i.
3. Write the notifier script to POST the unit name to a webhook, and mark it executable.
4. Run systemctl daemon-reload so systemd picks up the new units.
5. Force a real failure for a test — for example, temporarily point ExecStart at a command that exits 1 — and confirm the alert arrives.

Create a Pingwire API key from the developer page and send yourself a test notification first, before wiring it into OnFailure=; it takes about a minute and confirms the delivery path works before you rely on it for a real crash.

Frequently Asked Questions