systemd and journald¶

By Sameer Alam Updated 20 October 2026

What You'll Learn¶

What systemd manages, and the unit types you'll meet most often
How to write a production-ready service unit for your own application
How to change a packaged unit safely with drop-in overrides
How to read and fix existing cron jobs, and when to replace them with timers
How to sandbox a service and query logs with journalctl

Mental Model¶

systemd is PID 1. It starts everything else, tracks every process a service creates in a cgroup, restarts services that fail, and collects their output in the journal. You describe what should run in unit files; systemd works out the order from dependencies.

Unit type	Describes	Example
`.service`	A process to run	`nginx.service`
`.timer`	A schedule that activates another unit	`logrotate.timer`
`.socket`	A socket that starts a service on first connection	`ssh.socket`
`.target`	A group of units, like a runlevel	`multi-user.target`
`.mount`	A filesystem mount	`srv-data.mount`

Unit files live in /usr/lib/systemd/system/ (from packages — don't edit) and /etc/systemd/system/ (yours — takes precedence).

Everyday Commands¶

systemctl status nginx                  # state, main PID, recent log lines
sudo systemctl start|stop|restart nginx
sudo systemctl reload nginx             # reload config without dropping connections, if supported
sudo systemctl enable --now nginx       # start now and at boot
systemctl is-active nginx; systemctl is-enabled nginx
systemctl list-units --type=service --state=failed
systemctl cat nginx                     # the unit file plus any overrides, exactly as systemd sees it
systemctl show nginx -p MainPID,Restart,LimitNOFILE

Write a Service for Your Application¶

/etc/systemd/system/orders-api.service

[Unit]
Description=Orders API
Documentation=https://wiki.example.com/orders-api
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=orders
Group=orders
WorkingDirectory=/opt/orders-api
EnvironmentFile=/etc/orders-api/env
ExecStart=/opt/orders-api/.venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
ExecReload=/bin/kill -HUP $MAINPID

# Restart on crashes, but stop retrying if it fails 5 times in 2 minutes
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=120
StartLimitBurst=5

# Graceful shutdown: SIGTERM, then SIGKILL after 30 seconds
KillSignal=SIGTERM
TimeoutStopSec=30

LimitNOFILE=65536
UMask=0027

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload            # required after creating or editing unit files
sudo systemctl enable --now orders-api
systemctl status orders-api

Choosing `Type=`¶

Type	Use when the program…
`simple` / `exec`	Runs in the foreground (most modern apps). `exec` reports failure if the binary can't start.
`notify`	Tells systemd it's ready via `sd_notify` (for example, some databases)
`forking`	Daemonizes itself and the parent exits (older software, like SonarQube's `sonar.sh`)
`oneshot`	Runs a task and exits — used with timers

Change a Packaged Unit With an Override¶

Never edit files under /usr/lib/systemd/system — a package upgrade overwrites them. Use a drop-in:

sudo systemctl edit nginx

/etc/systemd/system/nginx.service.d/override.conf

[Service]
LimitNOFILE=131072
Restart=on-failure

systemctl cat nginx          # confirm the override is merged
sudo systemctl restart nginx

To replace a list setting such as ExecStart=, clear it first with an empty assignment, then set the new value:

[Service]
ExecStart=
ExecStart=/usr/sbin/nginx -g 'daemon off;' -c /etc/nginx/custom.conf

Reading and Writing cron Jobs¶

cron is still on almost every server, and you'll inherit jobs written for it. Know where they hide and why they fail.

crontab -l                       # your user's jobs
sudo crontab -l -u backup        # another user's jobs
crontab -e                       # edit yours (validated on save)
ls /etc/cron.d/ /etc/cron.daily/ # system jobs installed by packages and admins
cat /etc/crontab

# ┌───────── minute (0-59)
# │ ┌─────── hour (0-23)
# │ │ ┌───── day of month (1-31)
# │ │ │ ┌─── month (1-12)
# │ │ │ │ ┌─ day of week (0-7, Sunday is 0 or 7)
# │ │ │ │ │
 30 2 * * *   /usr/local/bin/db-backup.sh           # 02:30 every day
*/15 * * * *  /usr/local/bin/sync-reports.sh        # every 15 minutes
  0 6 * * 1-5 /usr/local/bin/weekday-report.sh      # 06:00 Monday to Friday
@reboot       /usr/local/bin/warm-cache.sh          # once at startup

Files in /etc/cron.d/ and /etc/crontab have an extra field, the user to run as, between the schedule and the command:

/etc/cron.d/db-backup

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=""
30 2 * * * backup flock -n /run/lock/db-backup.lock /usr/local/bin/db-backup.sh 2>&1 | logger -t db-backup

That one line avoids the four classic cron failures:

Problem	Why it happens	Fix in the line above
"Works in my shell, fails in cron"	cron runs with a minimal `PATH` and none of your profile	Set `PATH`, or use full paths
Output disappears	cron mails output to a local mailbox nobody reads	Pipe to `logger`, then read it with `journalctl -t db-backup`
Two runs overlap	cron starts a new run even if the last one is still going	`flock -n` skips the run if the lock is held
A `%` in the command breaks it	cron turns `%` into a newline	Escape it as `\%`, or move the command into a script

cron's own log shows whether a job started: journalctl -u cron on Ubuntu, journalctl -u crond on RHEL.

Before trusting a schedule, look at the next few runs. The preview applies cron's actual matching rules, including the surprising ones:

Cron schedule preview (needs JavaScript). Without it, check a systemd timer expression with systemd-analyze calendar --iterations=5 '<expr>', and remember that when both day-of-month and day-of-week are set, cron runs when either matches.

Timers Instead of Cron¶

Timers log to the journal, don't run twice if a job overlaps, can catch up after downtime, and are visible with systemctl.

/etc/systemd/system/db-backup.service

[Unit]
Description=Nightly database backup

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
Nice=10
IOSchedulingClass=idle

/etc/systemd/system/db-backup.timer

[Unit]
Description=Run the database backup nightly

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now db-backup.timer
systemctl list-timers db-backup.timer
systemd-analyze calendar '*-*-* 02:30:00'   # check the schedule expression
sudo systemctl start db-backup.service       # run once now to test

Persistent=true runs a missed job at boot if the machine was off at 02:30. RandomizedDelaySec stops a fleet of servers from all hitting the database at the same second.

Sandbox a Service¶

systemd can remove privileges a service doesn't need, with no code changes:

[Service]
NoNewPrivileges=true
ProtectSystem=strict          # the whole filesystem is read-only...
ReadWritePaths=/var/lib/orders-api /var/log/orders-api   # ...except these
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
CapabilityBoundingSet=

systemd-analyze security orders-api     # scores exposure and lists what's still allowed

Add options one or two at a time and test — a too-strict sandbox shows up as EACCES or "Read-only file system" errors in the logs.

Query Logs With `journalctl`¶

journalctl -u orders-api -f                        # follow one service
journalctl -u orders-api --since "1 hour ago"
journalctl -u orders-api -b                        # since the last boot
journalctl -u orders-api -b -1                     # the previous boot — what happened before the crash?
journalctl -p err -b                               # errors and worse, all units
journalctl -k | grep -i -E 'oom|killed process'    # kernel messages, including OOM kills
journalctl -u orders-api -o json-pretty -n 1       # structured fields
journalctl --disk-usage

Keep the journal from filling the disk¶

/etc/systemd/journald.conf.d/size.conf

[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=14day

sudo systemctl restart systemd-journald
sudo journalctl --vacuum-size=500M      # one-off cleanup

Boot and Failure Analysis¶

systemd-analyze blame | head            # slowest units at boot
systemd-analyze critical-chain          # what the boot waited on
systemctl list-dependencies orders-api
systemctl reset-failed orders-api       # clear the start limit after fixing the cause

What Most Engineers Miss¶

Day of month and day of week are ORed. 0 9 1 * 1 runs on the 1^st and on every Monday, not "the 1^st if it's a Monday". The rule flips to AND if either field starts with *, even */2. This is POSIX behavior, not a bug.
Steps restart at each boundary. */45 in the minute field runs at :00 and :45, so the gap after :45 is 15 minutes. Steps are "every Nth value of the field", not "every N minutes".
cron runs in the system time zone, and daylight saving bites. Jobs scheduled between 01:00 and 03:00 can run twice or not at all on change nights. Servers in UTC avoid it; cronie supports CRON_TZ=, and Kubernetes CronJobs have .spec.timeZone.
cgroup v1 is gone. systemd 258 removed support for the legacy and hybrid cgroup hierarchies, and needs kernel 5.4 or newer. Old container runtimes, Java versions, and monitoring agents that read /sys/fs/cgroup/memory/... paths break on it.
SysV init scripts no longer start. systemd 260 (March 2026) removed the generator that turned /etc/init.d scripts and /etc/rc.local into units. Third-party software that only ships an init script needs a real unit file before you upgrade.
systemctl status exit codes are meaningful. It returns 3 when the unit isn't running and 4 when the unit doesn't exist, so systemctl is-active --quiet app || alert is a complete health check.

Common Mistakes¶

Editing a unit file and forgetting systemctl daemon-reload, so systemd keeps running the old definition.
Editing packaged units in /usr/lib/systemd/system instead of using systemctl edit, and losing changes on upgrade.
Using Restart=always without a start limit, so a broken service restarts in a tight loop and floods logs.
Setting After=network.target and expecting the network to be usable — use network-online.target with Wants=.
Keeping cron jobs that fail silently instead of timers whose failures show up in systemctl list-units --failed and the journal.
Leaving journald without size limits on small disks.

Check Yourself¶

The first Monday of the month

You want a job at 09:00 on the first Monday of each month and write 0 9 1-7 * 1. When does it run?

Only on the first Monday
On days 1 through 7 and on every Monday
Never, because the fields conflict

With both day fields restricted, cron ORs them. Use 0 9 1-7 * * and test the weekday inside the command: [ "$(date +\%u)" = 1 ] && /usr/local/bin/report.sh.

A timer after downtime

A server was powered off at 02:30, when its nightly timer should have fired. With Persistent=true, what happens at boot?

The job runs once, shortly after boot
It runs once for every missed night
Nothing until 02:30 the next day

Persistent=true stores the last run time and catches up a missed run once. cron has no equivalent.

Interview Questions¶

What happens between systemctl start and your application accepting traffic?
How do you change a setting on a service installed by a package, safely?
A cron job works when you run it by hand but fails from cron. What are the usual causes?
Why might a timer be better than a cron job?
A service keeps restarting. Which commands do you run, in what order?
What does ProtectSystem=strict do, and how do you let the service write its data?

Next¶

Continue to Storage, Disks, and LVM.