The Log Book
Before systemd, logs were scattered across multiple text files in /var/log/. Finding out what happened during a crash meant correlating timestamps across /var/log/syslog, /var/log/auth.log, /var/log/nginx/error.log, and application-specific logs — all with slightly different timestamp formats, all requiring grep and awk to filter.
Systemd introduced journald (the journal daemon). It collects stdout, stderr, kernel messages, and syslog output from all services and stores them in a single, structured, binary format. Because the data is structured (not just lines of text), filtering is precise and fast.
To read the journal, you use journalctl. Run bare, it dumps the entire journal — in practice you almost always filter it: by unit (isolating one service's output, optionally following it live the way tail -f follows a file), by boot (scoping to the current startup or one further back), by a time window (accepting both absolute timestamps and relative phrases like "30 minutes ago"), by priority (surfacing only messages at or above a given severity), by line count (the most recent N entries), or as structured JSON output for piping into something like jq. These filters combine freely — filtering by unit, priority, and time window together is the standard way to scope a log search to an actual incident window.
Priority levels (from -p flag, lowest number = highest severity):
| Level | Number | Meaning |
|---|---|---|
| emerg | 0 | System is unusable |
| alert | 1 | Immediate action required |
| crit | 2 | Critical conditions |
| err | 3 | Error conditions |
| warning | 4 | Warning conditions |
| notice | 5 | Normal but significant |
| info | 6 | Informational |
| debug | 7 | Debug-level messages |
Managing journal disk usage:
The journal stores data in /var/log/journal/ (persistent) or /run/log/journal/ (volatile, cleared on reboot). journalctl --disk-usage reports current consumption, and a family of --vacuum-* flags trims it back down — by age, by total size, or by number of retained boots.
Journal retention is configured in /etc/systemd/journald.conf via SystemMaxUse=, MaxRetentionSec=, and related directives. On production systems, you typically configure this to prevent the journal from consuming unbounded disk space.
Interviewers often ask how you would debug a service that fails to start. The answer follows a clear sequence: systemctl status myservice.service for a quick summary and the last few log lines, then journalctl -u myservice.service -n 100 for more history, then journalctl -u myservice.service -b to see everything since the last boot. Mentioning journalctl -p err -b to filter for errors across all services shows you know how to investigate systemic issues, not just single-service problems.