The Collection Layer
Before logs can be queried, they must be collected and forwarded. In Kubernetes, containers write logs to stdout/stderr, and the kubelet writes those to files on the node at /var/log/pods/. A log shipper runs as a DaemonSet — one pod per node — and tails these files.
The two most common shippers are:
Fluentd / Fluent Bit (for ELK and others):
- Fluent Bit is the lightweight version (written in C, ~450KB binary). Fluentd is the full-featured version (Ruby-based, richer plugin ecosystem).
- Can parse, filter, enrich, and route logs to multiple destinations simultaneously.
- Fluent Bit is the standard choice for Kubernetes log collection due to its low resource footprint.
Promtail (for Loki):
- Developed by Grafana Labs specifically to ship logs to Loki.
- Automatically discovers pods via the Kubernetes API and attaches their labels to log streams.
- Minimal configuration needed in a standard Kubernetes setup.
A Minimal Promtail Configuration
A minimal Promtail configuration points its clients at the Loki push endpoint, tracks read position per file so a restart doesn't re-ship old lines, and defines a scrape config that discovers pods via the Kubernetes API, parses the Docker log wrapper format, and relabels the pod's app label and namespace into Loki labels.
This configuration tells Promtail to discover all pods via the Kubernetes API, read their log files, parse the Docker log wrapper format, and attach the pod's app label and namespace as Loki labels before shipping.