The Aggregation Challenge: ELK vs. Loki

Centralizing the Data

Because you cannot SSH into ephemeral containers, you need a pipeline that collects logs from every container the moment they're written and ships them to a central, persistent store. This is called Log Aggregation. The pipeline has three stages:

1. Collection: A lightweight agent runs on every node (as a Kubernetes DaemonSet) and tails the log output of every container on that node.
2. Transport: The agent forwards log lines — optionally with enrichment (adding Kubernetes metadata like pod name, namespace, node name) — to a central backend.
3. Storage & Query: The backend indexes and stores the logs, and exposes a query interface for search and dashboards.

Two architectures dominate this space, with fundamentally different philosophies about what to index.

1. The ELK Stack (Elasticsearch, Logstash/Fluentd, Kibana)

The traditional heavyweight. Elasticsearch treats every log line as a document and builds an Inverted Index — the same data structure used by search engines like Google — over every word in every field of every document.

Think of it like the index at the back of a textbook: if you want to find every page that mentions "timeout," the index tells you instantly without reading the whole book.

• Pros: Blazing fast full-text search. You can find any unique string — a specific error ID, a UUID, a customer email — across terabytes of logs in milliseconds. Extremely powerful query language (Kibana Query Language / Lucene).
• Cons: The index itself is enormous — often 30–50% the size of the raw data. Ingestion is CPU-intensive because every token in every field must be indexed. Running Elasticsearch at production scale requires significant infrastructure investment: large RAM allocations (the JVM heap needs to hold hot index segments), fast NVMe disks, and careful shard management.

When to choose ELK: When you need arbitrary full-text search across all log content, or when your team already has Elasticsearch expertise and infrastructure.

2. Grafana Loki

The modern challenger, designed from the ground up for Kubernetes. Loki's core insight is that most log queries start with "show me logs for this specific service in this environment" — not "find this string anywhere in all logs ever."

• How it works: Loki does not index the log text at all. It only indexes the labels (metadata) attached to each log stream — things like app="frontend", namespace="production", pod="frontend-abc123". The actual log lines are compressed and stored in chunks (in object storage like S3 or GCS), organized by label set and time range.
• Pros: Dramatically cheaper to operate. The index is tiny. Ingestion is fast and CPU-light. In Kubernetes, Loki re-uses the labels you already have on your pods — zero extra configuration. Storage costs are low because compressed chunks in object storage are cheap.
• Cons: "Needle in a haystack" searches — finding a specific UUID across all services — are slow because Loki must decompress and grep through chunks. The query language (LogQL) requires you to always specify label filters first to narrow the search space before doing text matching.

When to choose Loki: When you're running on Kubernetes, cost efficiency matters, and your queries are primarily label-scoped (which they almost always are in practice). Loki pairs naturally with Grafana, which most teams already use for metrics.

The practical rule: In Loki, always start your query with the most selective label filter. {app="payment-service", namespace="prod"} |= "timeout" is fast. {} |= "timeout" (no label filter) will time out or be rejected entirely.