The Architecture: Agents and Aggregators

Why Two Layers?

Consider a factory with 500 workers. You could have each worker walk to the central office to file their daily report — but that's inefficient and clogs the hallways. Instead, each floor has a local supervisor who collects reports, summarizes them, and sends a single consolidated report to the central office. The workers stay productive; the central office isn't overwhelmed.

Observability pipelines use the same pattern: lightweight edge agents on every node collect raw data locally, and centralized aggregators do the heavy processing.

Layer 1: Edge Agents (Lightweight Collectors)

Edge agents run on every node in your cluster, typically as a Kubernetes DaemonSet (one pod per node). Their job is narrow and fast: grab data from local sources and forward it upstream with minimal processing.

• Tools: Fluent Bit, Filebeat, Promtail
• Responsibilities:
• Tail log files from /var/log/containers/
• Forward to the aggregator layer
• Apply only the lightest transforms (e.g., add node.name tag)
• Resource footprint: ~50MB RAM, <1% CPU

Edge agents are intentionally dumb. Putting complex parsing logic on every node means deploying and updating that logic across hundreds of nodes — a maintenance nightmare.

Layer 2: Aggregators (Heavy Processors)

Aggregators run as a centralized Deployment or StatefulSet. They receive data from all edge agents, apply complex processing, and fan out to multiple backends.

• Tools: Vector, OpenTelemetry Collector, Logstash
• Responsibilities:
• Parse unstructured log lines into structured JSON
• Filter noise (health checks, debug logs)
• Sample high-volume low-value events
• Route to multiple backends
• Buffer data during backend outages
• Resource footprint: Scales with data volume; typically 2–8 GB RAM per instance

Node 1: [Fluent Bit] ──┐
Node 2: [Fluent Bit] ──┼──► [Vector Aggregator] ──► Datadog (errors only)
Node 3: [Fluent Bit] ──┘         │              └──► S3 (everything, compressed)
                                  │
                             [Buffer: disk]
                          (survives backend outages)

The Buffering Advantage

A critical but often overlooked feature of aggregators is buffering. If your Datadog endpoint goes down for 10 minutes, a direct-to-vendor setup loses those 10 minutes of telemetry. An aggregator with disk-backed buffering holds the data locally and replays it when the backend recovers. This is essential for SRE workflows where you need complete data during incident retrospectives.

Key Point: Edge agents are lightweight forwarders; aggregators are the intelligent processing layer. Separating these concerns keeps nodes healthy and centralizes operational complexity where it's easier to manage.