Why OTel Collector Exists
Before OpenTelemetry, every observability vendor had its own agent: Datadog Agent, New Relic Infrastructure Agent, Splunk Universal Forwarder. Running multiple vendors meant running multiple agents on every node, each consuming resources and requiring separate configuration.
The OpenTelemetry Collector is a vendor-neutral, open-source component that can receive, process, and export all three telemetry signals — metrics, logs, and traces — in a single process. It is governed by the Cloud Native Computing Foundation (CNCF) and supported by every major observability vendor.
The Three-Component Model
The Collector is configured as a pipeline with three stages:
1. Receivers — How data enters the Collector. Common receivers include otlp (accepting data from OTel SDKs over gRPC and HTTP), prometheus (scraping Prometheus-format metrics endpoints on a schedule), and filelog (tailing log files directly, the same job Fluent Bit does).
2. Processors — What happens to the data in transit. A typical chain includes memory_limiter (critical — prevents OOM crashes by applying backpressure once memory usage crosses a threshold), batch (groups events to reduce network calls), a filter (drops low-value data, like health-check log records), and an attributes processor (enriches events with metadata, like tagging everything with the deployment environment).
3. Exporters — Where data goes after processing. Each exporter knows one backend's wire format: an otlp exporter sending traces to a vendor like Honeycomb, a prometheusremotewrite exporter sending metrics to Thanos or Cortex, an awss3 exporter archiving logs to a bucket, or a logging exporter that just prints to stdout for debugging.
Wiring It Together: The Service Section
Defining receivers, processors, and exporters doesn't activate them. You must explicitly wire them into pipelines in the service section — for example, a traces pipeline chaining the otlp receiver through memory_limiter, a health-check filter, and batch, out to the Honeycomb exporter; a metrics pipeline combining otlp and prometheus receivers through its own processor chain to prometheusremotewrite; and a logs pipeline combining otlp and filelog receivers out to S3.
Processor Order Matters
The order of processors in the pipeline is sequential and significant:
memory_limitermust always be first. If the Collector runs out of memory, it needs to start dropping data before it crashes — not after.filterbeforebatch. Drop data before grouping it; don't waste batch slots on data you're about to discard.batchlast. Grouping is the final step before export.
A common interview question: "What happens if you put the batch processor before the memory_limiter in an OTel Collector pipeline?" The answer: during a traffic spike, the batch processor accumulates events in memory before the memory limiter can react, potentially causing an OOM crash. The memory limiter must be first in the chain so it can apply backpressure before memory is exhausted. This demonstrates understanding of processor ordering semantics, which separates candidates who've read the docs from those who've operated pipelines in production.
Key Point: The OTel Collector unifies metrics, logs, and traces in one vendor-neutral tool. Its three-stage model (receivers → processors → exporters) is explicit and composable. Processor order is not cosmetic — it directly affects reliability and cost.