The Problem
Imagine every department in a large company sends its mail directly to a different courier service — accounting uses FedEx, engineering uses UPS, HR uses DHL. If the company decides to switch couriers, every department has to change its process independently. Now imagine one department accidentally sends 10,000 packages in a single day. There's no central mailroom to catch the mistake before the bill arrives.
This is exactly what happens when applications send telemetry directly to observability vendors. In a naive setup, every service is hardwired to a specific backend:
- App A → Sends Logs → Splunk
- App B → Sends Metrics → Datadog
- App C → Sends Traces → Honeycomb
This creates two compounding problems:
1. Vendor Lock-in: Every application contains vendor-specific configuration — SDK initialization, endpoint URLs, authentication tokens. If your company decides to migrate from Splunk to Elasticsearch, you must locate, reconfigure, and redeploy every application that sends to Splunk. In a microservices environment with 50+ services, this is a multi-week project with significant risk.
2. Uncontrollable Costs: Observability vendors charge by data volume ingested. If App A has a bug that causes it to emit DEBUG-level logs at 1 TB/hour, you have no circuit breaker. The data flows directly to the vendor, and you receive a catastrophic bill. Fixing it requires finding the bug, deploying a fix, and waiting — all while the meter runs.
The Solution: The Observability Pipeline
An Observability Pipeline introduces a dedicated intermediary layer between your data sources and your storage backends. Instead of apps talking directly to vendors, they talk to the pipeline, and the pipeline talks to vendors:
App A ──┐
App B ──┼──► Pipeline (Vector / OTel Collector) ──► Splunk
App C ──┘ └──► S3 Archive
This single architectural change gives you three superpowers:
- Decoupling: Change your backend vendor by updating one pipeline config, not 50 application configs.
- Cost Control: Filter, sample, and compress data before it reaches the expensive storage tier.
- Flexibility: Route the same data to multiple destinations simultaneously — hot storage for debugging, cold storage for compliance.
Interviewers frequently ask: "How would you reduce your observability costs by 60% without losing visibility into production incidents?" The answer is an observability pipeline with filtering and sampling. The key insight to articulate: you're not reducing data collection — you're reducing data storage by dropping low-value signals (DEBUG logs, health check traces) before they hit the expensive backend. Always keep 100% of errors; sample everything else.
Key Point: A pipeline decouples applications from vendors and gives you cost control. It is the difference between being a passive observer of your telemetry bill and an active controller of it.