One Stream, Multiple Destinations
A pipeline can clone a data stream and send different subsets to different backends simultaneously. This enables a tiered storage strategy where you match data value to storage cost.
The Tiered Storage Model
| Tier | Data Type | Access Pattern | Storage Backend | Cost |
|---|---|---|---|---|
| Hot | Errors, security events | Daily, fast search | Datadog / Elasticsearch | High |
| Warm | Sampled request logs | Weekly, moderate search | OpenSearch | Medium |
| Cold | Full compliance archive | Yearly, bulk export | AWS S3 Glacier | Very Low |
Configuring a Fork-and-Route Pipeline
Here's how this looks in a Vector configuration: the incoming log stream is forked into two independent transforms — one filters for just ERROR/CRITICAL events, the other passes everything through while stamping archive metadata onto it — and each transform feeds its own sink: the filtered errors go straight to Datadog for hot, searchable storage, while the full stamped stream goes to a compressed, gzip-encoded S3 bucket for cold archival.
Routing Based on Metadata
Routing decisions can use any field in the event — not just log level. Common routing keys:
service.name: Send payments service logs to a dedicated high-retention indexkubernetes.namespace: Route production namespace logs to hot storage, staging to coldhttp.status: Route 5xx to PagerDuty-integrated alerting pipeline, 2xx to archiveteam: Route each team's logs to their own Datadog index for cost attribution
Key Point: Intelligent routing lets you satisfy competing requirements simultaneously — compliance archives, fast debugging, and cost efficiency — without forcing a single compromise. The pipeline is the place to encode these business rules, not the applications themselves.