How does a catalog actually get populated? There are two fundamentally different approaches, each with distinct trade-offs.
1. Pull-Based (Crawlers)
A crawler is a scheduled job that periodically connects to your data sources and extracts metadata by inspection.
How it works:
1. The crawler connects to Snowflake and runs SHOW TABLES and DESCRIBE TABLE to extract schema information.
2. It parses query logs (e.g., Snowflake's QUERY_HISTORY view) to infer lineage — "this query read from Table A and wrote to Table B, so there must be a dependency."
3. It writes all of this into the catalog's metadata store.
Tools: Apache Atlas, early versions of Amundsen, AWS Glue Data Catalog.
Pros:
- Zero changes to existing pipelines. Works with legacy systems immediately.
- Good for initial bootstrapping of a catalog.
Cons:
- Stale by design. If the crawler runs nightly, your catalog is always up to 24 hours behind reality.
- Brittle lineage. Parsing SQL to infer lineage is error-prone. Dynamic SQL, stored procedures, and Python-based transformations are often missed entirely.
- No runtime context. The crawler sees the schema but not whether the last job run succeeded or failed.
2. Push-Based (Event-Driven)
In a push-based architecture, the pipeline itself reports metadata to the catalog as it runs.
How it works:
1. An Airflow DAG runs an ETL job.
2. When the job starts, it emits an event: {"job": "orders_etl", "inputs": ["raw_orders"], "started_at": "2024-01-15T03:00:00Z"}.
3. When the job finishes, it emits: {"job": "orders_etl", "outputs": ["orders_cleaned"], "rows_written": 142000, "finished_at": "2024-01-15T03:04:22Z"}.
4. The catalog receives these events and updates lineage in real time.
Tools: DataHub, Marquez, OpenLineage (the standard protocol), Apache Atlas (newer versions).
Pros:
- Real-time. The catalog reflects the current state of your pipelines within seconds of job completion.
- Accurate. The job knows exactly what it read and wrote — no guessing from SQL parsing.
- Rich context. You can attach run statistics, data quality metrics, and failure reasons to each lineage event.
Cons:
- Requires instrumenting your pipelines. Existing jobs need to be modified to emit events.
- Adds a dependency: if the catalog's event receiver is down, jobs must handle this gracefully (usually by buffering events).
The Modern Standard: OpenLineage
OpenLineage is an open specification (not a tool) that defines a standard JSON schema for lineage events. Think of it as the HTTP protocol for data lineage — any pipeline that emits OpenLineage events can be consumed by any catalog that understands the spec.
{
"eventType": "COMPLETE",
"job": { "namespace": "orders-pipeline", "name": "orders_etl" },
"inputs": [{ "namespace": "postgres", "name": "raw_orders" }],
"outputs": [{ "namespace": "snowflake", "name": "orders_cleaned" }],
"run": { "runId": "d46e465b-d358-4d32-83d4-df2d1b73c14e" }
}
Airflow, dbt, Spark, and Flink all have OpenLineage integrations. This means you can switch catalog backends (from Marquez to DataHub, for example) without changing your pipeline code — only the event receiver changes.
A common interview question is: "How would you implement data lineage in a large organization?" Distinguish between pull and push approaches, explain why push-based with OpenLineage is the modern standard, and mention that column-level lineage (not just table-level) is required for GDPR compliance use cases. This signals you understand both the architecture and the business drivers.