Discovery and Trust
A Data Catalog is a centralized, searchable portal where anyone in the company — engineers, analysts, product managers, compliance officers — can find, understand, and evaluate data assets.
Popular open-source and commercial catalogs include Amundsen (LinkedIn), DataHub (LinkedIn, open-sourced), Apache Atlas, and Alation (commercial). They all solve the same two core problems:
1. Discovery — "Where is the data I need?"
Instead of pinging five people on Slack, you search "revenue" in the catalog and get back a ranked list of matching tables, views, and dashboards. You can filter by database, team, or tag. This is the same experience as a Google search, but scoped to your internal data assets.
2. Trust — "Can I rely on this data?"
Finding a table is only half the battle. The catalog also surfaces metadata that tells you whether to trust it:
| Metadata Field | Example Value | Why It Matters |
|---|---|---|
| Owner | @jane.doe (Finance Eng) | Who to contact with questions |
| Freshness | Last updated 2 hours ago | Is the data stale? |
| Description | "Daily revenue in USD, excluding refunds" | What does this actually measure? |
| Schema | amount DECIMAL(10,2), currency VARCHAR(3) |
What are the columns and types? |
| Tags | PII, GDPR, finance-critical |
Sensitivity and compliance classification |
| Popularity | 47 queries in the last 7 days | Is this table actively used? |
| Quality Score | 94% (null rate < 1%, freshness OK) | Automated quality signal |
Without a catalog, this knowledge is tribal — it lives in the heads of specific engineers. When those engineers leave, the knowledge disappears. New hires spend weeks reverse-engineering what veterans knew instinctively.
Interviewers often ask: "How do you handle data discovery at scale?" A weak answer is "we use a wiki." A strong answer describes a catalog with automated metadata ingestion, ownership assignment, and freshness monitoring — and explains why manual wikis fail (they go stale immediately and nobody updates them).