Free — no signup required

Introduction: The "Dark Data" Problem

2 min read

The Library Without an Index

Imagine walking into a massive library where the books are thrown in a pile on the floor. There is no card catalog. There are no genre labels. To find "Harry Potter," you have to pick up every single book and read the cover. The only person who knows where anything is happens to be on vacation.

This is the state of data in most companies. Data accumulates across databases, data lakes, spreadsheets, and SaaS tools — with no central record of what exists, who owns it, or how it connects to anything else.

  • Data Scientists waste 30–40% of their time asking, "Does this table already exist?" or "Who owns this data?" or "Can I trust these numbers?"
  • Data Engineers are afraid to rename a column because they don't know which dashboards, ML models, or downstream jobs depend on it.
  • Compliance teams cannot answer a GDPR "Right to be Forgotten" request because they don't know every system that holds a copy of a user's data.

This is called the dark data problem — data that exists but is invisible, undocumented, and therefore dangerous to touch and impossible to trust.

Data Cataloging is the index for your data library — a searchable registry of every data asset in your organization.
Data Lineage is the map showing how data moves, transforms, and branches through that library over time.

Together, they turn a pile of books on the floor into a navigable, trustworthy knowledge system.

This is one of 18 chapters

Every chapter is free — Kubernetes, Terraform, SRE, distributed systems, and more — with fast daily review built in.

Sign up free