Unit Tests for Data
In software engineering, we write unit tests to assert that code behaves correctly: assert calculate_tax(100, 0.08) == 8.0. If someone changes calculate_tax in a way that breaks this assertion, the CI pipeline fails before the code ships.
Data as Code applies this exact discipline to data. Instead of asserting that functions return correct values, we assert that datasets have correct properties. These assertions are called Expectations (in Great Expectations) or Checks (in Soda), and they live in version-controlled files that run automatically in your pipeline.
The two dominant open-source tools are:
- Great Expectations (GX): A Python library with a rich API for defining expectations programmatically or via a YAML-based suite. Best for complex, custom validations.
- Soda: A declarative, YAML-based tool designed for readability. Easier for data analysts who aren't Python developers.
How Great Expectations Works
Great Expectations organizes quality checks into Expectation Suites — collections of assertions about a dataset. You define the suite once, commit it to Git, and run it as a step in your pipeline. A typical suite for an orders table layers several categories of checks on top of each other: structural checks that required columns exist and aren't null, type/range checks that a price column is an integer within a sane bound, a volume check that the row count falls in an expected range (catching truncated files), and a statistical check that the column's mean falls in an expected range (catching the dollars-vs-cents class of silent failure) — all defined once and re-run on every pipeline run.
How Soda Works
Soda achieves the same thing with a declarative YAML syntax that's easier to read and write for non-Python users — the same categories of checks (row count, null counts, min/max bounds, average, format validation) expressed as one-line YAML assertions per check, run via a soda scan CLI invocation against the target warehouse.
The Circuit Breaker Pattern
The power of these tools comes from where you place them in the pipeline. The correct placement is after ingestion, before transformation:
If validation fails, the pipeline halts. Bad data stays in the staging area — isolated, inspectable, and never polluting the production warehouse. This is the Circuit Breaker pattern: fail fast, fail loudly, contain the blast radius.