How does Spark work its magic? It has evolved its core abstractions over time, and understanding this evolution explains why modern Spark code is both simpler and faster than early Spark code.
The Old Way: RDDs (Resilient Distributed Datasets)
An RDD is the lowest-level abstraction in Spark. Think of it as a distributed list that happens to be spread across many machines. Each machine holds a partition of the list. If a machine fails, Spark can recompute the lost partition from the original data — that's the "Resilient" part.
RDDs are powerful but low-level. Here's the problem: because an RDD can hold any type of Java or Python object, Spark treats it as a black box. It doesn't know what the data looks like inside. It can't optimize it. It just executes your code exactly as written, even if there's a much faster way to get the same result.
The Modern Way: DataFrames
A DataFrame is a higher-level abstraction built on top of RDDs. It's like a table in a relational database — it has named columns, and each column has a defined data type (string, integer, timestamp, etc.). This structure is called the schema.
Because Spark knows the schema, it can reason about your query logically before executing it. This is where the Catalyst Optimizer comes in.
The Catalyst Optimizer
When you write a DataFrame query like:
df.filter(col("country") == "US") \
.join(other_df, "user_id") \
.groupBy("city") \
.count()
You are writing a logical plan — what you want, not how to compute it. The Catalyst Optimizer takes this logical plan through four stages:
- Analysis: Resolves column names and types against the schema.
- Logical Optimization: Applies rule-based rewrites. For example, it pushes the
filterbefore thejoinso that fewer rows participate in the expensive join operation. - Physical Planning: Generates multiple candidate physical execution plans (e.g., broadcast join vs. sort-merge join) and picks the cheapest one based on statistics.
- Code Generation: Compiles the final plan into optimized JVM bytecode for execution.
This makes DataFrame operations orders of magnitude faster than equivalent hand-written RDD code, because the optimizer can make decisions a human programmer would never bother to make manually.
Interviewers frequently ask: "What's the difference between an RDD and a DataFrame in Spark?" The trap is answering only with "DataFrames have columns." The complete answer is: DataFrames expose a schema to the Catalyst Optimizer, which enables automatic query optimization — filter pushdown, join strategy selection, and code generation. RDDs are opaque to the optimizer and execute exactly as written. In practice, you should almost never write new code using RDDs directly.