Think in Chunks
Imagine trying to count all the cars that pass a toll booth in a year. You wouldn't count them one by one as they pass. You would wait until the end of the day, get a log file of all the cars, and count them all at once. That's Batch Processing.
Batch processing handles a large, bounded volume of data all at once. It runs at scheduled intervals — for example, every night at 2 AM or every hour on the hour. The data set has a clear beginning and end: "process all transactions from yesterday."
Pros:
- Great for complex analysis on large historical datasets.
- Highly efficient — you can optimize reads, sorts, and joins across the entire dataset at once.
- Simpler failure model — if a job fails, you re-run it on the same bounded dataset.
Cons:
- High latency. Insights are always "as of last night" or "as of last hour."
- Not suitable for use cases that require immediate action on new data.
The King of Batch: Apache Spark
Apache Spark is the de-facto standard for large-scale distributed batch processing. It is an open-source, in-memory data processing engine designed to process petabytes of data across clusters of hundreds or thousands of commodity machines — far faster than the older Hadoop MapReduce approach it largely replaced.
A typical Spark batch job looks like this in PySpark (Spark's Python API):
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count
spark = SparkSession.builder.appName("DailyRideCount").getOrCreate()
# Read a day's worth of ride data from a Data Lake (S3/GCS/ADLS)
rides_df = spark.read.parquet("s3://my-data-lake/rides/date=2024-01-15/")
# Transform: count completed rides per city
city_counts = (
rides_df
.filter(col("status") == "completed")
.groupBy("city")
.agg(count("*").alias("total_rides"))
)
# Load: write results to the Data Warehouse
city_counts.write.mode("overwrite").parquet("s3://my-data-warehouse/daily_city_rides/date=2024-01-15/")
Spark distributes this work automatically across the cluster. Each machine processes a partition of the data in parallel, and Spark coordinates the shuffle (data redistribution across machines) needed for the groupBy.