Skip to main content

Module 1 — Spark architecture: driver, executors, partitions

Every Spark job, from a one-liner on a laptop to a job on a five-hundred-node cluster, runs on the same handful of parts. Learning to name them, and to see them in the web UI, is what separates a Spark user from a Spark tourist — and it is what will let you decide, in the next modules, whether the tool is worth the cost for your problem.

The two kinds of processes​

A running Spark application is always a driver and one or more executors.

The driver is the process where your Python script lives. It holds the SparkSession, builds the logical plan of every operation you request, and hands out work. It does not process the data itself. If you write df.count(), it is the executors that count.

The executors are the processes that actually read files, run tasks, hold cached data in memory and send results back. There are many of them, they run on the workers of a cluster, and they are what you scale up when the data grows. On a laptop, in the local[*] mode used throughout this course, both the driver and the executors live inside the same Python process — but conceptually they remain distinct.

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("flight-delays")
    .master("local[*]")             # local mode: use every available core
    .config("spark.sql.shuffle.partitions", 32)
    .getOrCreate()
)

On a cluster you drop the .master(...) line, because the resource manager (YARN, Kubernetes, Databricks' runtime) provides it. Every other line stays the same. That is one of Spark's real selling points: the code you debug on your laptop is the code that runs on the cluster.

Tasks, stages and partitions​

Once the driver has your logical plan, it splits it into stages. A stage is a chunk of work that can run without shuffling data between machines. Between two stages, executors reshuffle the data — the expensive operation the next modules will devote themselves to avoiding.

Each stage is itself split into tasks, and there is exactly one task per partition of the data at that point of the plan. If your DataFrame has 200 partitions, a stage over it will produce 200 tasks. Tasks run in parallel across executors' cores.

Partitions are therefore the unit of parallelism. A DataFrame with one partition runs in one task, on one core, however many machines you paid for. A DataFrame with ten thousand partitions of a few kilobytes each spends most of its time scheduling rather than computing. Module 8 comes back to this in detail; for now, remember that the number of partitions is a first-order parameter of every Spark program, and that Spark rarely gets it right on its own.

The web UI is not optional​

Every SparkSession starts a web server, typically on http://localhost:4040, that shows the running jobs, their stages, their tasks and, crucially, how long each task took. Reading this UI is the fastest way to spot the two chronic problems of Spark applications: a skewed stage where a handful of tasks take ten times longer than the rest, and a shuffle stage where terabytes of intermediate data cross the network.

Whenever a Spark job feels slow, open this UI before you tune anything. Guessing at parameters without reading the UI is how teams spend a week doubling the executor memory of a job whose real problem was a hundred empty partitions.

When Spark is disproportionate​

Before you write a single transformation, ask whether the data even needs Spark. A rule of thumb that has survived years of practice: if the raw data fits comfortably on a laptop's disk and the working set fits in RAM, pandas or Polars will be faster than Spark and much easier to debug. Spark's fixed cost — driver startup, task scheduling, JVM overhead, plan compilation — is only paid back when the data no longer fits.

A Spark job that would have been a pandas one-liner

Running a five-line grouping on a two-gigabyte CSV in Spark, on a laptop, is a common anti-pattern. It takes twenty seconds where pandas would take two, produces a stack trace half a screen tall on the first typo, and teaches nothing about distributed computing. Reach for Spark when the data does not fit — never before.

The flight-delays project of this course uses tens of millions of rows precisely so that Spark starts to actually earn its keep. Module 10 will re-run the same pipeline on a pandas sample so you can see, on your own machine, where the crossover happens.

Summary​

  • A Spark application is a driver (planning) and executors (execution); the driver never processes data.
  • Work is split into stages separated by shuffles, then into tasks, one per partition.
  • Partitions are the unit of parallelism; too few starves cores, too many wastes them in scheduling.
  • The web UI on port 4040 is the first place to look at any Spark performance problem.

Next module: RDD, DataFrame and Dataset — the three APIs, and why the DataFrame won.