Skip to main content

Module 2 — RDD, DataFrame and Dataset

Spark has grown three data abstractions in ten years and it kept all three. Knowing why they coexist is not academic: it decides which API you write your job in, and how much of the optimizer you get for free. The driver-and-executors picture of module 1 stays true throughout; what changes is only what the executors are asked to do.

RDD — the historical bedrock​

The Resilient Distributed Dataset is the original abstraction, introduced in 2010. An RDD is, quite literally, a partitioned Python or Scala collection with a recipe for how to rebuild any partition that is lost. Every transformation you apply to it is executed as you wrote it: no rewriting, no optimization.

lines = spark.sparkContext.textFile("flights.csv")   # an RDD of raw strings
delays = lines.map(lambda row: parse(row)).filter(lambda r: r.delay > 15)
delays.count()

The RDD is what everything else is built on. You still see it inside Spark itself and inside pipelineRDD-style connectors, but for application code the RDD has become a niche tool: use it when you truly need custom partitioning, low-level state, or types that no DataFrame column can hold. For 95 % of jobs, including every model this course trains, the DataFrame is the right choice.

DataFrame — the API that won​

A DataFrame is a distributed table: rows and typed named columns, laid out on the executors like an RDD but exposed through a query-language style API.

from pyspark.sql import functions as F

flights = spark.read.parquet("s3://open-data/flights/")

late = (
    flights
    .filter(F.col("cancelled") == 0)
    .groupBy("carrier")
    .agg(F.avg("arrival_delay").alias("avg_delay"))
    .orderBy(F.desc("avg_delay"))
)
late.show(5)

Two things happen behind that code that never happened with RDDs. First, the operations you compose are not executed in order — they are recorded as a logical plan. Second, that plan is handed to the Catalyst optimizer, which rewrites it before a single row moves.

Catalyst — why the DataFrame is faster than the RDD it compiles to​

Catalyst is Spark's SQL optimizer. Given the logical plan of the query above, it applies rules from a catalogue you can browse in EXPLAIN EXTENDED:

  • predicate pushdown: the filter(cancelled == 0) is pushed down to the reader so that Parquet's row-group metadata can skip whole files
  • column pruning: only the four columns actually used are read; the other twenty are never materialized in memory
  • projection reordering: cheap transformations move above expensive ones
  • join reordering: the smaller side of a join is broadcast when it fits

The result is that a DataFrame query almost always compiles down to a faster RDD than the one you would have written by hand. Bypassing the DataFrame API — reaching for .rdd.map(...) in the middle of a pipeline — throws away every one of these optimizations.

Dataset — typed, in Scala​

The Dataset adds compile-time types on top of the DataFrame. In Scala or Java, a Dataset[Flight] is a DataFrame whose rows are known to be Flight case classes, and the compiler catches typos in column names. It gets you both Catalyst optimization and static typing.

In PySpark, the Dataset does not exist as a separate API: Python has no compile-time types, so DataFrame is the typed API. That is why every code example in this course is a DataFrame.

Spark SQL — the same engine, another dialect​

The exact same pipeline can be written in SQL:

flights.createOrReplaceTempView("flights")

spark.sql("""
  SELECT carrier, AVG(arrival_delay) AS avg_delay
  FROM flights
  WHERE cancelled = 0
  GROUP BY carrier
  ORDER BY avg_delay DESC
""").show(5)

Both queries produce the same logical plan and the same physical plan. Which one you write is a matter of team habit and readability. Mixing them — SQL for the shape of the query, DataFrame API for the model plumbing — is how most of this course's real code will look.

Which API to pick

Default to the DataFrame for everything, including model training. Reach for RDD only when a partitioning trick or a custom serialization forces your hand, and always inside a small, well-isolated function. Reach for SQL whenever the query would read like SQL — grouping, filtering, joining — because a colleague who does not know Spark can still read it.

Summary​

  • Spark exposes three APIs — RDD, DataFrame, Dataset — and keeps them all for backward compatibility.
  • The DataFrame is the default in 2026, because Catalyst rewrites its plans into faster RDDs than you would hand-write.
  • The Dataset adds static typing in Scala; in PySpark the DataFrame already plays that role.
  • Spark SQL is a second dialect of the same engine; compose it with the DataFrame API where each is clearer.

Next module: how those DataFrame operations actually execute — transformations, actions and lazy evaluation.