Skip to main content

Apache Spark ML

Train and score models on datasets that no longer fit on one machine, using the same handful of ideas the rest of your team already relies on: DataFrames, pipelines, cross-validation. This course shows exactly when Spark starts to earn its keep, and when it is a costly detour.

Course Duration: 7h​

What You'll Learn​

  • Describe Spark's execution model — driver, executors, tasks, stages, partitions — and diagnose it from the web UI
  • Choose between RDD, DataFrame and Dataset, and understand why the DataFrame is the default in 2026
  • Read execution plans, tell narrow from wide transformations, and predict where a shuffle will occur
  • Load large columnar datasets with an explicit schema and prune partitions instead of scanning them
  • Build a Spark ML pipeline with VectorAssembler, StringIndexer, OneHotEncoder and StandardScaler
  • Tune hyperparameters at cluster scale with CrossValidator and ParamGridBuilder without leakage
  • Pick the right algorithm inside MLlib and know which ones are missing and how to work around it
  • Repartition, coalesce and diagnose key skew from the Spark UI, and set executor memory sensibly
  • Write partitioned Parquet results, save a trained pipeline and reload it for batch scoring
  • Decide, before starting, whether the problem actually needs Spark or whether pandas is enough

Prerequisites​

  • Python (course 02)
  • Supervised learning (course 04)
  • Feature engineering (course 06)

Course Modules​

  1. Spark architecture: driver, executors, partitions
  2. RDD, DataFrame and Dataset
  3. Transformations, actions and lazy evaluation
  4. Reading data and columnar formats
  5. Spark ML: transformers and estimators
  6. Pipelines and distributed cross-validation
  7. Available algorithms and their constraints
  8. Tuning: partitions, memory, shuffle
  9. Writing results and downstream integration
  10. Project: a model trained on a large dataset
  11. Recap and exam

The running project​

Across all ten modules you build the same object: a model that predicts flight delays over a public dataset of tens of millions of flights, stored as Parquet. Each module adds one capability to that same project — a schema and a partitioned read (4), a features pipeline (5), a cross-validated model (6), an algorithm choice justified by MLlib's real inventory (7), a shuffle problem solved (8), production writes and reloads (9), and finally a full comparison with the same pipeline run on a pandas sample of the same data (10). Development runs on local PySpark; every command is followed by the equivalent for a cluster or a Databricks workspace.

Assessment and certificate​

The course ends with a 40-question exam covering every module. On success, a certificate of completion is issued; its number is verifiable on the platform.

Free courses, by contrast, end with a 5-question quiz and a preview of the certificate, without certification.