Recap and final exam
Ten modules to move from "we have some data on S3" to a versioned model scoring tens of millions of flights every night. Here is the course condensed, then the decision grids that stay with you.
The course at a glance
| Module | The essential point |
|---|---|
| 1. Architecture | Driver plans, executors execute; partitions are the unit of parallelism; the UI on port 4040 is the diagnosis. |
| 2. RDD, DataFrame, Dataset | DataFrame is the default because Catalyst rewrites its plans; drop to RDD only when partitioning forces it. |
| 3. Transformations and actions | Nothing runs until an action fires; wide transformations shuffle and dominate the cost. |
| 4. Reading and columns | Parquet with an explicit schema and a partitioned layout; CSV at scale is a trap. |
| 5. Transformers and estimators | StringIndexer -> OneHotEncoder -> StandardScaler on a vector -> VectorAssembler to one features column. |
| 6. Pipeline and cross-validation | Preprocessing outside the pipeline leaks; CrossValidator parallelises across grid points. |
| 7. Algorithms | MLlib covers linear, trees, GBT (binary only), k-means and ALS; sample and train locally for what is missing. |
| 8. Tuning | Set shuffle.partitions to 2–4× cores; salt hot keys; read the UI before touching memory. |
| 9. Writing and integration | Partitioned Parquet with partitionOverwriteMode=dynamic; version model folders; symmetric save/load. |
| 10. End-to-end project | Pandas on a sample first, translate to Spark only when the volume forces it. |
The threads running through the course
Everything comes back to the shuffle. The shuffle explains why partitions matter (module 3), why groupBy is expensive (module 3), why key skew slows a stage tenfold (module 8), and why the pipeline should stay in DataFrame API rather than RDD (module 2). Understanding the shuffle gives you a mental model that makes every tuning decision deducible instead of magical.
The Transformer / Estimator contract is what makes MLlib worth using. It is what lets the whole flight-delays pipeline serialise into one folder (module 6), reload without knowing the training code (module 9), and stay leakage-free inside a cross-validation (module 6). A pipeline that mixes MLlib estimators with hand-written scaling steps loses all three properties at once.
Spark's fixed cost is real, and pandas is not the enemy. Modules 1 and 10 make the same point from opposite ends: below the volume that actually justifies Spark, pandas or Polars is faster to write, faster to run, and faster to debug. This course teaches you to reach for Spark on purpose — not by default.
Diagnosis precedes tuning. The Spark UI on port 4040 is the single source of truth. Reading task-duration percentiles, shuffle read/write balance, and spill columns locates the bottleneck in minutes; changing configuration parameters without reading the UI is guessing at a menu.
The pandas-Spark-or-something-else decision grid
| Data size | Model needed | Right tool |
|---|---|---|
| Fits in RAM | Anything | pandas + scikit-learn or XGBoost |
| Larger than RAM, fits on disk | Linear, trees, k-means, ALS | MLlib on Spark, or sample and train locally |
| Truly distributed | Missing from MLlib | Sample, train locally, score with Spark |
| Truly distributed | Linear, trees, ALS | MLlib or SparkXGBoost |
| Streaming or transactional writes | Anything | Spark plus Delta Lake or Iceberg |
Print this table. It is the answer to nine out of ten "should we use Spark?" meetings.
The final exam
The exam has 40 questions covering the ten modules: Spark's architecture and its cost when disproportionate, the difference between narrow and wide transformations, reading Parquet with schemas and partition pruning, building a leakage-free pipeline, choosing the right MLlib algorithm and knowing its limits, spotting key skew from the UI, and writing predictions in a way a scheduled job can rely on.
Several questions present situations to diagnose: a groupBy that spends ten minutes on a single task, a schema inferred once too many times, a VectorAssembler that throws on a string column, a MulticlassClassifier that is not in MLlib, an overwrite write that wiped last month's partitions. It is judgment that is assessed, not the recitation of API signatures.
On success, your certificate of completion is issued immediately; its number is verifiable by any third party on the platform.
Take the table above and, for each row, ask yourself "how would I know I picked the wrong tool?". If you can say why a groupBy on a laptop-sized DataFrame is slower than pandas, why randomSplit before fit(scaler) leaks, and what partitionOverwriteMode=dynamic actually changes, you are ready. Good luck!
Final exam
Ready to validate this course?
40 questions drawn at random from the course bank · passing score 70% · verifiable PDF certificate issued immediately on success.
Start the examYou need to be signed in to your InSkillML account with an active subscription. You can also start the exam from My courses.