Lesson 4 — scikit-learn
If you do one thing with classical machine learning in Python, you do it with scikit-learn. Its value is not any single algorithm; it is that every algorithm behaves the same way, so learning the library once gives you access to all of it.
The interface that unified the field
Every model in scikit-learn, from a linear regression to a gradient-boosted ensemble, exposes the same three methods.
model.fit(X, y) # learn from features X and answers y
model.predict(X_new) # produce predictions for unseen data
model.score(X, y) # evaluate against known answers
Swapping a decision tree for a random forest or a support vector machine means changing one line — the constructor. Everything around it is untouched. That consistency is why comparing five approaches is an afternoon's work rather than a week's, and it has been copied by most libraries that came afterwards.
Data preparation follows a matching convention:
scaler.fit(X_train) # measure what is needed, here mean and spread
scaler.transform(X_train) # apply the transformation
The separation between fit and transform looks like ceremony until you see what it prevents, which is the subject of the last section of this lesson.
What it covers
| Family | What it does | Typical members |
|---|---|---|
| Classification | predict a category | logistic regression, random forest, gradient boosting, SVM |
| Regression | predict a number | linear and ridge regression, random forest, gradient boosting |
| Clustering | find groups with no labels | k-means, DBSCAN, hierarchical clustering |
| Dimensionality reduction | compress many features into few | PCA, t-SNE, UMAP via companion libraries |
| Preprocessing | scale, encode, impute | StandardScaler, OneHotEncoder, SimpleImputer |
| Model selection | split, cross-validate, tune | train_test_split, cross_val_score, GridSearchCV |
| Metrics | measure quality properly | accuracy, precision, recall, F1, ROC AUC, MAE, RMSE |
What it deliberately does not cover is deep learning. There is no GPU support and no neural network training worth using. That boundary is intentional: scikit-learn owns classical machine learning on tabular data, and PyTorch or TensorFlow own deep learning. Trying to make either do the other's job goes badly.
Pipelines: the feature that prevents mistakes
A pipeline chains preparation steps and a final model into one object that behaves like a model.
pipeline = Pipeline([
('impute', SimpleImputer(strategy='median')),
('scale', StandardScaler()),
('model', RandomForestClassifier()),
])
pipeline.fit(X_train, y_train)
pipeline.predict(X_test)
This is not a convenience wrapper. It closes three real failure modes.
It guarantees identical treatment at prediction time. When your model goes to production, the incoming data must be imputed and scaled exactly as during training. Doing that by hand in a second code path is how models quietly degrade: a subtly different scaling in production is invisible and ruins predictions.
It makes cross-validation honest. Inside a pipeline, every preparation step is refitted on each training fold only. Done manually, it is very easy to scale the whole dataset once before splitting, which leaks information and inflates your score.
It makes tuning cover the whole process. A grid search over a pipeline can search preparation choices and model settings together, which is what you actually want, since the best imputation strategy depends on the model.
Data leakage: the failure that flatters you
This deserves its own section because it is the most common way a machine learning result turns out to be worthless, and because it produces better numbers, so nothing looks wrong.
Data leakage is when information that would not be available at prediction time reaches the model during training.
The textbook case is scaling before splitting:
# Wrong: the scaler has seen the test rows
X_scaled = StandardScaler().fit_transform(X)
X_train, X_test = train_test_split(X_scaled)
# Right: the scaler only ever measures the training rows
X_train, X_test = train_test_split(X)
scaler = StandardScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
In the wrong version the mean and standard deviation were computed over rows the model is supposed to have never seen. The contamination is mild but real, and your test score is no longer an honest estimate.
Other frequent forms, all of which have shipped in production somewhere:
- A feature that encodes the answer. Predicting churn using a
cancellation_datecolumn. The score is near perfect and the model is useless, because that column is only filled in after the event you are predicting. - Random splitting of time series. Predicting tomorrow using rows from next week. For anything time-ordered, split by date, never at random.
- Duplicated rows across the split. The same record in both training and test means you are testing on memorised examples.
- Reusing the test set to choose. Once you have compared two models on it, it has become a validation set and no longer estimates real performance.
A score that is far better than you expected is more often leakage than insight. When a first model reaches 99% on a problem experts find hard, the correct reaction is to go looking for the leak, not to present the result.
Where scikit-learn fits, and what replaces it
For maximum accuracy on tabular data, the current practice is to use scikit-learn for the whole surrounding machinery — splitting, pipelines, cross-validation, metrics — and to plug in a dedicated gradient boosting library as the final estimator: XGBoost, LightGBM or CatBoost. All three implement the scikit-learn interface, so they drop into a pipeline unchanged.
That combination is, in practice, the strongest default for structured data, and it is what lesson 3 of the Introduction to AI course was pointing at.
In three sentences
scikit-learn's contribution is a single interface — fit, predict, transform — shared by every model, which makes comparing approaches cheap and made the library the standard for classical machine learning. Pipelines chain preparation and modelling into one object, which is what guarantees identical treatment in production and keeps cross-validation honest. Its most valuable lesson is data leakage: information reaching the model that would not exist at prediction time, which raises your score and destroys your result.