Lesson 2 — The algorithms and when to use them
There are hundreds of algorithms and you need roughly six. This lesson covers what each family does, what it is good at, and where it fails — because choosing badly is much more often about the shape of your data than about the sophistication of the method.
Linear and logistic regression
The simplest useful models, and far from obsolete.
Linear regression predicts a number as a weighted sum of the inputs. Logistic regression does the same and then squashes the result into a probability between 0 and 1, which makes it a classifier despite the name.
Their enduring advantage is transparency. The model is a list of coefficients: this feature pushes the prediction up by that much. You can print it, explain it to a regulator, and notice immediately when a coefficient has an implausible sign — which is one of the best available signals that something is wrong with your data.
Use them when you need to explain the decision, when you have few examples relative to features, or as the baseline that anything more complex must beat.
They fail when the relationship is not roughly additive. Logistic regression cannot learn "risky only if young and recently registered" unless you construct that combination yourself as a feature.
Decision trees
A tree asks a sequence of yes-or-no questions. Is tenure under six months? Then, were there more than two support tickets? Then, predict churn.
A single tree is wonderfully interpretable — you can draw it and follow any decision — and it handles non-linear relationships and feature interactions naturally, which linear models cannot.
Its fatal weakness is instability. Grown deep enough, a tree isolates individual training examples in their own leaves, which is memorisation in its purest form. And changing a few training rows can produce a completely different tree, which makes a single tree hard to trust.
Use one when you want a visual explanation of the logic, or a first look at which features matter.
Never use one alone as your production model. Which leads to the family that fixed this.
Ensembles: the answer for tabular data
The insight that made trees the strongest tool on structured data: combine many weak trees instead of trusting one good one.
Random forests train hundreds of trees, each on a random subset of the rows and of the features, and average their predictions. The individual trees are mediocre and their errors are uncorrelated, so averaging cancels much of the noise. Robust, hard to misuse, and requiring little tuning.
Gradient boosting takes a different route: train a small tree, look at what it got wrong, then train the next tree specifically to correct those mistakes, and repeat hundreds of times. Each tree fixes the residual errors of the ensemble so far.
Boosting generally beats forests on accuracy and demands more care, since it can overfit if you let it run too long. The three implementations you will meet are XGBoost, LightGBM and CatBoost; they differ in engineering rather than in principle, with LightGBM typically fastest on large data and CatBoost handling categorical columns most gracefully.
On a table of structured data, gradient boosting is the strongest general-purpose starting point, and it regularly beats neural networks on the same data while training in seconds and telling you which features drove the result. If someone proposes deep learning for a spreadsheet, this is the comparison to ask for.
Support vector machines and k-nearest neighbours
Two families worth recognising, less often reached for today.
Support vector machines find the boundary that separates classes with the widest possible margin, and can handle curved boundaries through a mathematical trick that avoids ever computing the higher-dimensional space explicitly. Genuinely strong on small datasets with many features — text classification was their classic home — and they scale poorly, becoming impractical past tens of thousands of examples.
k-nearest neighbours does not really train at all. To classify a new point it finds the k most similar training examples and takes a vote. Beautifully simple, and it degrades badly in high dimensions, where the curse of dimensionality makes every point roughly equidistant from every other. Its modern descendant is much more important than the classifier: vector search, which is exactly nearest-neighbour lookup over embeddings, and is the retrieval half of every RAG system.
The unsupervised toolkit
k-means partitions data into k clusters by repeatedly assigning points to the nearest centre and recomputing the centres. Fast and widely used. Two limitations to keep in mind: you must choose k in advance, and it assumes clusters are roughly spherical and similar in size, so it handles elongated or nested shapes poorly.
DBSCAN groups points by density instead, which means it discovers the number of clusters itself, finds arbitrarily shaped groups, and labels sparse points as noise rather than forcing them into a cluster. Better suited to messy real data, and more sensitive to its distance settings.
PCA reduces dimensions by keeping the directions of greatest variance. Used for compression, for removing redundancy, and as a preprocessing step. t-SNE and UMAP are for visualisation only: they produce beautiful two-dimensional maps of high-dimensional data, and the distances in those pictures should not be interpreted quantitatively, a caveat that is ignored constantly.
When deep learning is the answer
Deep learning takes over when the input is raw and unstructured, because that is when nobody can hand-craft the features:
- Images: pixels have no meaningful column names
- Text: for anything beyond word counting
- Audio: waveforms
- Sequences with long-range dependencies
And it is usually the wrong first choice on a table, for the reasons above.
The map
| Your situation | Reach for | Why |
|---|---|---|
| Table, want the best accuracy | XGBoost / LightGBM | strongest default on structured data |
| Table, must explain each decision | logistic / linear regression | the model is a readable list of coefficients |
| Table, want a robust result with little tuning | random forest | forgiving, hard to misuse |
| Need a baseline to beat | linear model, or always predict the majority | if nothing beats this, stop and check the data |
| No labels, want groups | k-means, or DBSCAN on messy shapes | one needs k, the other finds it |
| No labels, want the unusual | isolation forest, DBSCAN noise points | anomalies are too rare to label |
| Too many features | PCA | keeps variance, drops redundancy |
| Images, text, audio | deep learning | features cannot be hand-designed |
| Few examples, many features | SVM, or a linear model with regularisation | both cope well with wide, short data |
No algorithm is best everywhere
There is a formal result behind this, usually called the no free lunch theorem: averaged over all possible problems, every algorithm performs identically. It sounds like a curiosity and it has a practical meaning — an algorithm's advantage comes from assumptions that happen to match your data, so the only way to know what works on your problem is to try several.
Which is exactly why the uniform scikit-learn interface from the Python course matters so much. Trying five algorithms is five lines, so there is no excuse for defending a choice by reputation instead of by measurement.
In three sentences
You need about six algorithm families, and the right one is decided by the shape of your data rather than by sophistication: gradient boosting for tables, linear models when you must explain, deep learning for raw images, text and audio. A single decision tree memorises and is unstable, which is why ensembles of many trees replaced it and became the strongest default on structured data. No algorithm wins everywhere, since each one's advantage comes from assumptions that may or may not fit your data, so trying several is the only reliable method.