Skip to main content

Module 3 — Logistic regression and the decision boundary

Despite its name, logistic regression is a classification model — the most widely used for two-class problems. It reuses the linear machinery of module 2 and adds a single idea: turning a score into a probability. Simple, fast, interpretable, it remains the baseline to beat in classification.

From linear score to probability: the sigmoid​

Logistic regression first computes a linear score, exactly as in module 2, then passes it through the sigmoid function, which squashes any real number into the interval [0,1][0, 1]:

p=σ(w⋅x+b)=11+e−(w⋅x+b)p = \sigma(w \cdot x + b) = \frac{1}{1 + e^{-(w \cdot x + b)}}

The result pp reads as a probability of belonging to the positive class. A very negative score gives a probability near 0, a very positive one near 1, and a zero score gives exactly 0.5. The sigmoid's S-curve is smooth: near the boundary, a small change in the score notably shifts the probability; far from it, the effect fades.

from sklearn.linear_model import LogisticRegression
model = LogisticRegression().fit(X_train, y_train)
model.predict_proba(X_test)[:, 1]   # probabilities of the positive class
model.predict(X_test)                # decisions at the default 0.5 threshold

The decision boundary and the threshold​

To decide between the two classes, you compare the probability to a threshold — 0.5 by default. The set of points where p=0.5p = 0.5 forms the decision boundary: for logistic regression, a line (or a hyperplane in higher dimension). On one side, the positive class; on the other, the negative.

The threshold is not set in stone. Moving it shifts the trade-off between the two kinds of error:

  • Lower the threshold (e.g. 0.3): you classify positive more easily → more true positives detected, but more false positives. You favor recall.
  • Raise the threshold (e.g. 0.7): you announce positive only when confident → fewer false positives, but you miss some. You favor precision.

This setting is steered by the business cost of errors — detecting fraud has different stakes than recommending a movie. The metrics of module 9 formalize this trade-off.

Interpreting coefficients: direction and strength​

As in linear regression, each coefficient says which way a feature pushes the decision: a positive coefficient raises the probability of the positive class, a negative one lowers it. Its magnitude (on standardized features) indicates the strength of the effect. More precisely, a coefficient acts on the log-odds, but the intuition "sign = direction, size = strength" suffices in the vast majority of cases to explain a decision to a business stakeholder.

Regularization and multiclass, no surprises

Logistic regression is regularized exactly as in module 2: scikit-learn actually applies an L2L_2 penalty by default (set by the parameter C, the inverse of λ\lambda — a small C regularizes strongly). For more than two classes, the one-vs-rest trick trains a binary classifier per class and keeps the most probable; it's automatic in scikit-learn. You thus keep a simple linear model where many would rush to something more complex.

Summary​

  • Logistic regression computes a linear score then converts it to a probability via the sigmoid ([0,1][0,1]).
  • You decide by comparing the probability to a threshold; the set of points at p=0.5p = 0.5 is the decision boundary, linear.
  • Moving the threshold trades off precision and recall, depending on the business cost of errors.
  • Coefficients interpret as in linear regression (sign = direction, size = strength); L2L_2 regularization and multiclass mode are built in.

Next module: k-nearest neighbors and support vector machines — two geometric approaches that draw boundaries of an entirely different nature.