Module 9 — Metrics: accuracy, precision, recall, F1, ROC AUC

Cross-validation says how to measure; this module says what to measure. Choosing the wrong metric means optimizing the wrong objective — with models that look excellent and decisions that are disastrous. Everything starts from a small table with big consequences: the confusion matrix.

The confusion matrix: the four possible outcomes

In binary classification, every prediction lands in one of four boxes:

	Predicted positive	Predicted negative
Actually positive	true positive (TP)	false negative (FN)
Actually negative	false positive (FP)	true negative (TN)

The two kinds of error rarely cost the same. Missing a fraud (FN) and blocking a legitimate customer (FP) are not the same harm — every metric is just a way of weighting these four boxes, and the business context dictates the choice.

from sklearn.metrics import confusion_matrix, classification_report
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))   # everything at once

Accuracy: intuitive, and dangerous when imbalanced

Accuracy — the share of correct predictions — is the metric everyone understands. It becomes a trap as soon as classes are imbalanced. With 1% fraud, the lazy model "never fraud" scores 99% accuracy while detecting zero fraud. The metric rewards blindness. Reflex: as soon as one class is rare, accuracy no longer means much — switch to metrics that look at the rare class.

Precision and recall: the fundamental trade-off

• Precision $= \frac{TP}{TP + FP}$ — of my positive alerts, what share are correct? Punishes false positives.
• Recall $= \frac{TP}{TP + FN}$ — of the actual positives, what share did I find? Punishes false negatives.

The two pull in opposite directions, and the module 3 threshold is precisely the slider: lower the threshold and recall climbs while precision drops; raise it and the reverse. Neither is "better" — everything depends on the cost of each error:

Context	Costly error	Priority
Cancer screening	missing a case (FN)	recall
Email spam filter	blocking a legitimate email (FP)	precision
Fraud with human review	FN, review absorbs the FPs	recall then precision

F1-score, the harmonic mean of precision and recall, condenses both into one number — useful for comparing models in one glance. The harmonic mean punishes imbalance: a model with high precision but near-zero recall gets an F1 near zero, not a flattering average.

ROC AUC: judging across all thresholds

The metrics above depend on the chosen threshold. The ROC curve removes that dependence: it traces, for every possible threshold, the true-positive rate against the false-positive rate. The area under this curve, the AUC, summarizes ranking quality: probability that a random positive receives a higher score than a random negative. AUC = 0.5 means no better than chance; 1.0 means perfect separation.

from sklearn.metrics import roc_auc_score
roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])   # scores, not classes

The AUC compares models independently of the threshold — ideal for selection; the threshold gets set afterward, by the business. On strongly imbalanced classes, its cousin PR AUC (area under the precision-recall curve) is more demanding and more revealing, because it focuses on the rare class.

Choose the metric before training

The metric is a framing decision (module 1), not an afterthought: it flows from the business cost of errors, and cross-validation must optimize it (scoring= in scikit-learn). Choosing it after the fact invites picking the one that flatters the model.

Summary

• The confusion matrix distinguishes the four outcomes; FP and FN rarely cost the same, and everything follows from that.
• Accuracy misleads on imbalanced classes — the "always majority" model scores high while detecting nothing.
• Precision (reliable alerts) and recall (positives found) pull oppositely; the threshold is the slider and F1 condenses them.
• ROC AUC judges ranking quality across all thresholds; PR AUC is preferable when the positive class is rare.

Final content module: hyperparameter tuning and the end-to-end project, where every piece of the course assembles.