#classification-metrics — Supervised learning

Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.

What you'll play with

Welcome to #classification-metrics. On the left, the scores a model has assigned to 600 examples: negatives in blue, positives in pink (30% of the total), and a yellow blade set at 0.5: the threshold. In the middle, the confusion matrix that follows: true positives (green), true negatives (blue), false positives (yellow), false negatives (red). A model never answers 'yes' or 'no': it gives a score, and you choose where to cut. Everything that follows shows what this choice costs.
Raise the threshold: type /threshold 0.8. The yellow blade slides to the right: almost no negative gets through… but many positives stay below it.
Do the opposite: /threshold 0.2. This time we catch almost every positive… and a lot of negatives along with them.
Every threshold is a different trade-off. To see them all at once, draw the ROC curve: /roc.
Now a real-world case: fraud, a rare disease, a breakdown… Keep only 5% positives: /imbalance 0.05.
Where to cut, then? Let the machine find the threshold that maximises F1: /optimize f1.
The only real fix is a better model. Push the two distributions apart: /separation 2.5.
Your turn: /precision-recall for the precision-recall curve (more honest than ROC when positives are rare), /optimize youden for the threshold that maximises TPR − FPR, /noise 0.25 for a shakier model, /imbalance 0.5 for balanced classes, /n 2000 to smooth the histograms, /seed 42 for another draw, /reset to start over. Then: #logistic-regression, where these scores are built, and #overfitting, where we learn not to fool ourselves on the training data.

Channel commands

/threshold <0..1> — Moves the decision threshold: positive if score ≥ threshold.
/separation <0..3> — Model quality: gap between the scores of the two classes.
/imbalance <0.05..0.5> — Positive prevalence: 0.05 = rare class, 0.5 = balanced classes.
/roc — Shows or hides the curve (ROC or precision-recall) on the right.
/precision-recall — Toggles between the ROC curve and the precision-recall curve.
/optimize <f1|precision|recall|youden> — Places the threshold that optimizes the chosen criterion.
/n <100..2000> — Number of simulated examples (new draw).
/noise <0.05..0.3> — Score standard deviation σ: the larger it is, the more the classes overlap.
/seed <1..99> — Another random draw, same parameters.
/reset — Returns to threshold 0.5, separation 1.5, prevalence 0.3, 600 examples, curve hidden.

Glossary

Decision threshold: The value t above which a score is declared positive. The model does not pick it: it is a business dial that trades false positives against false negatives, and each dial position gives a different confusion matrix.
Confusion matrix: A 2×2 table crossing reality and prediction: true positives (TP, positives correctly caught), true negatives (TN), false positives (FP, false alerts) and false negatives (FN, missed positives). Every classification metric can be read from these four cells.
Precision: Out of the examples predicted positive, the share that really is: TP / (TP + FP). Measures the reliability of an alert; you favour it when a false alert costs a lot (spam, moderation).
Recall (sensitivity): Out of the true positives, the share the model retrieves: TP / (TP + FN). Also the true positive rate (TPR), the y-axis of the ROC curve; you favour it when missing a positive costs a lot (screening, fraud).
Specificity: Out of the true negatives, the share correctly left aside: TN / (TN + FP). Its complement 1 − specificity is the false positive rate (FPR), the x-axis of the ROC curve.
F1-score: Harmonic mean of precision and recall: 2·P·R / (P + R). High only when both are high, and it ignores true negatives: the metric to read when the positive class is rare.
Accuracy: Share of correct predictions: (TP + TN) / n. Misleading as soon as the classes are imbalanced: with 5% positives, always answering 'negative' scores 95% while detecting nothing.
ROC curve and AUC: Curve of the true positive rate against the false positive rate for every possible threshold; the diagonal is chance. The AUC (area under the curve) rates the ranking independently of the threshold: probability that a positive receives a higher score than a negative, 0.5 = chance, 1 = perfect.
Precision-recall curve: Precision against recall for every threshold. Because it ignores true negatives, it stays strict when positives are rare, where ROC flatters the model; its baseline is the prevalence, not the diagonal.
Class imbalance: A situation where one class is far more common than the other (fraud, rare disease, breakdown). Accuracy becomes useless; read precision, recall, F1 or the precision-recall curve instead, and pick the threshold with full knowledge of the trade-off.

Other channels in Supervised learning

#live-training — Six algorithms learning before your eyes, like a video: REC, timecode, subtitles, live metrics. Watching is free; touching the model is Premium.
#linear-regression — Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.
#logistic-regression — Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.
#decision-trees — A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.
#knn — k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.
#svm-margins — Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.
#classification-metrics — Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.