#svm-margins — Supervised learning

Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.

What you'll play with

Welcome to #svm-margins. On the plane: 80 points in two classes, blue and pink, well separated. The pink line is the separating hyperplane found by a linear SVM with C = 1; the two yellow dashed lines are the edges of its margin, the empty band it leaves on each side. Infinitely many lines separate these two clouds. Out of them all, the support vector machine picks the one that leaves the most room: the widest possible margin. Think of a corridor traced down the middle of a street rather than hugging the sidewalk: fewer passers-by will cross it.
Who decides where this line sits? Not the 80 points: only the ones touching the margin. Type /vectors to ring them in yellow.
The C parameter sets the price of a misplaced point (inside the margin or on the wrong side). Push it to the minimum: /c 0.01.
The opposite: push it to the maximum: /c 100. Every point in the margin now costs a fortune.
Change problem: /dataset rings. One class at the centre, the other all around it.
Switch to the Gaussian kernel: /kernel rbf. The kernel projects the points into a space where they become separable by a hyperplane; seen from our plane, that hyperplane becomes a curve.
gamma sets the reach of each point in the RBF kernel. Crank it up: /gamma 10.
Your turn: /gamma 0.3 for a smooth boundary, /dataset overlap then /c 0.1 and /c 100 to watch C trade margin width against misclassified points, /margin to hide or show the margin edges, /train 10 to catch the optimizer before it converges, /seed 12 for another draw, /reset to start over. To judge these boundaries with more than your eyes, head to #classification-metrics; for a method that draws no explicit boundary at all, #knn.

Channel commands

/c <C=0.01..100> — Sets the C parameter: the price of a point inside the margin (small = wide margin, large = hard margin).
/kernel <linear|rbf> — Picks the kernel: linear (a line) or RBF (the boundary curves).
/gamma <gamma=0.1..10> — Sets gamma, the reach of each point in the RBF kernel (small = smooth boundary, large = jagged).
/vectors — Shows or hides the yellow rings around the support vectors.
/margin — Shows or hides the two margin edges (yellow dashed lines, linear kernel).
/train <passes=10..500> — Sets the number of optimization passes (SMO) and retrains: few passes = coarse solution.
/dataset <separable|overlap|rings> — Switches the 2D dataset and retrains the current SVM.
/noise <0..1> — Sets the point spread (0 = tight clouds, 1 = very spread out) and retrains.
/seed <1..99> — Changes the random draw seed (same dataset, another sample) and retrains.
/reset — Returns to the initial state: separable, linear kernel, C = 1, gamma = 1, 200 passes.

Glossary

Support vector machine (SVM): Supervised classifier that looks for the separating hyperplane leaving the widest possible margin between the two classes. Its boundary depends only on a handful of points, the support vectors.
Separating hyperplane: The set of points where w·x + b = 0: a line in the plane, a plane in 3D, a hyperplane beyond. The sign of w·x + b gives the predicted class.
Margin: Distance between the hyperplane and the points closest to each class, i.e. 1/||w|| once we normalise y·(w·x + b) ≥ 1. The SVM maximises it: a wide margin tolerates new points better.
Support vector: Training point sitting on the margin edge or inside it (y·f(x) ≤ 1), i.e. with a non-zero α coefficient. Only these points determine the boundary; the others could disappear without changing a thing.
C parameter (regularization): Price paid for each misplaced point in the objective ½||w||² + C·Σξᵢ. Small C: strong regularization, wide margin, errors tolerated. Large C: narrow margin hugging the points, risk of overfitting.
Hinge loss: Loss max(0, 1 − y·f(x)): zero as soon as a point is on the right side and beyond the margin, growing after that. This is what the SVM minimises together with the regularization; its mean measures how well the data respects the margin.
Kernel: Function K(x, z) that measures the similarity of two points as an inner product in a higher-dimensional space, without ever going there. Linear: x·z; RBF: exp(−gamma·||x − z||²).
Kernel trick: Replace every inner product in the algorithm by K(x, z): the SVM then finds a maximum-margin hyperplane in kernel space, which appears as a curved boundary in the original plane. The cost stays O(n²) whatever the implicit dimension.
RBF (Gaussian) kernel and gamma: K(x, z) = exp(−gamma·||x − z||²): each support vector emits a Gaussian bump. gamma sets its reach (≈ 1/√(2·gamma)): small, the boundary is smooth; large, it twists around each point.
Soft margin / hard margin: Hard margin: no point may enter the margin (separable data, C → ∞). Soft margin: overshoots ξᵢ are allowed at price C, which makes the SVM usable on noisy or non-separable data.

Other channels in Supervised learning

#live-training — Six algorithms learning before your eyes, like a video: REC, timecode, subtitles, live metrics. Watching is free; touching the model is Premium.
#linear-regression — Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.
#logistic-regression — Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.
#decision-trees — A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.
#knn — k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.
#svm-margins — Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.
#classification-metrics — Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.