Module 2 — Linear regression and Ridge/Lasso regularization

Linear regression is the mandatory starting point: simple, fast, interpretable, and already carrying every concept — cost, optimization, regularization — that reappears in the most complex models. It is also the baseline against which everything else is judged.

The model: a weighted sum of features

Linear regression predicts the target as a combination of the features, each multiplied by a coefficient:

$$\hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b$$

Each coefficient $w_i$ says by how much the prediction changes when feature $x_i$ increases by one unit, all else equal. This is what makes the model so readable: you can read the coefficients as each feature's influence. Training seeks the coefficients that minimize the mean squared error, via the gradient descent of the mathematics course.

from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X_train, y_train)
model.coef_        # one coefficient per feature
model.intercept_   # the bias b

Interpreting coefficients with care

A coefficient is only readable if the features are on the same scale. Otherwise, a large coefficient may merely reflect a small-amplitude feature, not its importance. Hence the reflex to standardize before comparing coefficients — a direct echo of the norms from the mathematics course. Also beware: coefficients measure association, not causation, and two correlated features "share" an effect in sometimes misleading ways (collinearity).

Why regularize: taming variance

When features are numerous or correlated, ordinary linear regression can produce huge, unstable coefficients: it clings to the noise of the sample. This is excess variance — the overfitting from the mathematics course. Regularization adds a penalty on coefficient size to the cost, forcing the model to stay sober:

$$\text{Cost} = \text{error} + \lambda \cdot \text{penalty on coefficients}$$

The parameter $\lambda$ sets the strength of the brake: at zero, you recover ordinary regression; too large, the model becomes too simple (excess bias). It's the bias-variance trade-off, set by a slider.

Ridge and Lasso: two penalties, two behaviors

from sklearn.linear_model import Ridge, Lasso
Ridge(alpha=1.0).fit(X_train, y_train)   # L2 penalty
Lasso(alpha=0.1).fit(X_train, y_train)   # L1 penalty

	Penalty	Effect on coefficients	Typical use
Ridge	$L_2$ (sum of squares)	shrinks them all, none to zero	correlated features, keep all info
Lasso	$L_1$ (sum of absolute values)	sets some exactly to zero	automatic feature selection

The distinction is concrete and valuable. Ridge stabilizes against collinearity without dropping any feature. Lasso does selection: by zeroing the least useful coefficients, it produces a sparse model, keeping only the features that matter — a major asset when there are hundreds. The geometry behind it (the "corner" of the $L_1$ ball touching the axes) explains why only the $L_1$ penalty zeroes coefficients.

Elastic Net: the compromise of the two

When in doubt — many features and collinearity — Elastic Net combines both penalties and offers the best of both worlds: Lasso's selection and Ridge's stability. In practice, try Ridge and Lasso as baselines, then Elastic Net if neither clearly dominates. In all cases, $\lambda$ (called alpha in scikit-learn) is tuned by cross-validation, never by eye.

Summary

• Linear regression predicts a weighted sum of the features; its readable coefficients measure association and require features on the same scale.
• Without a guardrail, it overfits when features are numerous or correlated: excess variance.
• Regularization penalizes coefficient size; $\lambda$ sets the bias-variance trade-off.
• Ridge ($L_2$) shrinks all coefficients; Lasso ($L_1$) zeroes some and selects features; Elastic Net combines both.

Next module: logistic regression, which adapts this linear machinery to classification and introduces the decision boundary.