Module 1 — Why features matter more than the algorithm

The two previous courses presented some fifteen algorithms. Here is the uncomfortable truth practice imposes: on tabular data, moving from a logistic regression to a finely tuned gradient boosting gains a few points. Adding the right feature sometimes gains twenty. This course covers the step that really decides, and to which practitioners devote most of their time.

A feature as a representation, not as raw data

A feature is not a column of the source file: it is a way of presenting information to the model. The same reality can be described in several ways, and they are not equivalent.

Take a date of birth. As is, it is a character string no model can exploit. Converted into age, it becomes an ordered, useful number. Split into age brackets, it captures non-linear threshold effects. Crossed with place of residence, it can reveal a demographic profile. One original column, four features of increasing quality — and the model itself changed nothing.

This is the heart of the craft: a model learns regularities in the space you give it. If the useful information is not expressed there, no quantity of trees will invent it.

Why an algorithm cannot rescue a poor representation

Take an elementary example. You want to predict whether a point in the plane lies inside a circle centered at the origin. With coordinates $x$ and $y$ as the only features, a linear model fails: the boundary is a circle, and no straight line approximates it. Add the feature $r = \sqrt{x^2 + y^2}$ and the problem becomes trivial — a single comparison to a threshold suffices.

Nothing changed in the information available: $r$ is entirely derived from $x$ and $y$. What changed is that the information is now expressed in the form the model can exploit. This is exactly what the SVM kernel trick does, or what deep layers learn on their own; feature engineering does it explicitly, and interpretably.

What a good feature does

Three criteria let you judge a candidate feature, and sort ideas before writing code:

• it carries new information, not already contained in the others. A feature strongly correlated with an existing one costs complexity and adds nothing;
• it is available at prediction time. This is the framing rule from the supervised course, and the main source of disillusion in production;
• it has an explainable business meaning. A feature you cannot justify is a feature you will not be able to defend, nor fix when it drifts.

Domain knowledge as raw material

Where do good features come from? Almost always from understanding the phenomenon, not from statistical inspiration. To detect bank fraud, the raw amount of a transaction says little; the gap between that amount and the customer's habit says a great deal. That feature is not invented by staring at a table: it comes from knowing how fraud works.

Hence a practical consequence for how you work: the best features are born from a conversation with those who know the business, not from an automated loop over possible combinations. Automated methods have their place, but they explore the space you defined; they do not redefine it.

Where this step sits in the project

Let us situate the work properly. Preparation — fixing missing values, outliers, duplicates — makes the data usable; it is a prerequisite, not value creation. Feature engineering transforms and combines to express useful information. Selection (module 9) then prunes what contributes nothing.

This course follows that order: handle (modules 2 and 3), encode and build (modules 4 to 7), clean up (module 8), select (module 9), industrialize (module 10).

Most of the time goes here, and that is normal

Industry surveys regularly put between 60 and 80% the time spent on preparing and building features. This is not a symptom of poor organization: it is where performance is decided. The opposite temptation — cycling through models on mediocre features hoping an algorithm saves the day — is the most expensive misconception in the field.

Summary

• A feature is a representation of information, not a raw column; the same reality admits expressions of very unequal quality.
• A model only learns what the feature space makes expressible: the feature $r$ solves a problem no straight line can.
• A good feature carries new information, is available at prediction time, and has a defensible business meaning.
• The raw material is domain knowledge; automated methods explore the space you define, they do not create it.

Next module: missing values, the first concrete obstacle, and the three strategies for handling them without introducing bias.