Lesson 3 — Features decide everything
Beginners spend their time choosing algorithms. Experienced practitioners spend it on features, because a model can only exploit information that is present in its input, and no algorithm recovers a signal that is not there.
What a feature is
A feature is one input variable the model reads. If your data is a table, each column is a candidate feature — but the columns you happen to have are rarely the features you want.
Consider predicting whether a customer will churn. Your database gives you a signup date and a list of orders. Neither is directly usable. What predicts churn is more likely:
- Days since last order — computed from the order list
- Order frequency trend — orders in the last 30 days compared with the previous 30
- Change in average basket — is the customer spending less than they used to
- Support tickets in the last week — a spike is a warning
- Ratio of days active to days since signup — engagement, normalised for tenure
None of those existed in your database. Every one is constructed, and each encodes a piece of understanding about how customers behave. That construction is feature engineering, and it is where domain knowledge enters the model.
A feature that captures a real mechanism gives the model information it could not have derived on its own from the raw columns. Switching from a random forest to gradient boosting typically moves your metric by a small margin; adding "days since last order" to a churn model can move it dramatically. The leverage is not comparable.
The transformations you will always need
Categorical columns
Models consume numbers, so text categories must be encoded. How you do it matters.
One-hot encoding creates one binary column per category. colour becomes colour_red, colour_blue, colour_green. Correct and safe, and it explodes when a column has thousands of distinct values — a postcode column can add ten thousand columns, most of them almost always zero.
Ordinal encoding maps categories to integers. Legitimate only when the order is real: small, medium, large becomes 1, 2, 3 sensibly. Applying it to red, blue, green tells the model that green is three times red, which is meaningless and will be used anyway.
Target encoding replaces each category with the average outcome for that category. Powerful on high-cardinality columns, and dangerous: computed over the whole dataset it leaks the answer straight into the feature. It must be computed inside cross-validation folds, which is precisely what a scikit-learn pipeline enforces.
Dates and times
A raw timestamp is almost useless as a number, and enormously useful once decomposed. From one date you can extract hour of day, day of week, month, whether it is a weekend, whether it is a public holiday, and days elapsed since some reference event.
The subtlety worth knowing: cyclical features need care. Encoding the hour as 0 to 23 tells the model that 23:00 and 00:00 are maximally distant, when they are adjacent. The standard fix represents the hour as a pair of coordinates on a circle, which restores the true adjacency.
Missing values
Real data has holes, and how you fill them is a modelling decision rather than a cleanup chore.
Filling with the median is a safe default for numbers. Filling with the mean is more sensitive to outliers. Filling with a sentinel value like -1 works for tree models, which can isolate it. Dropping rows is acceptable when few are affected and dangerous otherwise.
The point that gets missed: the fact that a value is missing is often informative. A customer with no recorded income may have declined to provide it, which correlates with something. Adding a binary "was missing" column beside the filled value preserves that signal, and it frequently helps.
Scaling
Some models care about the magnitude of your numbers and some do not, and knowing which saves confusion.
Scaling matters for anything based on distances or gradients: k-nearest neighbours, support vector machines, k-means, PCA, and neural networks. Without it, a salary column measured in tens of thousands drowns out an age column measured in tens, purely because of units.
Scaling does not matter for tree-based models. A tree asks whether a value is above a threshold, and that question is unaffected by units.
Two common forms: standardisation centres each feature at zero with a spread of one, and normalisation squeezes each into a fixed range such as 0 to 1. Standardisation is the usual default.
Features you must not use
Some features improve your score and destroy your model. These are the leakage cases from the Python course, seen from the feature side.
Anything recorded after the event you predict. A cancellation_reason column in a churn model gives near-perfect accuracy and cannot exist at prediction time, when the customer has not cancelled yet.
Anything unavailable in production. A feature computed from a nightly batch job is not available for a real-time prediction. The offline model works, the online one cannot be built.
Proxies for protected attributes. A postcode can encode ethnicity, a first name can encode gender, and a model using them discriminates while never seeing the protected attribute itself. Removing the sensitive column is not sufficient, which is the central practical difficulty in AI ethics.
Will this value be available, computed the same way, at the moment I need a prediction? If the answer is no or unclear, the feature is not usable, regardless of how much it helps offline. Asking this early prevents rebuilding a model from scratch after it fails to deploy.
Which features mattered
After training, most models can report feature importance: which inputs the model relied on. This is useful in two directions.
It validates the model. If the top features match what a domain expert would expect, confidence rises. If a meaningless identifier column ranks first, you have found a leak.
It simplifies the model. Frequently a handful of features carry nearly all the signal, and dropping the rest gives you something faster, more robust and easier to explain, at negligible cost in accuracy.
Two cautions. Importance shows what the model used, not what causes the outcome — it remains correlation, and reading causation into it is the expensive mistake from the maths course. And when two features are strongly correlated, the model may lean on one arbitrarily, making the other look unimportant when it carries the same information.
In three sentences
A feature is what the model gets to see, and the columns in your database are rarely the features you want, so constructing them is where domain knowledge enters and where the largest gains usually come from. Categories, dates, missing values and scale all need deliberate treatment, and each choice is a modelling decision rather than housekeeping. The test that keeps a project alive is asking, of every feature, whether it will be available and computed identically at prediction time.