Module 8 — Bayes' theorem and probabilistic reasoning

Bayes' theorem is one of the most useful — and most misunderstood — formulas in the whole field. It answers a pervasive question: how do I update what I believe when new data arrives? Module 7 posed the problem (conditioning is not symmetric); Bayes gives the exact solution.

The formula and its intuition

$$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}$$

Let's break it down in plain terms, with $A$ = "being ill" and $B$ = "positive test":

• $P(A)$ — the prior: what we believed before the test (frequency of the disease).
• $P(B \mid A)$ — the likelihood: the test's reliability among the ill.
• $P(A \mid B)$ — the posterior: what we believe after seeing the positive test.

Bayes is a machine for revising a belief: start from a prior, an observation arrives, obtain a posterior. That posterior becomes the prior for the next observation — the reasoning refines indefinitely.

The example that defeats intuition: the base-rate trap

A disease affects 1 person in 1000. A test detects it at 99% among the ill and errs at 5% among the healthy (false positives). Your test is positive: what is the probability you are actually ill?

Intuition whispers "about 99%." Reality is quite different. Out of 100,000 people:

• 100 are ill → 99 test positive (true positives).
• 99,900 are healthy → about 4,995 test positive wrongly (false positives).

$$P(\text{ill} \mid \text{positive}) = \frac{99}{99 + 4995} \approx 1.9\,\%$$

Less than 2%. The reason: the disease is so rare (low base rate) that false positives, though infrequent in proportion, crush the true positives in number. Ignoring the base rate is one of the most widespread reasoning errors — including among professionals.

The naive Bayes classifier

Bayes isn't just an exercise: it is a real classification algorithm. The naive Bayes classifier estimates $P(\text{class} \mid \text{features})$ via Bayes' formula, adding a simplifying assumption — features are assumed independent of one another (hence "naive," module 7).

That assumption is almost always false strictly speaking, yet the algorithm works remarkably well, notably for spam filtering: each word of an email brings a clue, and Bayes combines all those clues into a probability that the message is unwanted. Fast, lightweight, interpretable — an excellent starting point for text classification.

Bayesian thinking, beyond the formula

Retain above all the way of thinking: a belief is never fixed, it updates as evidence accumulates. This is the foundation of a whole current of AI (Bayesian statistics, uncertainty quantification, A/B tests). When a model shows "92% confidence," that's a posterior: the fruit of a prior revised by training data.

Summary

• Bayes' theorem links the two directions of conditioning: it turns $P(B \mid A)$ into $P(A \mid B)$ via the prior.
• It reads as a belief revision: prior → observation → posterior, repeated indefinitely.
• The base-rate trap: when an event is rare, even a reliable test yields mostly false positives; ignoring the base frequency badly distorts reasoning.
• The naive Bayes classifier applies Bayes assuming independent features; simplistic but effective, it excels at spam filtering.

Next module: expectation, variance and common distributions — the statistics that summarize a distribution and quantify spread.