Module 8 — Quality metrics: FID and inception score

Every previous module treated "the samples look plausible" as the informal quality signal. That is fine when comparing checkpoints of the same run, useless when comparing two runs, two models, two published papers. This module introduces the two dominant automated metrics — inception score and FID — and, just as importantly, the situations in which both mislead.

Inception score: diversity and confidence in one number

The inception score (Salimans 2016) tries to capture two intuitions at once. A good generator produces images that (a) a classifier can recognise with high confidence (each image looks clearly like something) and (b) span many classes (the marginal distribution across generated images is spread out).

Formally, run each generated image $x$ through a pretrained Inception V3 classifier, get the class probability distribution $p(y \mid x)$, and compute the KL divergence to the marginal $p(y) = \mathbb{E}_x p(y \mid x)$:

$\text{IS} = \exp\Big( \mathbb{E}_x \, D_{\text{KL}}(p(y \mid x) \,\|\, p(y)) \Big)$

High IS means high confidence per image and a diverse mix of predicted classes. Higher is better.

import torch
import torch.nn.functional as F
from torchvision.models import inception_v3

@torch.no_grad()
def inception_score(images, batch=32, splits=10):
    net = inception_v3(pretrained=True, transform_input=False).eval().to(images.device)
    net.fc = torch.nn.Identity()  # keep raw logits from the classifier's head
    scores = []
    for i in range(splits):
        chunk = images[i * len(images) // splits:(i + 1) * len(images) // splits]
        preds = torch.cat([F.softmax(net(x), dim=1) for x in chunk.split(batch)])
        p_y = preds.mean(0, keepdim=True)
        kl = (preds * (preds.log() - p_y.log())).sum(1).mean()
        scores.append(kl.exp().item())
    return sum(scores) / len(scores)

The two problems with IS are widely documented and worth stating clearly. It never looks at real images: it can be gamed by a generator that produces high-confidence, class-diverse images that share no statistical properties with the training set. And it depends critically on the classifier: an Inception V3 trained on ImageNet gives meaningful scores on ImageNet-like data and near-random ones on faces, medical images or line drawings.

FID: comparing distributions in Inception's feature space

The Fréchet inception distance (Heusel 2017) fixes both problems by comparing generated images to real ones through Inception's feature representation.

The recipe:

1. Take a large sample of real images and a large sample of generated images.
2. Pass all of them through Inception V3 and take the activations of the penultimate layer — a 2048-dimensional feature vector per image.
3. Compute the mean $\mu_r$, $\mu_g$ and covariance $\Sigma_r$, $\Sigma_g$ of each set of features.
4. Compute the Fréchet distance between the two Gaussians those statistics define:

$\text{FID} = \| \mu_r - \mu_g \|^2 + \text{Tr}\Big( \Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{1/2} \Big)$

Lower is better; a perfect match gives $0$. FID is now the standard reference metric across image generation papers.

import numpy as np
from scipy.linalg import sqrtm

def frechet_distance(mu_r, sigma_r, mu_g, sigma_g):
    diff = mu_r - mu_g
    covmean, _ = sqrtm(sigma_r @ sigma_g, disp=False)
    if np.iscomplexobj(covmean):
        covmean = covmean.real
    return diff @ diff + np.trace(sigma_r + sigma_g - 2 * covmean)

Two important practical points. FID is biased downwards for small samples: computing FID on 1,000 images gives a systematically better score than on 50,000, even for the same model. Papers that report FID must state the sample size. And the number is only comparable on the same dataset with the same reference statistics: an FID of 5.2 on CIFAR-10 is not the same as an FID of 5.2 on CelebA.

FID pitfalls that matter in practice

FID is the reference metric, and yet it can be misled in several documented ways.

A model that memorises the training set has a very low FID and no generative value. Since FID compares statistics of feature vectors, a generator that outputs training images directly matches every moment perfectly. Papers routinely report FID against a held-out test set and add a nearest-neighbour check to detect memorisation, and you should too.

FID does not tell you where the distribution mismatch is. A high FID could mean the generator misses a whole class, or produces the right classes with a wrong colour bias, or produces flawless images with too much sharpness compared to the training set. The number gives no diagnostic. Precision-recall metrics (below) fix this.

FID depends on the resolution. Inception V3 was trained at 299 by 299 pixels. Comparing generators that output 128 by 128 to generators that output 512 by 512 requires deciding on a common resizing strategy, and different codebases have made different choices — enough that the FID reported in two papers is not always comparable.

Two models with the same FID can be very different generators

The distribution-fitting nature of FID means that a generator producing many blurry-but-diverse images and a generator producing few sharp-but-similar images can score the same FID by different routes. You cannot decide which one is "better" from FID alone; you have to state what "better" means for the target application.

Precision and recall for generation

Precision-recall for generation (Sajjadi 2018) decomposes the FID intuition into two axes. Precision measures the fraction of generated samples that are indistinguishable from real ones — high precision means high sample quality. Recall measures the fraction of real samples that are close to some generated ones — high recall means the generator covers the whole data distribution.

The two decouple problems that FID conflates. A GAN with mode collapse has high precision, low recall: every image it produces looks real, but a large portion of the real distribution is unreachable. A VAE with blurry outputs has low precision, high recall: every real digit style has a matching blurry sample, but the samples themselves are not indistinguishable from reals.

The computation uses k-nearest-neighbour manifolds in Inception feature space, and standard implementations exist in every deep learning framework.

Human evaluation is not optional

Every automated metric — IS, FID, precision, recall, and their newer cousins CLIP score, DINO similarity, KID — captures a slice of image quality that correlates imperfectly with human judgement. Two studies published in 2023 and 2024 showed that FID rankings among top image generators disagreed with human preference rankings in one third of pairwise comparisons.

The consequence for practice: on any consequential release, plan for a human evaluation on a controlled protocol. A common design is a two-alternative forced choice: show a human judge two images side by side, one from model A and one from model B, on the same prompt, and ask which is more realistic. Aggregate over hundreds of judges and thousands of pairs. The result is expensive and slow but decisive.

Automated metrics for iteration, human evaluation for release

Use FID and its cousins to compare checkpoints of the same model across a training run, or to compare small architectural variants where relative ranking matters more than absolute value. Rely on human evaluation only when the decision to ship or not depends on it. Investing more automated metric compute past a threshold rarely changes anything.

What to report in a serious evaluation

A reproducible generation-quality report for an image model typically states:

• FID on a large fixed test set (at least 10,000 images each side), with the reference statistics and code version noted.
• Precision and recall at a stated $k$.
• A nearest-neighbour visualisation: for a handful of generated samples, show the closest training image. This catches memorisation that no scalar metric will.
• The sample count used for the metric — a number without this is uninterpretable.
• A fixed prompt set if the model is conditional, so that later reproductions use the same conditioning.

Anything less lets the number be silently non-comparable, which is what has happened repeatedly with FID-based leaderboards.

In summary

• The inception score rewards confident and diverse predictions from a pretrained classifier, but ignores real images and depends on the classifier's training data.
• FID compares Inception-space statistics of real and generated samples; lower is better, it is the reference metric, and it is biased downwards for small samples.
• FID is fooled by memorisation and gives no diagnostic when it flags a mismatch; precision-recall decouples sample quality from distribution coverage.
• Automated metrics rank models imperfectly relative to human preference; a serious release plans for human evaluation on a controlled protocol.

Next module: how the three families of this course extend to text, audio and multimodal generation, and which family wins on which modality.