Module 10 — SavedModel and serving with TensorFlow Serving

A model that is not served has produced no value. This module covers the last stretch: getting the model out of the notebook and making it queryable, without repeating the mistake that breaks half of all deployments.

Two formats, two purposes

Keras offers two output formats, and confusing them costs time.

Format	Command	Contains	Use
.keras	model.save("m.keras")	architecture, weights, optimiser state	resume training
SavedModel	model.export("m/")	optimised graph and signatures	serve in production

The .keras format is an archive aimed at Python: it lets you reload the model and continue training where it left off. A SavedModel is a directory holding a serialised graph, with no dependency on Python: it is what TensorFlow Serving, TensorFlow Lite or a C++ runtime know how to read.

model.save("models/classifier.keras")   # to resume training
model.export("models/served/1")         # for production

That trailing 1 is not decorative: it is the version number, and TensorFlow Serving requires it.

Preprocessing must live inside the model

This is the most important point in the module. In module 8, images went through preprocess_input before entering the network. If that step stays in your Python training script, it does not exist in the exported SavedModel. The service then receives raw pixels and applies a network that expects normalised values.

The result is the worst possible case: no error, no alert, and predictions degraded in a way that is hard to trace back. This is training-serving skew, and it almost always shows up in preprocessing.

The fix is to build preprocessing in as model layers:

from tensorflow import keras
from tensorflow.keras import layers

inputs = keras.Input(shape=(None, None, 3), dtype="uint8", name="image")
x = layers.Resizing(224, 224)(inputs)
x = layers.Rescaling(1.0 / 127.5, offset=-1.0)(x)
x = base(x, training=False)
x = layers.GlobalAveragePooling2D()(x)
outputs = layers.Dense(num_classes, activation="softmax")(x)

served_model = keras.Model(inputs, outputs)
served_model.export("models/served/1")

The model now accepts images of any size as unsigned integers and handles resizing and normalisation itself. The client no longer needs to know anything about internal conventions. The Normalization, TextVectorization and StringLookup layers follow the same logic for tabular and text data: they fit to the data with adapt during training, then carry their statistics into the export.

Training-serving skew is the number one production bug

Every time a transformation exists in the training script but not in the exported artifact, it becomes a divergence. The decisive test fits in one sentence: does the SavedModel accept exactly what the client will send? If the answer assumes an upstream step, that step must join the model or be versioned alongside it.

Inspect before deploying

A SavedModel can be inspected without writing a line of Python, and the check takes ten seconds.

saved_model_cli show --dir models/served/1 --tag_set serve \
  --signature_def serving_default

The output describes the expected inputs and outputs:

inputs['image'] tensor_info:
    dtype: DT_UINT8
    shape: (-1, -1, -1, 3)
outputs['output_0'] tensor_info:
    dtype: DT_FLOAT
    shape: (-1, 10)

The -1 entries are free dimensions, including the batch. Two checks: the input dtype must match what the client will produce, and the output shape must match your class count. A shape: (-1, 1) where you expected ten classes flags a badly sized head, and it is far better to find that here than after deployment.

Serving the model

TensorFlow Serving runs as a container and exposes two interfaces.

docker run -p 8501:8501 \
  --mount type=bind,source=$(pwd)/models/served,target=/models/classifier \
  -e MODEL_NAME=classifier \
  tensorflow/serving

The mounted directory is the parent of the versions, not the version itself. Serving looks there for numeric subdirectories and loads the highest one.

Queries then go over HTTP:

curl -X POST http://localhost:8501/v1/models/classifier:predict \
  -d '{"instances": [[[[12, 34, 56], [78, 90, 12]]]]}'

The REST interface on port 8501 is convenient and readable. The gRPC interface on port 8500 is markedly faster, because it avoids encoding tensors as JSON — costly as soon as inputs get large. For a high-traffic image service, gRPC is not a refinement but a requirement.

Versioning is free, so use it

The directory structure is all it takes to manage versions:

models/served/
  1/    saved_model.pb  variables/
  2/    saved_model.pb  variables/

Dropping in a 2 directory is enough: Serving detects it, loads it, shifts traffic to it and unloads the previous one, with no downtime. A rollback means removing the offending directory.

A serving policy takes this further, keeping two versions active at once so you can compare their responses on real traffic before switching. That is the foundation of progressive rollout, covered in the production module of course 20.

Verify after deployment, not only before

One final check avoids unpleasant surprises:

import numpy as np, requests

batch = test_images[:8]
expected = served_model.predict(batch)

response = requests.post(
    "http://localhost:8501/v1/models/classifier:predict",
    json={"instances": batch.tolist()},
).json()
obtained = np.array(response["predictions"])

print("max difference:", np.abs(expected - obtained).max())

The difference should stay at the level of numerical noise, on the order of $10^{-5}$. Anything larger reveals a preprocessing divergence, a loaded version other than the one you expected, or a dtype conversion lost along the way. This test takes ten lines and catches nearly every serving error.

Key takeaways

• The .keras format is for resuming training, the SavedModel for production serving; the latter has no Python dependency and lives in a numbered directory.
• Preprocessing must be a model layer, otherwise it disappears on export and creates silent training-serving skew.
• saved_model_cli show verifies signature dtypes and shapes before deployment, in ten seconds.
• Serving watches the parent directory of versions and switches with no downtime; a test comparing local and served predictions catches most remaining errors.

Next module: the course recap and the 40-question final exam.