Module 1 — Tensors, variables and computation graphs

Course 07 explained what a network computes. This one explains what you build it with. And it starts with two objects that are easy to confuse, even though their roles are opposites.

A tensor never changes, a variable exists to change

A tf.Tensor is immutable. Any operation on a tensor produces a new one; it never modifies the original.

import tensorflow as tf

x = tf.constant([[1.0, 2.0], [3.0, 4.0]])
y = x * 2          # new tensor, x is untouched
print(x.shape)     # (2, 2)
print(x.dtype)     # <dtype: 'float32'>

A tf.Variable is mutable, and that is exactly what makes it the right container for a model's weights: training has to overwrite them thousands of times.

w = tf.Variable([[0.5, -0.2]])
w.assign([[0.7, -0.1]])       # replace the contents
w.assign_sub([[0.01, 0.01]])  # subtract in place

The rule to remember: data are tensors, learned parameters are variables. A naive w = w - 0.01 * gradient on a variable replaces it with a plain tensor and silently removes it from the model's trainable parameters. Training continues without error, but that weight never moves again.

Three attributes determine everything else

A tensor is described by its shape, its dtype and its device.

Attribute	What it carries	Typical mistake
shape	dimensions, None for a free size	forgetting the leading batch dimension
dtype	float32 by default, float16 in mixed precision	mixing int32 and float32 in one operation
device	processor or accelerator	implicit copies that dominate runtime

The leading dimension is almost always the batch. A batch of 32 images at 224 by 224 pixels with three channels has shape (32, 224, 224, 3). When Keras prints (None, 224, 224, 3), that None means "any batch size", which is what lets you train on batches of 32 and predict on a single image.

Type conversion does not happen on its own

TensorFlow refuses to add an int32 to a float32. Unlike NumPy, it does not silently promote types: it raises an exception. This is deliberate, because an implicit promotion on an accelerator is expensive and rarely noticed. Use tf.cast(x, tf.float32) explicitly.

Eager execution versus graph mode

By default, TensorFlow runs line by line, like any Python code. This is eager execution: you can print a tensor, set a breakpoint, inspect an intermediate value. Debugging comfort is complete, speed is ordinary.

Graph mode is the other regime. TensorFlow analyses the function, builds a representation of it as a graph of operations, then optimises it: fusing adjacent operations, eliminating dead computation, parallelising. The result is faster and, more importantly, exportable — a graph can be serialised and served in production with no Python interpreter, which is the subject of module 10.

Switching between the two comes down to one decorator:

@tf.function
def step(x, w):
    return tf.matmul(x, w)

What tf.function really does

Here is the point most tutorials skip, and it explains the majority of puzzling behaviour.

On the first call, tf.function executes the Python body to observe which TensorFlow operations are performed, and derives the graph from that. This is tracing. On subsequent calls the Python body is no longer executed: only the graph runs.

Three direct consequences:

@tf.function
def chatty(x):
    print("Python trace")          # once, at tracing time
    tf.print("graph execution")    # on every call
    return x * 2

chatty(tf.constant(1.0))  # prints both lines
chatty(tf.constant(2.0))  # prints only the second

First, a Python print appears only at tracing time, whereas tf.print becomes a graph operation and always runs. Second, a Python counter incremented in the body stays frozen at its tracing value. Third, a Python for loop over a fixed number of iterations is unrolled into the graph, which can produce an enormous graph; tf.while_loop stays a loop.

The second trap is retracing. TensorFlow traces one graph per input signature. Calling the function with shapes that change every time triggers a trace on every call, and the code becomes slower than eager execution.

@tf.function(input_signature=[tf.TensorSpec([None, 10], tf.float32)])
def stable(x):
    return tf.reduce_sum(x, axis=1)

Fixing a signature with None on the batch dimension forces a single graph, valid for every batch size.

GradientTape records so it can differentiate

Graph mode alone is not enough to compute derivatives: you need to know which operations were applied and in what order. That is the job of tf.GradientTape, which records operations involving watched variables.

w = tf.Variable(3.0)

with tf.GradientTape() as tape:
    loss = w ** 2

gradient = tape.gradient(loss, w)   # 2 * w = 6.0

The tape is consumed by the first call to gradient, to release memory. Two derivatives from one tape require persistent=True, and an explicit deletion afterwards. tf.Variable objects are watched automatically; a plain tensor needs an explicit tape.watch(x).

This is the machinery model.fit uses under the hood, and the one you take back by hand in module 4 when the standard loop is no longer enough.

Key takeaways

• A tensor is immutable, a variable is mutable: data are tensors, learned weights are variables, and overwriting a variable with a Python assignment removes it from the trainable parameters.
• Shape, dtype and device describe a tensor; the leading dimension is the batch, and None there means "any batch size".
• tf.function traces the Python body once to derive a graph, then stops executing it: Python side effects only happen at tracing time, and an unstable input signature causes expensive retracing.
• tf.GradientTape records operations to enable differentiation; it is consumed on the first call unless declared persistent.

Next module: the sequential API, which builds a complete model in a few lines without ever touching a tape.