Tensors and autograd

Welcome to Deep Learning with PyTorch. This course takes you from a single trainable weight to a working transformer, and then on to models that generate images. It assumes you can already write Python, are comfortable with NumPy arrays, and know the basics of machine learning: training and test sets, overfitting, a loss function and what a classifier does. If those ideas feel shaky, the Machine Learning Fundamentals course covers them first.

The fourteen lessons are in four parts. Foundations (lessons 1 to 3) covers tensors, automatic differentiation, neural networks and the practical craft of training them. Computer vision (lessons 4 to 6) builds convolutional networks, walks through the architectures that made them famous, and shows how to reuse a network trained by someone else. Sequences and language (lessons 7 to 11) moves from recurrent networks through LSTMs, word embeddings and attention to the transformer. Generative models (lessons 12 to 14) covers autoencoders, GANs and neural style transfer. Every example trains on a laptop CPU in about a minute, mostly on small datasets that ship with scikit-learn.

This first lesson covers the two things everything else rests on. Tensors are PyTorch's arrays: NumPy arrays that also know how to live on a GPU and how to remember the calculations that produced them. Autograd is the machinery that uses that memory to compute gradients automatically. Once you can compute the gradient of a loss with respect to a model's parameters, training any neural network, however large, is the same loop of "predict, measure the error, nudge the parameters downhill", and you will write that loop by the end of this lesson.

Setting up

You need Python 3.11 or newer and five libraries. Install them in a virtual environment with:

pip install torch torchvision numpy matplotlib scikit-learn

On Linux the default torch wheel from PyPI includes CUDA support and is large (several gigabytes). If you only want the CPU build, which is all this course needs, use pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu and install the other three separately. On macOS and Windows the default wheels are fine. torchvision is used from lesson 5 onwards for image utilities and pretrained models; lessons 6 and 14 download pretrained weights the first time they run, and say so when they do. Everything else in the course uses data generated in code or loaded from scikit-learn, which needs no network access.

Check the installation:

import sys
import torch, numpy as np, sklearn
print(sys.version.split()[0])
print(torch.__version__, np.__version__, sklearn.__version__)
print("CUDA available:", torch.cuda.is_available())

3.12.12
2.14.1 2.5.4 1.9.1
CUDA available: False

The outputs in this course came from PyTorch 2.14 on a CPU. Every example sets a seed with torch.manual_seed, so the same versions give the same numbers; on other versions or hardware the last decimals of a loss can drift, but the story each output tells will not. The examples use print throughout, so they behave the same as scripts or in a notebook.

The intuition

A neural network is a function with a very large number of adjustable knobs, called parameters or weights. Training means turning the knobs until the outputs match the examples. With millions of knobs you cannot try each one in turn; for every knob at once, you want to know "if I nudge this one slightly upwards, how much does the error change?" That number is the partial derivative of the error with respect to the knob, and the collection of all of them is the gradient.

The gradient points uphill. Take a small step in the opposite direction and the error goes down. Repeat thousands of times and the knobs settle somewhere useful. That is gradient descent, and it is how every model in this course learns.

PyTorch works out those derivatives for you. As you compute with tensors, PyTorch quietly records each operation in a graph. When you ask for gradients, it walks that graph backwards, applying the chain rule at each step. You write only the forward calculation; the backward one comes free.

How it works

Tensors

A tensor is an n-dimensional array with a fixed data type. A single number is a 0-dimensional tensor, a vector is 1-dimensional, a table is 2-dimensional, and a batch of colour images is 4-dimensional: (batch, channels, height, width).

import numpy as np
import torch

torch.manual_seed(0)

a = torch.tensor([[1.0, 2.0, 3.0],
                  [4.0, 5.0, 6.0]])
print(a)
print("shape:", a.shape, "| dtype:", a.dtype, "| device:", a.device)

b = torch.zeros(2, 3)
c = torch.randn(2, 3)          # standard normal, reproducible with the seed
d = torch.arange(6)            # integers 0..5, dtype int64
print(c)
print(d, d.dtype)

# NumPy round trip
arr = np.array([0.5, 1.5, 2.5])
t = torch.from_numpy(arr)      # shares memory with arr, dtype float64
print(t, t.dtype)
t32 = t.float()                # copy as float32, the usual dtype for models
print(t32.dtype, t32.numpy())

tensor([[1., 2., 3.],
        [4., 5., 6.]])
shape: torch.Size([2, 3]) | dtype: torch.float32 | device: cpu
tensor([[ 1.5410, -0.2934, -2.1788],
        [ 0.5684, -1.0845, -1.3986]])
tensor([0, 1, 2, 3, 4, 5]) torch.int64
tensor([0.5000, 1.5000, 2.5000], dtype=torch.float64) torch.float64
torch.float32 [0.5 1.5 2.5]

Three attributes describe every tensor:

• shape: the size of each dimension. Most deep learning bugs are shape bugs, so print shapes often.
• dtype: the element type. Model weights are float32, but NumPy defaults to float64, so cast converted arrays with .float(). Class labels are int64 (torch.long).
• device: where the data lives, cpu or a GPU such as cuda:0 (mps on Apple silicon). All inputs to an operation must share a device; the usual pattern is device = "cuda" if torch.cuda.is_available() else "cpu", then .to(device) on the model and data. This course stays on the CPU.

Operations, broadcasting and reshaping

Tensor operations look like NumPy, with dim= where NumPy says axis=:

import torch
torch.manual_seed(0)

x = torch.arange(12, dtype=torch.float32).reshape(3, 4)
print(x)
print("sum of each column:", x.sum(dim=0))
print("mean of each row:  ", x.mean(dim=1))

# Elementwise arithmetic and broadcasting: (3, 4) minus (4,) works row by row
col_means = x.mean(dim=0)
centred = x - col_means
print(centred)

# Matrix multiplication: (3, 4) @ (4, 2) -> (3, 2)
w = torch.ones(4, 2)
print((x @ w).shape)

# Reshaping and adding a batch dimension
img = torch.rand(8, 8)
batch = img.unsqueeze(0).unsqueeze(0)   # (1, 1, 8, 8): batch, channel, height, width
print(batch.shape, batch.squeeze().shape)
print(x.view(-1).shape, x.T.shape)

tensor([[ 0.,  1.,  2.,  3.],
        [ 4.,  5.,  6.,  7.],
        [ 8.,  9., 10., 11.]])
sum of each column: tensor([12., 15., 18., 21.])
mean of each row:   tensor([1.5000, 5.5000, 9.5000])
tensor([[-4., -4., -4., -4.],
        [ 0.,  0.,  0.,  0.],
        [ 4.,  4.,  4.,  4.]])
torch.Size([3, 2])
torch.Size([1, 1, 8, 8]) torch.Size([8, 8])
torch.Size([12]) torch.Size([4, 3])

Broadcasting follows NumPy's rules: dimensions are compared from the right, and a dimension of size 1 (or a missing one) is stretched to match. That is why subtracting a (4,) vector of column means from a (3, 4) matrix centres every column. @ is matrix multiplication, the operation at the heart of every dense layer. unsqueeze and squeeze add and remove size-1 dimensions; you will use them constantly to add the batch and channel dimensions that layers expect. view(-1) flattens, with -1 meaning "whatever size makes the total work".

Autograd: gradients for free

Set requires_grad=True on a tensor and PyTorch records every operation that uses it. Calling .backward() on a scalar result computes the derivative of that scalar with respect to every recorded input and stores it in the input's .grad attribute.

import torch

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x          # y = x^2 + 3x
y.backward()                # compute dy/dx and store it in x.grad
print("y =", y.item(), "| dy/dx at x=2:", x.grad.item())

# Gradients with respect to several inputs at once
w = torch.tensor([1.0, -2.0, 0.5], requires_grad=True)
inp = torch.tensor([3.0, 1.0, 4.0])
out = torch.sigmoid((w * inp).sum())
out.backward()
print("out =", round(out.item(), 4))
print("d out / d w =", w.grad)
print("graph node that made out:", out.grad_fn.name())

y = 10.0 | dy/dx at x=2: 7.0
out = 0.9526
d out / d w = tensor([0.1355, 0.0452, 0.1807])
graph node that made out: SigmoidBackward0

Check the first result by hand: the derivative of x² + 3x is 2x + 3, which is 7 at x = 2. The second example is a single artificial neuron, which lesson 2 builds on. Its gradient follows from the chain rule:

z   = w · inp = 1·3 + (−2)·1 + 0.5·4 = 3
out = σ(z) = 0.9526
d out / d w = σ(z) · (1 − σ(z)) · inp = 0.0452 · [3, 1, 4] = [0.1355, 0.0452, 0.1807]

PyTorch reached the same numbers without being told the derivative of the sigmoid. Each operation in the forward pass leaves a node in the computation graph (here SigmoidBackward0, linked to a SumBackward0, linked to a MulBackward0), and each node knows its own local derivative. backward() multiplies those local derivatives together from the output back to the inputs. This is reverse-mode automatic differentiation, and backpropagation in neural networks is exactly this applied to a large graph.

The graph is rebuilt on every forward pass, which is why PyTorch is called "define by run": ordinary Python if statements and loops inside a model just work, because the graph records whatever actually happened.

Worked example: fitting a line by gradient descent

Now put the pieces together. We generate 100 points from y = 3x − 2 plus noise and ask gradient descent to recover the slope and intercept, starting from zero. The loss is the mean squared error.

import torch
torch.manual_seed(42)

# Synthetic data: y = 3x - 2 plus noise
X = torch.rand(100, 1) * 4 - 2          # 100 points in [-2, 2]
y = 3 * X - 2 + 0.3 * torch.randn(100, 1)

w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
lr = 0.1

for step in range(101):
    pred = X * w + b
    loss = ((pred - y) ** 2).mean()      # mean squared error
    loss.backward()                      # fills w.grad and b.grad
    with torch.no_grad():                # update without recording history
        w -= lr * w.grad
        b -= lr * b.grad
    w.grad.zero_()                       # gradients accumulate unless cleared
    b.grad.zero_()
    if step % 20 == 0:
        print(f"step {step:3d}  loss {loss.item():.4f}  w {w.item():.3f}  b {b.item():.3f}")

step   0  loss 15.4288  w 0.792  b -0.362
step  20  loss 0.0606  w 2.959  b -1.977
step  40  loss 0.0597  w 2.965  b -2.000
step  60  loss 0.0597  w 2.965  b -2.000
step  80  loss 0.0597  w 2.965  b -2.000
step 100  loss 0.0597  w 2.965  b -2.000

Reading the output:

• Step 0 starts with w = b = 0, so the loss is large (15.4). One update already moves the slope to 0.79.
• By step 20 the parameters are close to the true values and the loss has dropped by a factor of more than 250.
• From step 40 nothing changes. The loss settles at 0.0597, not 0: the noise we added has a standard deviation of 0.3, so its variance is 0.09, and no line can explain noise. A remaining loss near the noise level is the sign of a good fit, not a failure. The slope of 2.965 rather than exactly 3 is also the noise talking; with 100 noisy points, that is the best line for this particular sample.

Two lines in that loop deserve attention. torch.no_grad() tells PyTorch not to record the update itself in the graph; the update is bookkeeping, not part of the model. And w.grad.zero_() is essential, because PyTorch adds new gradients to whatever is already in .grad:

import torch
x = torch.tensor(3.0, requires_grad=True)
for i in range(3):
    y = x ** 2
    y.backward()
    print("call", i + 1, "x.grad =", x.grad.item())

call 1 x.grad = 6.0
call 2 x.grad = 12.0
call 3 x.grad = 18.0

The true derivative is 6 every time; without zeroing, it piles up. Accumulation is deliberate (it lets you sum gradients over several small batches), but forgetting to clear it is one of the most common PyTorch bugs.

The same model, the PyTorch way

Writing parameters and updates by hand does not scale. PyTorch packages them: nn.Linear holds a weight and bias, nn.MSELoss computes the loss, and an optimiser from torch.optim performs the update and the zeroing.

import torch
from torch import nn
torch.manual_seed(42)

X = torch.rand(100, 1) * 4 - 2
y = 3 * X - 2 + 0.3 * torch.randn(100, 1)

model = nn.Linear(in_features=1, out_features=1)
loss_fn = nn.MSELoss()
optimiser = torch.optim.SGD(model.parameters(), lr=0.1)

for step in range(101):
    optimiser.zero_grad()            # 1. clear old gradients
    pred = model(X)                  # 2. forward pass
    loss = loss_fn(pred, y)          # 3. loss
    loss.backward()                  # 4. backward pass
    optimiser.step()                 # 5. update parameters
    if step % 25 == 0:
        print(f"step {step:3d}  loss {loss.item():.4f}")

print("learned weight:", round(model.weight.item(), 3), "bias:", round(model.bias.item(), 3))
for name, p in model.named_parameters():
    print(name, tuple(p.shape))

step   0  loss 13.4355
step  25  loss 0.0598
step  50  loss 0.0597
step  75  loss 0.0597
step 100  loss 0.0597
learned weight: 2.965 bias: -2.0
weight (1, 1)
bias (1,)

It reaches the same line. The starting loss differs (13.4 rather than 15.4) because nn.Linear initialises its weight and bias with small random values instead of zeros. The five numbered lines in the loop are the training loop, and you will write them in almost every lesson of this course. Lesson 2 swaps the single linear layer for a multilayer network, and lesson 3 swaps plain SGD for better optimisers; the loop itself stays the same.

Reference: tensor and autograd essentials

Common pitfalls

• Forgetting zero_grad(). Gradients accumulate across backward() calls, as shown above. Training then behaves erratically or diverges. Call optimiser.zero_grad() once per step.
• Mixing dtypes. A float64 tensor from NumPy multiplied by float32 model weights raises RuntimeError: mat1 and mat2 must have the same dtype. Convert inputs with .float() and labels with .long().
• Calling .numpy() on a tracked tensor. NumPy cannot hold the graph, so PyTorch refuses. Use t.detach().numpy().
• Shape mismatches that broadcast silently. If pred has shape (100, 1) and y has shape (100,), pred - y broadcasts to (100, 100) and the loss is computed over 10,000 meaningless pairs without an error. PyTorch's loss functions warn about this; hand-written losses do not. Check shapes before computing a loss.
• Accumulating graphs in a log. Appending loss rather than loss.item() to a list keeps every computation graph alive and memory grows each step. Log plain numbers.

Key takeaways

• Tensors are NumPy-like arrays with a shape, a dtype (usually float32 for data and weights, int64 for class labels) and a device.
• Setting requires_grad=True makes PyTorch record a computation graph; backward() walks it in reverse with the chain rule and fills each tensor's .grad.
• Gradient descent repeats: forward pass, loss, backward pass, update, zero the gradients.
• Layers, loss functions and optimisers package that loop into five lines; gradients accumulate by design, so clear them every step.

Try it yourself

1. In the hand-written gradient descent, change the learning rate to 0.01 and then to 1.1. How many steps does the first need to reach a loss of 0.06, and what happens to the loss with the second? Explain the second result in terms of stepping downhill.
2. Fit a quadratic y = 0.5x² − x + 1 (plus noise) by gradient descent with three parameters a, b, c and the prediction a * X**2 + b * X + c. Check that the learned values come close to 0.5, −1 and 1.
3. Use autograd to compute the derivative of torch.sin(x) * torch.exp(-x) at 50 points between 0 and 5 (create x = torch.linspace(0, 5, 50, requires_grad=True), sum the output, then call backward()). Compare x.grad with the analytical derivative cos(x)·e^(−x) − sin(x)·e^(−x) and print the largest absolute difference.