Introduction to Machine Learning
A deeper dive into the inner working of deep learning: the chain rule by hand, a network from scratch, the same network in PyTorch, and what to do when training goes wrong.
The mathematics is in the neural networks deck, Part 6. Today we run it: by hand, in numpy, in PyTorch, on the same 16 by 16 digits.
\( \mathbf{z}^{[l]} = \mathbf{W}^{[l]} \mathbf{v}^{[l-1]} + \mathbf{b}^{[l]}, \quad \mathbf{v}^{[l]} = \sigma(\mathbf{z}^{[l]}) \)
\( \boldsymbol{\delta}^{[L]} = \hat{\mathbf{y}} - \mathbf{y}, \qquad \boldsymbol{\delta}^{[l]} = \big(\mathbf{W}^{[l+1]\top} \boldsymbol{\delta}^{[l+1]}\big) \odot \sigma'(\mathbf{z}^{[l]}) \)
\( \partial \ell / \partial \mathbf{W}^{[l]} = \boldsymbol{\delta}^{[l]}\, \mathbf{v}^{[l-1]\top}, \qquad \partial \ell / \partial \mathbf{b}^{[l]} = \boldsymbol{\delta}^{[l]} \)
The claim was that this gives every gradient for the price of about one forward pass. Is that true, and does it matter?
Nudge one weight up and down by \(h = 10^{-5}\), recompute the loss, divide: \( \dfrac{\partial \mathcal{L}}{\partial w} \approx \dfrac{\mathcal{L}(w + h) - \mathcal{L}(w - h)}{2h} \). Two forward passes per weight.
One forward pass, one backward pass, every derivative.
Press Compute: both methods run here, in your browser, on the same weights and digits.
Part 2 of 5
Can you backpropagate through a graph with a pencil?
| Node | Forward | Backward: each input receives | So it acts as |
|---|---|---|---|
| add | \( z = x + y \) | \( \bar z \) and \( \bar z \) | a distributor: the same gradient to every input |
| multiply | \( z = x y \) | \( \bar z\, y \) and \( \bar z\, x \) | a swap: each input gets the other one |
| max, ReLU | \( z = \max(x, y) \) | \( \bar z \) to the larger, 0 to the other | a router: only the winner learns |
| matrix product | \( \mathbf{z} = \mathbf{W} \mathbf{v} \) | \( \bar{\mathbf{z}}\, \mathbf{v}^\top \) and \( \mathbf{W}^\top \bar{\mathbf{z}} \) | the multiply rule, with transposes that make the shapes fit |
Here \(\bar z = \partial L / \partial z\), the gradient arriving from above. The last row is the whole of backpropagation for a layer.
Part 3 of 5
Can you train a network with nothing but numpy?
Pick "all weights 0.5" on the last slide and train. The hidden units start identical, compute the same value, receive the same \(\boldsymbol{\delta}\), and take the same step. They stay identical forever: a network of many units that behaves like one.
This is why weights start random: randomness breaks the symmetry, so each unit can become a different feature.
import numpy as np, pandas as pd
d = pd.read_csv('mnist_16x16.csv')
X, y = d.drop(columns='label').values / 255, d.label.values
test = np.arange(len(y)) % 5 == 0
Y = np.eye(10)[y] # one-hot labels
rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, np.sqrt(2 / 256), (32, 256)), np.zeros(32)
W2, b2 = rng.normal(0, np.sqrt(2 / 32), (10, 32)), np.zeros(10)
for epoch in range(10):
for i in rng.permutation(np.where(~test)[0].reshape(-1, 32)): # batches of 32
z1 = X[i] @ W1.T + b1; v1 = np.maximum(0, z1) # forward, and store
s = v1 @ W2.T + b2; p = np.exp(s - s.max(1, keepdims=True)); p /= p.sum(1, keepdims=True)
d2 = (p - Y[i]) / len(i) # backward: delta at the output
d1 = (d2 @ W2) * (z1 > 0) # delta one layer down
W2 -= 0.5 * d2.T @ v1; b2 -= 0.5 * d2.sum(0)
W1 -= 0.5 * d1.T @ X[i]; b1 -= 0.5 * d1.sum(0)
p = np.maximum(0, X[test] @ W1.T + b1) @ W2.T + b2
print('epoch', epoch + 1, ' held-out accuracy', round(np.mean(p.argmax(1) == y[test]), 3))
You should see the held-out accuracy reach about 0.90 within a few epochs, bouncing by a point or two: the same network as the digits slides of the last deck, now with every derivative written by hand. Try a learning rate of 5 instead of 0.5.
import numpy as np, pandas as pd
d = pd.read_csv('mnist_16x16.csv')
X, y = d.drop(columns='label').values[:20] / 255, d.label.values[:20] # 20 digits are enough
rng = np.random.default_rng(1)
W1, W2 = rng.normal(0, 0.1, (8, 256)), rng.normal(0, 0.1, (10, 8))
def loss(W1, W2): # softmax cross-entropy, sigmoid hidden layer
v = 1 / (1 + np.exp(-X @ W1.T)); s = v @ W2.T
return np.mean(np.log(np.exp(s).sum(1)) - s[np.arange(20), y])
def backprop(W1, W2):
v = 1 / (1 + np.exp(-X @ W1.T)); s = v @ W2.T
p = np.exp(s) / np.exp(s).sum(1, keepdims=True); p[np.arange(20), y] -= 1
d2 = p / 20; d1 = (d2 @ W2) * v * (1 - v)
return d1.T @ X, d2.T @ v
g1, g2 = backprop(W1, W2)
h, worst = 1e-5, 0
for _ in range(10): # ten random weights of W1, checked numerically
i, j = rng.integers(8), rng.integers(256)
E = np.zeros_like(W1); E[i, j] = h
numeric = (loss(W1 + E, W2) - loss(W1 - E, W2)) / (2 * h)
worst = max(worst, abs(numeric - g1[i, j]) / max(abs(numeric) + abs(g1[i, j]), 1e-12))
print('largest relative error:', f'{worst:.1e}')
Correct code gives a relative error around \(10^{-8}\). Now break it: delete * v * (1 - v) in backprop and run again. The error jumps to about 0.6. Finite differences are far too slow to train with, and exactly right for checking.
Part 4 of 5
What does a library do for you, and what does it not?
import torch
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x[0]**2 + x[1]**2 # forward: builds the graph
y.backward() # backward: fills x.grad
print(x.grad) # tensor([4., 6.])
\(y = x_0^2 + x_1^2\), so \(\nabla y = (2x_0, 2x_1) = (4, 6)\) at \((2, 3)\). You never wrote the derivative.
Each tensor with requires_grad=True remembers the operation that made it. backward() is the backward sweep of the one-neuron graph, for any graph you build.
It is the micrograd of the Math and Python deck, for tensors, on a GPU.
model = SimpleNN() # 784, 128 hidden, 10
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)
train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
for epoch in range(20):
model.train()
for images, labels in train_loader:
optimizer.zero_grad() # 1. clear old gradients
loss = criterion(model(images), labels) # 2. forward
loss.backward() # 3. backward
optimizer.step() # 4. update
model.eval()
with torch.no_grad(): # no graph needed to evaluate
dev_accuracy = evaluate_model(dev_loader)
Forget step 1 and gradients add up across batches: PyTorch accumulates on purpose.
Train and eval modes matter once layers behave differently while training (dropout).
The dev set is checked every epoch; the test set once, at the end.
Regularization is one argument: optim.Adam(..., weight_decay=1e-3).
nn.Module, optimizers.All on intro2ml.com, on the schedule for today.
Part 5 of 5
What do you check when the loss will not go down?
Backward, each layer multiplies \(\boldsymbol{\delta}\) by \(\sigma'(\mathbf{z})\), and a sigmoid's derivative is at most \(1/4\). Eight layers: eight shrinkings. ReLU's derivative is 1 wherever the unit is on, which is a large part of why deep networks became trainable.
Forward stores every value; backward multiplies local derivatives from the loss down, reusing them. Hundreds of times faster than finite differences on the digits network, a gap that grows with every weight you add, and exactly as accurate.
Break symmetry with random weights, check gradients numerically, sweep the learning rate, prefer ReLU in deep networks, and watch both loss curves.