The same ink, the same shape, and the network is just as sure of the wrong answer. This is not one unlucky digit: on the 600 held-out digits, the network reads 91% as written and 24% once each is moved two pixels up, down, left or right. That is the failure the lecture ended on. Today you fix it.

from sklearn.neural_network import MLPClassifier

net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=500, random_state=0)
net.fit(X_train, y_train)                 # the 1,900 training digits, 256 pixels each
score(net, 'the lecture network')         # both tests, and the lower of the two
    

You should see accuracy 1.000 on the training digits, and 0.888 as written, 0.262 moved on the validation digits: it has learned its 1,900 digits, not to read moved ones. A hidden first cell loads the digits and builds the three sets (X_train, X_val, X_val_shift, and the hidden test digits), shift(X, dx, dy) and score(net).

from sklearn.neural_network import MLPClassifier

HIDDEN = (32,)      # hidden layer widths: try (128,), or two layers (64, 32)
ALPHA = 0.0001      # the penalty on large weights (weight decay): try 0.001 or 0.01
SHIFTS = []         # moves (dx, dy) to copy every training digit with, e.g.
                    # [(2, 0), (-2, 0), (0, 2), (0, -2), (1, 0), (-1, 0), (0, 1), (0, -1)]
UPDATES = 5000      # the most weight updates training may make, each on a batch of 200 digits

X_more = np.vstack([X_train] + [shift(X_train, dx, dy) for dx, dy in SHIFTS])
y_more = np.tile(y_train, len(SHIFTS) + 1)
passes = max(1, UPDATES * 200 // len(y_more))   # passes over the data that make that many updates:
                                                # 526 over the 1,900 digits, 58 over them and 8 copies
                                                # (it stops sooner once the loss stops improving)
net = MLPClassifier(hidden_layer_sizes=HIDDEN, alpha=ALPHA, max_iter=passes, random_state=0)
net.fit(X_more, y_more)
score(net, f'{HIDDEN}, alpha {ALPHA}, {len(SHIFTS)} shifted copies')
    

Which helps the moved accuracy more: a wider network, or the shifted copies? The copies are moved training digits, never validation or test digits. With copies a run takes up to a minute here.

class Forecaster(nn.Module):
    def __init__(self, n_in, hidden=16, kind='LSTM'):
        super().__init__()
        self.rnn = getattr(nn, kind)(n_in, hidden,
                                     batch_first=True)
        self.out = nn.Linear(hidden, 1)
    def forward(self, x):          # x: (days, 14, inputs)
        h, _ = self.rnn(x)         # the state after each day
        return self.out(h[:, -1]).squeeze(-1)
            

Data: ERA5 reanalysis for Beirut via the Open-Meteo archive API, CC-BY 4.0.

1. Check the loss before training. With ten classes it should be near \(\ln 10 \approx 2.30\). Much larger means the starting weights are too big.
2. Print the shapes after every layer for one batch. Most errors in a new network are shape errors.
3. Overfit ten examples. A healthy network reaches zero training loss on ten digits in a few hundred steps. If it cannot, the bug is in the code, not the data.
4. Sweep the learning rate by factors of ten before touching anything else.
5. Change one thing at a time, and run two seeds before believing a difference.
6. Plot training and held-out loss together. Both high: underfitting. Training falling while held-out climbs: overfitting.

from sklearn.neural_network import MLPClassifier

# The sets, as the hidden first cell builds them for everyone:
test = np.arange(len(y)) % 5 == 0          # every fifth digit: the 600 test digits
X_test, y_test = X[test], y[test]
X_shift = moved(X_test, 0)                 # the same 600 test digits, each moved two pixels
# (and from the other 2,400: X_train, 1,900 to train on; X_val and X_val_shift, 500 to choose with)

# The assistant: "Train on shifted examples so the network learns that position does not matter:"
X_more = np.vstack([X_train, X_shift])
y_more = np.concatenate([y_train, y_test])
net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=500, random_state=0)
net.fit(X_more, y_more)
score(net, "the assistant's network", show_test=True)   # this once, the test accuracy too
    

Validation says 0.898 and 0.650: better on moved digits than the baseline's 0.262, nothing strange. Test says 1.000 on the moved digits. The advice sounds right, and the code runs. Read where X_shift comes from: which line is the problem, what does 1.000 measure, and why did validation not see it?

Width alone bought nothing. Shifted copies did most of the work, for every architecture: they show the network what to ignore. The convolution did the rest, with a ninth of the weights of the wide network. And the moved 2 from the start? The lecture network reads a 6; with shifted copies, or as a CNN, a 2.

The LSTM beat the straight line by a tenth of a degree on maxima alone, and a straight line that reads the night minimum, the rain and the season does as well as it. Most of what the past two weeks can say about tomorrow, a linear model already hears. The rest of the error is weather that is not in the file: a front arriving from the sea, a dust storm from the east.

1. A fully connected network gives every pixel its own weight, so a two-pixel move turns 91% into 24%, with the same confidence.
2. An assumption the network needs can be built into the architecture (a convolution shares its kernel across positions) or shown in the data (shifted copies). On these digits, showing it did most of the work, and the two together did best with the fewest weights.
3. On Beirut's temperatures, the recurrent network bought a tenth of a degree over a straight line. A better model of the wrong inputs has a floor.
4. Debug in order: the starting loss, the shapes, ten examples, the learning rate, one change at a time.
5. A number too good to be true is a bug until shown otherwise; an assistant makes such bugs as easily as it makes working code.