Y

Deep learning · Advanced

Convolutions, validation and early stopping

3 min read

Why it matters

Getting a network to fit training data is easy. Getting it to perform on data it has not seen is the job. AI/ML engineering roles expect you to read training curves, choose regularisation and stop training at the right moment, because these decisions decide whether a model survives production.

The concept

  • Convolutional layers slide small filters over an image and share weights across positions, so they learn local patterns (edges, strokes) with far fewer parameters than dense layers.

  • Three splits: train (fit weights), validation (choose epochs and settings), test (report once).

  • Overfitting shows as training loss still falling while validation loss rises.

  • Regularisation: dropout (randomly zero activations during training), weight decay (penalise large weights), data augmentation, smaller models.

  • Early stopping: keep the weights from the epoch with the best validation loss, and stop after a few epochs without improvement (the patience).

Worked example: a small CNN on digits

digits_cnn.py
import copy
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.manual_seed(0)
digits = load_digits()
X = torch.tensor(digits.images / 16.0, dtype=torch.float32).unsqueeze(1)   # (1797, 1, 8, 8)
y = torch.tensor(digits.target)
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.2, random_state=42, stratify=y_rest)
train = DataLoader(TensorDataset(X_train, y_train), batch_size=64, shuffle=True)

model = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, padding=1), nn.ReLU(),
    nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(),
    nn.MaxPool2d(2),                                  # 8x8 -> 4x4
    nn.Flatten(),
    nn.Dropout(0.3),
    nn.Linear(32 * 4 * 4, 10),
)
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4)
loss_fn = nn.CrossEntropyLoss()

def evaluate(X, y):
    model.eval()
    with torch.no_grad():
        logits = model(X)
        return loss_fn(logits, y).item(), (logits.argmax(1) == y).float().mean().item()

best_loss, best_state, patience, bad_epochs = float("inf"), None, 5, 0
for epoch in range(1, 101):
    model.train()
    for xb, yb in train:
        optimizer.zero_grad()
        loss_fn(model(xb), yb).backward()
        optimizer.step()
    val_loss, val_acc = evaluate(X_val, y_val)
    if val_loss < best_loss - 1e-4:
        best_loss, best_state, bad_epochs = val_loss, copy.deepcopy(model.state_dict()), 0
    else:
        bad_epochs += 1
    if bad_epochs == patience:
        print(f"early stop at epoch {epoch}")
        break

model.load_state_dict(best_state)
print("validation loss %.3f" % best_loss, "test accuracy %.3f" % evaluate(X_test, y_test)[1])
print("parameters", sum(p.numel() for p in model.parameters()))

On a laptop CPU, training stopped at epoch 34 with a test accuracy of about 0.98, using 9,930 parameters. The dense network in A neural network and its training loop in PyTorch used about 9,600 parameters and reached about 0.97. The convolutions use a similar budget far better, because they reuse the same filters everywhere in the image.

Exact numbers vary a little between machines and library versions; the pattern does not.

Common mistakes

  • Early stopping on the test set. Then the test score is no longer an honest estimate.

  • Not restoring the best weights. Stopping after patience runs out leaves you with worse weights than the best epoch.

  • Dropout at evaluation time. Forgetting model.eval() makes predictions noisy and worse.

  • Tuning many things at once. Change one setting, compare validation curves, then the next.

Interview angle

"Your validation loss rises after epoch 10 while training loss falls. What do you do?" Name overfitting, then list remedies in order of cost: early stopping, more data or augmentation, dropout or weight decay, a smaller model. Say how you would confirm each helped.

Practice task

  1. Remove dropout and weight decay. Plot training and validation loss per epoch and find where they diverge.

  2. Add data augmentation by shifting images one pixel (torch.roll) during training.

  3. Count parameters of the dense and convolutional models and compare accuracy per parameter.

Further reading