A Verbose Guide to LeNet: The Pioneer of Convolutional Neural Networks

LeNet-1 (1989)

LeCun’s first published CNN, described in “Backpropagation Applied to Handwritten Zip Code Recognition” (Neural Computation, 1989). It had:

• Input: 16×16 images
• Two convolutional layers with 4 and 12 feature maps
• Two subsampling layers
• A fully connected output layer
• ~2,600 trainable parameters

It demonstrated that convolutional networks with weight sharing could learn from raw pixel data more efficiently than fully connected networks.

LeNet-4 (1993)

An intermediate architecture not described in detail in the 1998 paper but referenced. It introduced larger feature maps and demonstrated improved accuracy.

LeNet-5 (1998)

The definitive version and the architecture most people refer to when they say “LeNet.” Described in the 1998 IEEE paper, it operates on 32×32 grayscale images and has approximately 60,000 trainable parameters. This is the architecture examined in detail throughout this guide.

Layer C1: First Convolutional Layer

C1 applies 6 independent 5×5 convolution kernels to the input image. Each kernel slides across the 32×32 image with a stride of 1 pixel. Because there is no padding and the kernel is 5×5, the output spatial dimensions follow:

\[ \text{output\_size} = \frac{\text{input\_size} - \text{kernel\_size}}{\text{stride}} + 1 = \frac{32 - 5}{1} + 1 = 28 \]

The result is 6 feature maps of size 28×28, visualised below:

flowchart LR
    subgraph IN["Input Image"]
        direction TB
        I["32×32×1 (grayscale)"]
    end
    subgraph K["Learned Kernel"]
        direction TB
        KK["5×5 weights + 1 bias = 26 params"]
    end
    subgraph OP["Convolution (stride 1, no pad)"]
        direction TB
        CALC["(32−5)/1+1 = 28"]
    end
    subgraph OUT["Output Feature Map"]
        direction TB
        O["28×28 (one of 6 maps)"]
    end

    IN  -->|"slide"| OP
    K   -->|"dot product"| OP
    OP  --> OUT

    style IN  fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    style K   fill:#dcfce7,stroke:#16a34a,color:#14532d
    style OP  fill:#fef9c3,stroke:#ca8a04,color:#713f12
    style OUT fill:#f3e8ff,stroke:#9333ea,color:#3b0764

Filters at this stage tend to learn low-level features: oriented edges, blobs, corners, and gradients.

Layer S2: First Subsampling (Pooling) Layer

S2 downsamples each of the 6 feature maps by a factor of 2 in both spatial dimensions. The original LeNet-5 used a learnable average pooling: each 2×2 block is summed, multiplied by a trainable scalar, added to a trainable bias, then passed through a sigmoid.

Subsampling serves several purposes:

1. Reduces spatial resolution, decreasing computation in subsequent layers.
2. Increases the effective receptive field of deeper neurons.
3. Provides translation invariance: small input shifts produce the same pooled output.

Layer C3: Second Convolutional Layer

C3 applies 16 filters to the 14×14×6 feature maps from S2. Each filter produces a 10×10 output (14 − 5 + 1 = 10).

Layer S4: Second Subsampling Layer

Identical in structure to S2, S4 downsamples to 5×5×16. After this layer, the spatial dimensions are small enough that the next convolution will reduce them to 1×1.

Layer C5: Third Convolutional Layer

C5 is technically a convolutional layer, but because the input is exactly 5×5 and the kernel is 5×5, the output collapses to 1×1. It is mathematically identical to a fully connected layer operating on the flattened 400-dimensional vector.

Layer F6: Fully Connected Layer

F6 maps the 120-dimensional representation from C5 to an 84-dimensional vector. LeCun chose 84 because it corresponds to the number of bits in a 7×12 bitmap font for ASCII characters — allowing the network’s outputs to be interpreted as character codes.

The original activation was a scaled tanh:

\[ f(x) = A \cdot \tanh(S \cdot x), \quad A = 1.7159,\; S = \tfrac{2}{3} \]

These constants were chosen to keep output variance close to 1 and delay saturation.

Output Layer

The original LeNet-5 output used Euclidean RBF units. Each output unit \(y_i\) for class \(i\) computed:

\[ y_i = \sum_j (x_j - w_{ij})^2 \]

where \(w_{ij}\) is a fixed, hand-coded ASCII bitmap pattern for digit \(i\). The class with the smallest \(y_i\) was the prediction. Modern implementations replace this with standard softmax + cross-entropy loss.

Convolution

Convolution is the mathematical operation at the heart of CNNs. In the discrete 2D case:

\[ (I \ast K)[i, j] = \sum_{u=0}^{m-1} \sum_{v=0}^{n-1} I[i+u,\, j+v] \cdot K[u, v] \]

Intuition: A convolutional kernel is a small “template.” Sliding it across the image and computing the dot product at each position yields a high value wherever the image locally matches the template pattern.

Receptive Fields and Shared Weights

The receptive field of a neuron is the region of the input image that influences its output. The effective receptive field \(r_l\) at layer \(l\) grows recursively:

\[ r_0 = 1, \qquad r_l = r_{l-1} + (k_l - 1) \cdot \prod_{i=1}^{l-1} s_i \]

where \(k_l\) is the kernel size and \(s_i\) is the stride of layer \(i\). By the time we reach F6 in LeNet-5, the receptive field covers the entire 32×32 input.

Weight sharing means the same filter is applied at every spatial position. This is justified by the assumption of stationarity: useful feature detectors should work everywhere in the image, not just at one fixed location.

Subsampling / Pooling

Pooling reduces the spatial resolution of feature maps. The two most common types are:

Average pooling:

\[ y[i,j] = \frac{1}{|W|} \sum_{(u,v) \in W} x[2i+u,\; 2j+v] \]

Max pooling:

\[ y[i,j] = \max_{(u,v) \in W} x[2i+u,\; 2j+v] \]

Max pooling is preferred in modern networks because it is more tolerant of exact feature position, provides stronger non-linearity, and works better with ReLU activations. LeNet-5 originally used average pooling for probabilistic principled reasons.

Activation Functions

Scaled Hyperbolic Tangent (original LeNet):

\[ f(x) = 1.7159 \cdot \tanh\!\left(\tfrac{2}{3} x\right) \]

• Output range: \((-1.7159,\; +1.7159)\)
• Symmetric, smooth, differentiable everywhere
• Saturates for large \(|x|\) → can cause vanishing gradients

ReLU (modern replacement):

\[ f(x) = \max(0,\; x) \]

• Non-saturating for positive inputs
• Sparse activations
• Computationally cheap
• Alleviates vanishing gradients

Forward Pass

Let \(\mathbf{x}^{(0)} \in \mathbb{R}^{32 \times 32}\) be the input image. The forward pass computes:

C1:

\[ a_{k}^{C1}[i,j] = b_k^{C1} + \sum_{u=0}^{4} \sum_{v=0}^{4} W_k^{C1}[u,v] \cdot x^{(0)}[i+u, j+v], \quad k=1,\ldots,6 \]

\[ x_{k}^{C1}[i,j] = f\!\left(a_k^{C1}[i,j]\right) \]

S2:

\[ x_{k}^{S2}[i,j] = f\!\left(w_k^{S2} \cdot \sum_{u=0}^{1} \sum_{v=0}^{1} x_k^{C1}[2i+u, 2j+v] + b_k^{S2}\right) \]

C3 (full connectivity):

\[ a_{k}^{C3}[i,j] = b_k^{C3} + \sum_{m=1}^{6} \sum_{u=0}^{4} \sum_{v=0}^{4} W_{k,m}^{C3}[u,v] \cdot x_m^{S2}[i+u, j+v], \quad k=1,\ldots,16 \]

S4, C5, and F6 follow the same pattern. The final softmax output:

\[ \hat{y}_i = \frac{e^{z_i}}{\sum_{j=0}^{9} e^{z_j}} \]

Loss Function

With softmax output, the cross-entropy loss is:

\[ \mathcal{L} = -\sum_{i=0}^{9} y_i \log \hat{y}_i \]

For a single correct class \(c\), this simplifies to:

\[ \mathcal{L} = -\log \hat{y}_c = -z_c + \log \sum_j e^{z_j} \]

Parameter Count Summary

Gradient w.r.t. Bias

\[ \frac{\partial \mathcal{L}}{\partial b_k} = \sum_{i,j} \delta_k[i,j] \]

where \(\delta_k[i,j] = \partial \mathcal{L} / \partial a_k[i,j]\) is the upstream error signal.

Gradient w.r.t. Kernel Weights

\[ \frac{\partial \mathcal{L}}{\partial W_k[u,v]} = \sum_{i,j} \delta_k[i,j] \cdot x[i+u, j+v] \]

This is the cross-correlation of the input with the upstream gradient.

Gradient w.r.t. Input

\[ \frac{\partial \mathcal{L}}{\partial x[i,j]} = \sum_k \sum_{u,v} \delta_k[i-u, j-v] \cdot W_k[u,v] \]

This is a full convolution (the kernel is flipped), not cross-correlation.

Gradient Through Pooling

Average pooling — gradient is distributed equally:

\[ \frac{\partial \mathcal{L}}{\partial x[2i+u,\, 2j+v]} = \frac{1}{4} \cdot \delta[i,j] \]

Max pooling — gradient flows only to the winning position:

\[ \frac{\partial \mathcal{L}}{\partial x[2i+u,\, 2j+v]} = \delta[i,j] \cdot \mathbf{1}\!\left[(2i+u, 2j+v) = \operatorname{argmax}\right] \]

Optimization

The original LeNet-5 used SGD with a scheduled learning rate and a diagonal Hessian approximation for adaptive rates — a precursor to modern optimizers like Adam. Modern training uses:

• SGD with momentum (typically \(\mu = 0.9\))
• Adam (adaptive per-parameter learning rates)
• Weight decay / L2 regularization
• Learning rate schedules (step decay, cosine annealing)

Historical Performance on MNIST

A 0.95% error rate was extraordinary for its time and vastly exceeded all competing methods.

Replace tanh with ReLU

# Old
activation = "tanh"

# New
activation = "relu"

ReLU trains faster, avoids the vanishing gradient problem, and produces sparse activations.

Replace Average Pooling with Max Pooling

import torch.nn as nn

# Old
pool = nn.AvgPool2d(kernel_size=2, stride=2)

# New
pool = nn.MaxPool2d(kernel_size=2, stride=2)

Add Batch Normalization

import torch.nn as nn
import torch.nn.functional as F

conv1 = nn.Conv2d(1, 6, 5)
bn1   = nn.BatchNorm2d(6)

# In forward():
# x = F.relu(bn1(conv1(x)))

Add Dropout

import torch.nn as nn

dropout = nn.Dropout(p=0.5)

# In forward(), after F6:
# x = dropout(F.relu(fc1(x)))

Data Augmentation

from torchvision import transforms

transform_train = transforms.Compose([
    transforms.RandomAffine(degrees=15, translate=(0.1, 0.1), scale=(0.9, 1.1)),
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])