AlexNet: A Comprehensive Guide

The ImageNet Dataset

To appreciate AlexNet’s significance, we must first understand the challenge it was designed to tackle. ImageNet is a massive visual database organized according to the WordNet hierarchy. For the ILSVRC competition, the dataset contained:

• ~1.2 million training images
• 50,000 validation images
• 150,000 test images
• 1,000 object categories (classes)

This scale was unprecedented at the time. Prior CNN architectures (like LeNet-5, introduced in 1998 for digit recognition) were trained on small datasets with grayscale images of a single domain. The sheer diversity and volume of ImageNet posed a completely different engineering and statistical challenge.

The State of Deep Learning Before 2012

Neural networks had fallen somewhat out of favor in the 2000s. Despite theoretical appeal, they were difficult to train at scale due to:

• Vanishing gradients: Deep networks were notoriously hard to train because gradients diminished as they backpropagated through many layers.
• Computational constraints: Training large networks on CPUs was prohibitively slow.
• Overfitting: With millions of parameters and limited regularization techniques, large models quickly overfit to small datasets.

Researchers like Yann LeCun had demonstrated the power of CNNs for constrained domains (handwritten digits) [@lecun1998gradient], but scaling to general object recognition remained elusive. Geoffrey Hinton’s group had been steadily working on deep network training through the 2000s (deep belief networks, restricted Boltzmann machines), laying the groundwork for what was to come.

The GPU Revolution

The critical enabling factor for AlexNet was the availability of fast, programmable GPUs — specifically NVIDIA’s CUDA platform (introduced in 2006–2007), which allowed general-purpose computation on graphics cards. By 2012, a pair of NVIDIA GTX 580 GPUs with 3 GB of VRAM each gave the Toronto team enough raw computational power to train a massive network in a tractable amount of time (about 5–6 days). This hardware innovation made AlexNet possible.

Input

• Size: 227×227×3 (height × width × RGB channels)
• Images are preprocessed by subtracting the per-channel mean computed over the training set. This zero-centers the data, which helps with training stability.
• During training, 227×227 patches are randomly cropped from 256×256 images (data augmentation — discussed in detail in Section 1.5.6).

Layer 1 — Convolutional Layer (Conv1)

Operation: Convolution → ReLU → Local Response Normalization → Max Pooling

• Filters: 96 kernels of size 11×11×3, applied with stride 4
• Output before pooling: (227 - 11) / 4 + 1 = 55×55×96
• LRN: Applied across channels (described in Section 1.5.3)
• Max Pooling: 3×3 kernel, stride 2 → output 27×27×96
• Parameters: 96 × (11×11×3 + 1 bias) = 96 × 364 = 34,944

The large 11×11 kernels in the first layer capture low-level features such as edges, colors, and basic textures at multiple orientations. The aggressive stride of 4 dramatically reduces spatial dimensions early, keeping computation tractable. The 96 filters learn a diverse set of Gabor-like edge detectors and color blobs — visualizations of these learned filters were famously included in the original paper and became iconic images in the deep learning literature.

Layer 2 — Convolutional Layer (Conv2)

Operation: Convolution → ReLU → Local Response Normalization → Max Pooling

• Filters: 256 kernels of size 5×5×48 (per GPU, since the 96 channels are split across 2 GPUs), effectively 5×5×96 when combined
• Stride: 1, Padding: 2 (same padding)
• Output before pooling: 27×27×256
• LRN: Applied
• Max Pooling: 3×3 kernel, stride 2 → output 13×13×256
• Parameters: 256 × (5×5×96 + 1) = 256 × 2,401 = 614,656

The smaller 5×5 kernels in Conv2 build on the edge detectors from Conv1, combining them into more complex texture and shape detectors. The large increase in filter count (from 96 to 256) allows the network to represent a richer vocabulary of intermediate features. This layer captures corners, curves, and simple textures.

Layer 3 — Convolutional Layer (Conv3)

Operation: Convolution → ReLU

• Filters: 384 kernels of size 3×3×256
• Stride: 1, Padding: 1 (same padding)
• Output: 13×13×384
• No pooling, no LRN
• Parameters: 384 × (3×3×256 + 1) = 384 × 2,305 = 884,992

Conv3 is the first layer where both GPU streams interact — the full 256-channel input (from both halves of Conv2) feeds into all 384 filters. This cross-GPU communication was a deliberate design choice to allow the two GPU streams to mix learned representations. Conv3 captures higher-level textures and object parts.

Layer 4 — Convolutional Layer (Conv4)

Operation: Convolution → ReLU

• Filters: 384 kernels of size 3×3×192 (per GPU, each seeing half the 384 channels)
• Stride: 1, Padding: 1
• Output: 13×13×384
• No pooling, no LRN
• Parameters: 384 × (3×3×192 + 1) = 384 × 1,729 = 663,936

Conv4 continues refining high-level feature representations. The two GPU streams remain separate in this layer (unlike Conv3). Neurons in this layer have receptive fields covering large portions of the original input, allowing them to detect object parts and their spatial relationships.

Layer 5 — Convolutional Layer (Conv5)

Operation: Convolution → ReLU → Max Pooling

• Filters: 256 kernels of size 3×3×192 (per GPU)
• Stride: 1, Padding: 1
• Output before pooling: 13×13×256
• Max Pooling: 3×3 kernel, stride 2 → output 6×6×256
• Parameters: 256 × (3×3×192 + 1) = 256 × 1,729 = 442,624

Conv5 is the final convolutional layer. After the max pooling, the spatial map is reduced to 6×6, and the output is flattened to a 6×6×256 = 9,216-dimensional vector before entering the fully connected layers. By this stage, each neuron in the feature map has a receptive field spanning the majority of the original 227×227 image.

Layers 6–8 — Fully Connected Layers

FC6:

• Neurons: 4,096
• Operation: Linear → ReLU → Dropout (p=0.5)
• Input: 9,216-dimensional vector
• Parameters: 9,216 × 4,096 + 4,096 = 37,752,832

FC7:

• Neurons: 4,096
• Operation: Linear → ReLU → Dropout (p=0.5)
• Input: 4,096-dimensional vector
• Parameters: 4,096 × 4,096 + 4,096 = 16,781,312

FC8:

• Neurons: 1,000
• Operation: Linear → Softmax
• Input: 4,096-dimensional vector
• Parameters: 4,096 × 1,000 + 1,000 = 4,097,000

The fully connected layers serve as the “classifier head” on top of the convolutional feature extractor. FC6 and FC7 learn complex non-linear combinations of the convolutional features. The 4,096-dimensional activations of FC6/FC7 became widely used as general-purpose image feature vectors (a precursor to modern transfer learning). FC8 maps these features to the 1,000 class logits, which are then normalized by softmax to produce a probability distribution.

Output Layer

• Neurons: 1,000 (one per ImageNet class)
• Activation: Softmax

The softmax function converts raw logits \(z_i\) into probabilities:

\[ P(\text{class} = i) = \frac{e^{z_i}}{\sum_j e^{z_j}} \tag{1}\]

The predicted class is the one with the highest probability. During training, the cross-entropy loss between these probabilities and the one-hot encoded ground truth label is minimized.

ReLU Activation

The Problem with Saturating Activations

Prior to AlexNet, the most commonly used activation functions in neural networks were the sigmoid function:

\[ \sigma(x) = \frac{1}{1 + e^{-x}} \tag{2}\]

and the hyperbolic tangent (tanh):

\[ \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \tag{3}\]

Both of these are saturating functions — their gradients approach zero as their inputs become very large or very small. This causes the vanishing gradient problem: during backpropagation through many layers, the gradients become exponentially small, making it impossible to train deep networks effectively. Neurons in earlier layers receive almost no gradient signal and fail to learn.

The ReLU Solution

The Rectified Linear Unit (ReLU) activation function is defined as:

\[ f(x) = \max(0, x) \tag{4}\]

Its key properties:

• Non-saturating on the positive side: For \(x > 0\), the gradient is always 1, which flows back undiminished during backpropagation.
• Sparsity: For \(x \leq 0\), the output is exactly 0, effectively silencing that neuron. In practice, approximately half of all neurons are inactive at any given time, which introduces a useful form of sparsity.
• Computational efficiency: Computing \(\max(0, x)\) is trivially fast — no exponentials required.
• Fast convergence: Krizhevsky et al. demonstrated that networks trained with ReLUs reach a given training error rate 6× faster than equivalent networks with tanh units.

Dead ReLU Problem

A known drawback of ReLU is that neurons can “die” — if the inputs to a ReLU neuron are always negative, it will output 0 for every input and its gradient will always be 0, meaning it will never update. Proper weight initialization and careful learning rate selection mitigate this. Later variants like Leaky ReLU, PReLU, and ELU address this issue more directly.

Despite this limitation, ReLU was transformative and remains the default activation function in most modern deep learning architectures.

GPU Training

Training AlexNet on the ImageNet dataset took approximately 5–6 days using two NVIDIA GTX 580 GPUs, each with 3 GB of VRAM. The computational requirements were estimated at roughly 1.5 billion multiply-add operations per forward pass — completely infeasible on contemporary CPUs.

The authors used NVIDIA’s CUDA platform to implement highly optimized GPU kernels for convolution, pooling, and matrix multiplication. Because a single GPU at the time didn’t have enough memory to hold all the parameters and activations, the network was split across two GPUs (see Section 1.9 for details on the dual-GPU split).

This work demonstrated conclusively that deep learning was not just a theoretical pursuit — it could be engineered efficiently at scale with commodity hardware. It catalyzed the entire field’s shift toward GPU-based training, spawning a massive ecosystem of deep learning frameworks (Theano, Caffe, MXNet, TensorFlow, PyTorch) optimized for GPU execution.

Local Response Normalization (LRN)

Local Response Normalization is a form of lateral inhibition inspired by biological neuroscience — the idea that highly activated neurons suppress their neighbors, creating competition among features.

For activity \(a^i_{x,y}\) of neuron \(i\) at position \((x, y)\), the normalized response \(b^i_{x,y}\) is:

\[ b^i_{x,y} = \frac{a^i_{x,y}}{\left(k + \alpha \sum_{j=\max(0,\, i-n/2)}^{\min(N-1,\, i+n/2)} \left(a^j_{x,y}\right)^2\right)^\beta} \tag{5}\]

Where:

• \(N\) is the total number of feature maps
• \(n\) is the number of adjacent kernel maps over which normalization occurs
• \(k\), \(\alpha\), \(\beta\), \(n\) are hyperparameters (set to \(k=2\), \(\alpha=10^{-4}\), \(\beta=0.75\), \(n=5\) in AlexNet)

LRN normalizes across adjacent feature maps at the same spatial location, effectively making features compete across channels. The authors reported that LRN reduced their top-1 error rate by 1.4% and top-5 error rate by 1.2% on the validation set.

Overlapping Max Pooling

Traditional pooling in CNNs used non-overlapping windows (i.e., the stride equaled the window size). AlexNet used overlapping max pooling with a pool size of 3×3 and stride 2, meaning adjacent pooling windows overlap by 1 pixel in each direction.

Max pooling selects the maximum activation within each window:

\[ y_{i,j} = \max_{(p,q) \in \text{window at } (i,j)} x_{p,q} \tag{6}\]

The overlapping scheme provides several advantages:

• Translation invariance: The network becomes slightly more robust to small translations of features.
• Better generalization: The authors reported that overlapping pooling reduced top-1 and top-5 error rates by 0.4% and 0.3% respectively compared to non-overlapping pooling (stride=2, size=2).
• Richer information flow: By overlapping, some information from each region is passed forward through multiple pooling windows, providing redundancy.

Dropout Regularization

The Overfitting Problem

With ~62 million parameters and “only” 1.2 million training images, overfitting was a severe risk. A model with this many parameters can easily memorize the training set rather than learning generalizable features.

What is Dropout?

Dropout [@srivastava2014dropout] is a regularization technique that, during each forward pass in training, randomly “drops” (sets to zero) each neuron’s activation with probability \(p\) (typically \(p=0.5\)). The neurons that are dropped do not contribute to the forward pass and do not receive gradient updates in the backward pass.

Mathematically, for a layer with activations \(\mathbf{h}\), the dropout mask \(\mathbf{m} \sim \text{Bernoulli}(1-p)\) gives:

\[ \mathbf{h}_{\text{dropped}} = \mathbf{h} \odot \mathbf{m} \tag{7}\]

During inference (test time), all neurons are active, but their outputs are scaled by \((1-p)\) to compensate for the fact that during training only \((1-p)\) fraction of neurons were active on average. (Equivalently, using “inverted dropout” — the standard modern approach — you scale by \(\frac{1}{1-p}\) during training and do no scaling at test time.)

Why Does Dropout Work?

Dropout can be understood in several complementary ways:

1. Ensemble approximation: Each forward pass uses a different random subset of neurons, effectively training an exponential number of different “thinned” networks. At test time, the full network approximates averaging over this ensemble.

2. Prevention of co-adaptation: Neurons cannot rely on the presence of specific other neurons. This forces each neuron to learn features that are independently useful, rather than complex co-dependent features that are highly specific to the training data.

3. Noise injection: Adding Bernoulli noise to activations acts as a regularizer that prevents overfitting.

In AlexNet, dropout with \(p=0.5\) was applied to the outputs of FC6 and FC7. The authors estimated it roughly doubled the training time to convergence (because of the noise introduced) but significantly reduced overfitting. They noted that without dropout, their model exhibited substantially worse generalization.

Data Augmentation

The second major technique used to combat overfitting was data augmentation — artificially increasing the effective size and diversity of the training dataset by applying label-preserving transformations to the images.

AlexNet used two forms of data augmentation:

1. Random Cropping and Horizontal Flipping

• Training images were resized to 256×256 pixels.
• During training, 227×227 patches were randomly extracted from random positions in the 256×256 image, along with their horizontal reflections. This gave \((256-227)^2 \times 2 = 841 \times 2 \approx 1{,}682\) unique patches per image.
• At test time, 5 fixed crops (four corners + center) plus their horizontal reflections (10 patches total) were extracted, and the softmax probabilities were averaged over all 10 predictions.

2. PCA Color Augmentation

PCA was performed on the set of RGB pixel values across the training set. For each training image, random multiples of the found principal components were added to each pixel:

\[ \Delta \mathbf{p} = [\mathbf{p}_1, \mathbf{p}_2, \mathbf{p}_3]\, [\alpha_1 \lambda_1,\; \alpha_2 \lambda_2,\; \alpha_3 \lambda_3]^\top \tag{8}\]

where \(\mathbf{p}_i\) and \(\lambda_i\) are the eigenvectors and eigenvalues of the 3×3 RGB covariance matrix, and \(\alpha_i \sim \mathcal{N}(0, 0.1)\) are random Gaussian scalings drawn once per training image.

This augmentation captures the property that object identity is approximately invariant to changes in illumination color and intensity. The authors reported it reduced top-1 error by over 1%.

Strengths

• End-to-end learning: Unlike classical pipelines, AlexNet learns features directly from raw pixels, requiring no hand-crafted feature engineering.
• Scalability: The architecture clearly benefits from more data and more computation — a property that subsequent years of research would confirm as a general principle.
• Transfer learning: Features learned by AlexNet generalize remarkably well to other visual tasks. Fine-tuning a pretrained AlexNet (or just using its FC6/FC7 activations as features) became a standard baseline for many vision tasks for years.
• Practical training innovations: ReLU, dropout, data augmentation, and GPU training are all practically important techniques that were packaged into a single working system.

Limitations

• Extremely large FC layers: The three fully connected layers consume ~95% of the parameters while contributing relatively little to representational power. This is computationally and memory-inefficient.
• Fixed input size: The FC layers require a fixed-size input, which means all images must be resized to 227×227. This is inflexible for tasks requiring variable-resolution inputs.
• Large first-layer kernels: The 11×11 kernel with stride 4 in Conv1 is very large and can miss fine-grained details at the first convolutional stage.
• Local Response Normalization: LRN adds complexity and was later found to provide minimal benefit.
• No skip connections: AlexNet’s sequential stack makes it susceptible to the vanishing gradient problem in deeper variants. Residual connections [@he2016deep] solve this.
• Relatively shallow: By modern standards, 8 layers is shallow. Networks today routinely have hundreds of layers.
• Dual-GPU complexity: The split architecture added engineering complexity for marginal benefit; modern hardware easily fits the entire network in VRAM.

Direct Successors

• ZFNet (2013) [@zeiler2014visualizing]: Matthew Zeiler and Rob Fergus made incremental improvements to AlexNet (smaller first-layer kernel, modified strides), winning ILSVRC-2013 with a top-5 error of ~11.7%.
• VGGNet (2014) [@simonyan2015very]: Simonyan and Zisserman replaced all large kernels with stacks of small 3×3 convolutions, showing that depth was the key to performance. Top-5 error: ~7.3%.
• GoogLeNet/Inception (2014): Szegedy et al. introduced Inception modules and global average pooling, massively reducing parameter count while improving accuracy. Top-5 error: ~6.7%.
• ResNet (2015) [@he2016deep]: He et al. introduced residual (skip) connections, enabling training of networks with 100s of layers. Surpassed human-level performance on ImageNet with ~3.6% top-5 error.

Broader Impact

• Transfer learning revolution: AlexNet popularized the idea of pretraining a CNN on ImageNet and fine-tuning it for downstream tasks. This paradigm — which evolved into the massive pretrained models of today (GPT, BERT, ViT) — is arguably AlexNet’s most lasting contribution.
• Deep learning hardware ecosystem: AlexNet’s success accelerated development of GPU hardware and software specifically for deep learning. NVIDIA’s revenue from data center GPUs grew from near-zero in 2012 to tens of billions of dollars annually by the early 2020s.
• Benchmark culture: The success of ILSVRC popularized the use of large-scale benchmarks to drive research progress. This benchmark-driven culture (for better or worse) shapes ML research to this day.
• Democratization: AlexNet demonstrated that groundbreaking results in AI could be achieved with commodity hardware (consumer GPUs), lowering the barrier to entry for researchers worldwide.
• Industry transformation: The dramatic demonstration of deep learning’s potential at ILSVRC-2012 triggered massive investment from tech companies, reshaping research labs at Google, Facebook, Microsoft, Baidu, and others within months.

Model Definition

import torch
import torch.nn as nn
from torchvision import transforms


class AlexNet(nn.Module):
    """
    AlexNet: Krizhevsky, Sutskever, Hinton (2012).

    Architecture:
        5 convolutional layers followed by 3 fully connected layers.
        Uses ReLU activations, overlapping max pooling, and dropout.

    Args:
        num_classes (int): Number of output classes. Default: 1000 (ImageNet).
        dropout (float): Dropout probability in FC layers. Default: 0.5.
    """

    def __init__(self, num_classes: int = 1000, dropout: float = 0.5) -> None:
        super(AlexNet, self).__init__()

        # Feature extractor (convolutional layers)
        self.features = nn.Sequential(
            # Conv1: 227x227x3 -> 55x55x96 -> (pool) -> 27x27x96
            nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=0),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            # Conv2: 27x27x96 -> 27x27x256 -> (pool) -> 13x13x256
            nn.Conv2d(96, 256, kernel_size=5, stride=1, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            # Conv3: 13x13x256 -> 13x13x384  (no pooling)
            nn.Conv2d(256, 384, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),

            # Conv4: 13x13x384 -> 13x13x384  (no pooling)
            nn.Conv2d(384, 384, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),

            # Conv5: 13x13x384 -> 13x13x256 -> (pool) -> 6x6x256
            nn.Conv2d(384, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )

        # Adaptive average pooling (allows flexible input resolution)
        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))

        # Classifier head (fully connected layers)
        self.classifier = nn.Sequential(
            nn.Dropout(p=dropout),
            nn.Linear(256 * 6 * 6, 4096),   # FC6
            nn.ReLU(inplace=True),
            nn.Dropout(p=dropout),
            nn.Linear(4096, 4096),            # FC7
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),     # FC8
        )

        self._initialize_weights()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

    def _initialize_weights(self) -> None:
        """Initialize weights following the original paper."""
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.normal_(m.weight, mean=0, std=0.01)
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.Linear):
                nn.init.normal_(m.weight, mean=0, std=0.01)
                nn.init.constant_(m.bias, 1)  # Positive bias for ReLU

    def get_feature_vector(
        self, x: torch.Tensor, layer: str = "fc7"
    ) -> torch.Tensor:
        """
        Extract feature vectors from FC6 or FC7 for transfer learning.

        Args:
            x: Input tensor of shape (batch, 3, H, W)
            layer: 'fc6' or 'fc7'
        Returns:
            Feature vector of shape (batch, 4096)
        """
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier[0](x)  # Dropout
        x = self.classifier[1](x)  # Linear (FC6)
        x = self.classifier[2](x)  # ReLU
        if layer == "fc6":
            return x
        x = self.classifier[3](x)  # Dropout
        x = self.classifier[4](x)  # Linear (FC7)
        x = self.classifier[5](x)  # ReLU
        return x

Example Usage

# Standard ImageNet preprocessing
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(227),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
])

model = AlexNet(num_classes=1000)
model.eval()

# Parameter counts
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params:,}")

# Forward pass
dummy_input = torch.randn(4, 3, 227, 227)
with torch.no_grad():
    output = model(dummy_input)
print(f"Input shape:  {dummy_input.shape}")
print(f"Output shape: {output.shape}")

# Transfer learning features
features = model.get_feature_vector(dummy_input, layer="fc7")
print(f"FC7 feature vector shape: {features.shape}")

Training Loop

import torch.optim as optim


def train_alexnet(model, train_loader, val_loader, num_epochs=90):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = model.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(
        model.parameters(),
        lr=0.01,
        momentum=0.9,
        weight_decay=5e-4,
    )
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(
        optimizer, mode="min", factor=0.1, patience=5, verbose=True
    )

    for epoch in range(num_epochs):
        model.train()
        running_loss = 0.0

        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        val_loss = validate(model, val_loader, criterion, device)
        scheduler.step(val_loss)
        print(
            f"Epoch [{epoch+1}/{num_epochs}] | "
            f"Train Loss: {running_loss/len(train_loader):.4f} | "
            f"Val Loss: {val_loss:.4f}"
        )


def validate(model, val_loader, criterion, device):
    model.eval()
    total_loss, correct_top1, correct_top5, total = 0.0, 0, 0, 0

    with torch.no_grad():
        for images, labels in val_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)
            total_loss += criterion(outputs, labels).item()

            _, predicted = outputs.max(1)
            correct_top1 += predicted.eq(labels).sum().item()

            _, top5_preds = outputs.topk(5, dim=1)
            correct_top5 += (
                top5_preds.eq(labels.view(-1, 1)).any(dim=1).sum().item()
            )
            total += labels.size(0)

    print(f"Top-1 Accuracy: {100.*correct_top1/total:.2f}%")
    print(f"Top-5 Accuracy: {100.*correct_top5/total:.2f}%")
    return total_loss / len(val_loader)