Vision Transformers (ViT): A Simple Guide

Image Patch Embedding

1. Patch Division: An input image (typically 224×224 pixels) is divided into fixed-size patches (commonly 16×16 pixels), resulting in a sequence of patches
2. Linear Projection: Each patch is flattened into a vector and linearly projected to create patch embeddings
3. Position Encoding: Since transformers don’t inherently understand spatial relationships, positional encodings are added to maintain spatial information
4. Classification Token: A special learnable [CLS] token is prepended to the sequence, similar to BERT’s approach

flowchart LR
    A[Input Image 224×224] --> B[Divide into 16×16 patches]
    B --> C[196 patches]
    C --> D[Flatten each patch]
    D --> E[Linear projection]
    E --> F[Add positional encoding]
    F --> G[Prepend CLS token]
    G --> H[Sequence ready for transformer]

Mathematical Formulation

For an image of size \(H \times W \times C\) divided into patches of size \(P \times P\):

• Number of patches: \(N = \frac{H \times W}{P^2}\)
• Each patch becomes a vector of size \(P^2 \times C\)
• After linear projection: embedding dimension \(D\)

Patch Embedding Layer

The patch embedding layer converts image patches into token embeddings that the transformer can process. This involves:

• Reshaping patches into vectors
• Linear transformation to desired embedding dimension
• Adding positional encodings

Transformer Encoder

The core of ViT consists of standard transformer encoder blocks, each containing:

• Multi-Head Self-Attention (MSA): Allows patches to attend to all other patches
• Layer Normalization: Applied before both attention and MLP layers
• Multi-Layer Perceptron (MLP): Two-layer feedforward network with GELU activation
• Residual Connections: Skip connections around both attention and MLP blocks

graph LR
    A[Input Patches] --> B[Patch Embedding]
    B --> C[Add Position Encoding]
    C --> D[Add CLS Token]
    D --> E[Transformer Encoder Block 1]
    E --> F[Transformer Encoder Block 2]
    F --> G[...]
    G --> H[Transformer Encoder Block N]
    H --> I[Extract CLS Token]
    I --> J[Classification Head]
    J --> K[Output Predictions]

Classification Head

The final component extracts the [CLS] token’s representation and passes it through:

• Layer normalization
• Linear classifier to produce class predictions

Attention Maps

• Each patch can attend to every other patch in the image
• Attention weights reveal which parts of the image are most relevant for classification
• This enables modeling of long-range spatial dependencies

Global Context

The ability to model global context from the first layer is a key advantage of ViTs over traditional CNNs.

Data Requirements

Vision Transformers typically require large amounts of training data to perform well:

• Pre-training: Often trained on large datasets like ImageNet-21k or JFT-300M
• Fine-tuning: Then adapted to specific tasks with smaller datasets
• Data Efficiency: ViTs can be less data-efficient than CNNs when training from scratch

Optimization Challenges

• Initialization: Careful weight initialization is crucial
• Learning Rate: Often requires different learning rates for different components
• Regularization: Techniques like dropout and weight decay are important
• Warmup: Learning rate warmup is commonly used

ViT Variants

Architectural Improvements

• Hierarchical Processing: Multi-scale feature extraction
• Local Attention: Restricting attention to local neighborhoods
• Hybrid Models: Combining CNN features with transformer processing

Strengths

Performance Benefits

• State-of-the-art results on image classification
• Strong performance on object detection and segmentation when adapted
• Excellent transfer learning capabilities across domains

Current Limitations

Ongoing Research Areas

• Improving data efficiency
• Reducing computational requirements
• Better integration of spatial inductive biases
• Hybrid CNN-Transformer architectures

Computer Vision Tasks

mindmap
  root((ViT Applications))
    Object Detection
      DETR
      Deformable DETR
    Segmentation
      SETR
      SegFormer
    Generation
      VQGAN
      DALL-E 2
    Video Analysis
      TimeSformer
      Video ViT

Multimodal Applications

• Vision-Language Models: CLIP and similar models combining vision and text
• Visual Question Answering: Integrating visual and textual understanding
• Image Captioning: Generating descriptions from visual content

Model Selection

Choose ViT variants based on:

Training Strategy

• Use pre-trained models when possible
• Apply appropriate data augmentation
• Consider knowledge distillation for smaller models
• Monitor for overfitting, especially on smaller datasets

Optimization Tips

# Example training configuration
training_config = {
    "mixed_precision": True,
    "gradient_checkpointing": True,
    "weight_decay": 0.05,
    "learning_rate": 1e-3,
    "warmup_epochs": 5,
    "batch_size": 512
}

Research Trends

Emerging Applications

• Real-time vision applications
• Mobile and edge deployment
• Scientific imaging and medical applications
• Autonomous systems and robotics