BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

The problem: VLP is getting too expensive

By 2022–2023, vision-language pre-training had converged on a fairly predictable recipe: take a large image-text paired dataset, build a big multimodal architecture (dual-encoder, fusion-encoder, encoder-decoder, or unified transformer), and train the whole thing end-to-end with some mix of contrastive, matching, and generative objectives. Models like CLIP, ALIGN, BEiT-3, SimVLM, and Flamingo pushed benchmarks steadily upward, but at a steep cost: Flamingo80B has 10.2B trainable parameters and requires enormous compute budgets to pre-train from scratch (or near-scratch) on billions of image-text pairs.

At the same time, two adjacent research communities had already produced excellent unimodal building blocks: the vision community had strong pre-trained image encoders (CLIP ViT-L/14, EVA-CLIP ViT-g/14), and the NLP community had increasingly capable LLMs (GPT-3, OPT, FlanT5) with strong zero-shot and instruction-following behavior. BLIP-2’s central bet is that VLP doesn’t need to reinvent these capabilities — it should bootstrap from them.

Why not just freeze everything and fine-tune a small adapter?

This had already been tried, and it doesn’t work well out of the box. The paper identifies the core obstacle as the modality gap: an LLM has never seen an image during its unimodal pre-training, so if you freeze it, there’s no learned pathway for visual information to enter its representation space. Prior approaches to this problem include:

• Frozen (Tsimpoukelli et al., 2021): fine-tunes an image encoder so that its output can be used directly as a “soft prompt” (a sequence of embeddings) fed to a frozen LLM, trained with an image-to-text generation loss.
• Flamingo (Alayrac et al., 2022): inserts new cross-attention layers inside the frozen LLM to let it attend to visual features, training these new layers (via a Perceiver Resampler) on billions of image-text pairs.

Both rely purely on a generative (image-to-text) loss to force alignment. BLIP-2’s authors argue — and empirically demonstrate (see the Experiments section, “Effect of Vision-Language Representation Learning”) — that this generative-loss-only approach is insufficient to bridge the modality gap efficiently; it needs enormous data and often still suffers catastrophic forgetting in the LLM (observed with OPT in their ablation).

BLIP-2’s proposal

BLIP-2 introduces a lightweight, trainable, two-sided bridge — the Q-Former — and a two-stage pre-training strategy: first teach the bridge to speak “vision,” then teach it to speak “LLM.” Only the Q-Former (188M parameters) and a small linear projection layer are ever trained; the image encoder and the LLM stay frozen throughout. This has three practical consequences, all validated experimentally:

1. Compute efficiency. No need to update billions of frozen parameters — pre-training the largest BLIP-2 variant (ViT-g + FlanT5-XXL) takes under 9 days total on a single 16×A100 (40GB) machine.
2. Modularity. Because the image encoder and LLM are swappable black boxes, BLIP-2 can “plug in” better vision or language backbones as they become available, and performance improves monotonically with backbone quality.
3. Inherited capabilities. Because the LLM is frozen and untouched, its pre-existing instruction-following and generative fluency transfer directly to the vision-language setting — enabling zero-shot instructed image-to-text generation without ever training on instruction data.

Q-Former architecture

The Q-Former (“Querying Transformer”) is the single trainable component that connects the frozen image encoder to the frozen LLM. Structurally, it consists of two transformer submodules that share the same self-attention layers:

• An image transformer that interacts with the frozen image encoder through cross-attention layers (inserted every other transformer block).
• A text transformer that can act as either a text encoder or a text decoder, depending on the training objective.

The input to the image transformer is a fixed set of 32 learnable query embeddings, each of dimension 768 (matching the Q-Former’s hidden size). These queries:

• attend to each other through self-attention,
• attend to the frozen image features through cross-attention,
• and (depending on a task-specific attention mask) can also attend to input text tokens through the shared self-attention layers.

Because there are only 32 queries of dimension 768 (a 32×768 output), compared to, e.g., 257×1024 raw patch features from a ViT-L/14, the Q-Former acts as a hard information bottleneck: it is architecturally forced to compress the image into the most textually-relevant information, discarding the rest. This bottleneck, combined with the training objectives below, is the crux of why BLIP-2 doesn’t need to update the underlying vision or language backbones.

The Q-Former is initialized from BERT-base (Devlin et al., 2019) pre-trained weights, except the cross-attention layers, which are randomly initialized. In total it has 188M parameters (queries included, since they are treated as learnable model parameters, not just inputs).

Stage 1 — Vision-language representation learning (bootstrapped from a frozen image encoder)

In this stage, the Q-Former sits directly on top of the frozen image encoder (no LLM involved yet) and is trained on image-text pairs with three jointly-optimized objectives, all sharing the same underlying architecture and parameters but differing in the self-attention mask applied between the queries and the text tokens. This design (queries/text sharing weights but interacting under different masks) is inherited from the original BLIP paper.

1. Image-Text Contrastive Learning (ITC). Aligns the image representation (the queries’ output, Z, a set of 32 embeddings) with the text representation (t, the [CLS] embedding from the text transformer) by maximizing agreement between paired image-text embeddings vs. in-batch negatives. Because Z contains 32 separate vectors, ITC computes the similarity between each query output and t, and takes the max as the overall image-text similarity. Uses a unimodal attention mask: queries and text cannot see each other, preventing information leakage that would make the contrastive task trivial. Because the image encoder is frozen, more samples fit per GPU, so BLIP-2 uses in-batch negatives rather than BLIP’s momentum queue.
2. Image-grounded Text Generation (ITG). Trains the Q-Former to generate a caption conditioned on the image, but crucially, the architecture prevents the frozen image encoder from talking to text tokens directly — any visual information used for text generation must first be extracted by the queries and passed to the text tokens via the shared self-attention layer. This directly forces the queries to capture everything the text generator needs. Uses a multimodal causal mask (à la UniLM): queries attend to each other (not text), while each text token attends to all queries and to previous text tokens. The [CLS] token is swapped for a [DEC] token to signal that decoding is starting.
3. Image-Text Matching (ITM). A binary classification task (matched / not matched) that learns fine-grained image-text alignment. Uses a bi-directional mask — queries and text can fully attend to each other — so the output query embeddings capture rich multimodal information. Each query’s output is passed through a 2-class linear classifier, and the logits are averaged across all 32 queries to get the final matching score. Hard negative mining (from the earlier ALBEF/BLIP work) is used to create informative negative pairs.

Running all three objectives jointly, with different masking patterns over the same shared weights, is analogous to a multi-task curriculum that squeezes the queries from three different angles: global similarity (ITC), generative sufficiency (ITG), and fine-grained matching (ITM).

Stage 2 — Vision-to-language generative learning (bootstrapped from a frozen LLM)

Once Stage 1 has trained the Q-Former to extract “language-informative” visual features, Stage 2 connects the Q-Former (with its frozen image encoder still attached) to a frozen LLM. A single fully-connected (FC) layer linearly projects the Q-Former’s output query embeddings Z into the same dimensionality as the LLM’s text embedding space. These projected embeddings are then prepended to the input text embeddings, functioning as soft visual prompt tokens — 32 extra “words” describing the image that the LLM conditions on before generating its output.

Two LLM families are handled, with two different losses:

• Decoder-only LLMs (OPT family): standard language-modeling loss — the frozen decoder generates text conditioned on the visual soft prompts.
• Encoder-decoder LLMs (FlanT5 family): prefix language-modeling loss — the text is split into a prefix and a suffix; the prefix is concatenated with the visual embeddings and fed to the encoder, while the suffix serves as the decoder’s generation target.

Because the Q-Former has already learned (in Stage 1) to extract the visual information most relevant to accompanying text, it substantially reduces the burden on the LLM to learn cross-modal alignment from scratch — which is exactly what mitigates catastrophic forgetting in the frozen LLM (this is empirically verified in the ablation below, where skipping Stage 1 causes a severe performance drop, especially for OPT).

Pre-training data and compute

• Data: the same 129M-image corpus as BLIP: COCO, Visual Genome, CC3M, CC12M, SBU, plus 115M images from LAION-400M. Synthetic captions are generated via BLIP’s CapFilt bootstrapping method (10 captions per web image from a BLIP-large captioner, re-ranked by CLIP ViT-L/14 similarity against the original web caption; top-2 kept, one randomly sampled per training step).
• Backbones explored: image encoders — CLIP ViT-L/14 and EVA-CLIP ViT-g/14 (last layer removed, second-to-last layer’s features used); LLMs — the unsupervised OPT family (decoder-only) and the instruction-tuned FlanT5 family (encoder-decoder).
• Schedule: 250k steps for Stage 1, 80k steps for Stage 2. Batch sizes range from 1520–2320 depending on backbone. Frozen ViTs and LLMs are cast to FP16 (BFloat16 for FlanT5) with no observed degradation vs. FP32.
• Optimizer: AdamW (β₁=0.9, β₂=0.98, weight decay 0.05), cosine LR decay, peak LR 1e-4, 2k-step linear warmup, min LR 5e-5 in Stage 2. Images at 224×224 with random resized crop + horizontal flip.
• Compute cost: the largest configuration (ViT-g + FlanT5-XXL) needs under 6 days for Stage 1 and under 3 days for Stage 2 on a single 16×A100(40GB) node — dramatically cheaper than end-to-end alternatives of comparable scale.

Headline comparison

Across zero-shot VQAv2, NoCaps captioning, and Flickr30K retrieval, BLIP-2 (188M trainable params) matches or beats much larger models: BLIP (583M), SimVLM (1.4B), BEiT-3 (1.9B), and Flamingo (10.2B trainable params). This is the paper’s central empirical claim — better performance at a small fraction of the trainable-parameter cost.

Instructed zero-shot image-to-text generation

Because the LLM is frozen and instruction-capable (particularly FlanT5), simply appending a text prompt after the visual soft prompt lets BLIP-2 do zero-shot instructed generation: visual knowledge reasoning, visual commonsense reasoning, multi-turn visual conversation, storytelling, and personalized image description — all without any instruction-tuning data, purely inherited from FlanT5’s pre-existing instruction-following ability.

For quantitative zero-shot VQA, prompts are templated ("Question: {} Answer:" for OPT, "Question: {} Short answer:" for FlanT5), with beam search (width 5) and a length penalty of −1 to bias toward short, VQA-style answers.

Results: BLIP-2 sets a new state of the art on VQAv2 and GQA, beating Flamingo80B by 8.7% on zero-shot VQAv2 despite 54× fewer trainable parameters (188M/104–108M-ish Q-Former variants vs. Flamingo80B’s 10.2B trainable params). On OK-VQA, BLIP-2 trails Flamingo80B — the authors attribute this to OK-VQA’s heavier reliance on open-world knowledge rather than visual understanding, where Flamingo80B’s 70B-parameter Chinchilla LM simply knows more facts than BLIP-2’s 11B FlanT5-XXL.

A clean scaling trend emerges: (1) ViT-g beats ViT-L for both LLM families; (2) larger LLMs within a family outperform smaller ones; (3) FlanT5 (instruction-tuned) beats OPT (purely unsupervised) on VQA. This supports the authors’ framing of BLIP-2 as a generic bootstrapping recipe — better unimodal backbones directly translate to better multimodal performance without redesigning the bridge.

Ablation: effect of Stage 1 representation learning

Removing the representation-learning stage (i.e., going straight to Stage 2’s generative training, which becomes conceptually similar to Flamingo’s Perceiver Resampler) causes a significant drop in zero-shot VQA performance for both LLM families. OPT is hit particularly hard, showing classic catastrophic-forgetting behavior where performance actively degrades as training proceeds. This is arguably the paper’s most important ablation: it’s direct evidence that the ITC/ITG/ITM pre-training in Stage 1 — not just having a bottleneck architecture — is what makes the bridge work.

Image captioning

Fine-tuned on COCO with the LLM frozen (only Q-Former + image encoder updated), using the prompt "a photo of" as the generation seed. BLIP-2 achieves state-of-the-art results on the zero-shot-transfer NoCaps benchmark (in-domain, near-domain, and out-domain subsets) and is competitive on COCO Karpathy-test, demonstrating strong generalization to unseen object categories.

Visual Question Answering fine-tuning

Here the Q-Former is additionally conditioned on the question: question tokens are fed into the Q-Former as input and interact with the queries via self-attention, steering the cross-attention layers toward image regions relevant to the specific question being asked (rather than a generic caption-style summary). Fine-tuned on VQAv2 + Visual Genome (following BLIP’s recipe), BLIP-2 achieves state-of-the-art results among open-ended generation models, and is competitive even against closed-ended classification models like BEiT-3.

Image-text retrieval

Since retrieval doesn’t need language generation, this task fine-tunes only the image encoder + Q-Former (no LLM) on COCO with the same ITC/ITM/ITG objectives, using a two-stage inference (top-128 candidates by contrastive similarity, then re-ranked by ITM score). BLIP-2 achieves state-of-the-art zero-shot transfer to Flickr30K. A targeted ablation shows that even though ITG doesn’t directly train retrieval, adding it improves retrieval metrics — supporting the paper’s hypothesis that ITG forces the queries to extract more genuinely language-relevant visual features, which also happens to help fine-grained alignment.

Strengths

• Compute-efficiency-per-unit-performance is a genuinely different axis of contribution, not just “another SOTA claim.” The 54× reduction in trainable parameters relative to Flamingo80B, combined with sub-9-day pre-training on a single node, made BLIP-2-style bootstrapping practically reproducible for academic labs in a way Flamingo-scale training was not.
• Clean, well-isolated ablations. The “with vs. without Stage 1” comparison is a rare example of a paper isolating exactly which component is doing the causal work, rather than just reporting an aggregate SOTA number.
• Genuine modularity. The scaling trend across ViT-L→ViT-g and OPT→FlanT5-XXL demonstrates the architecture isn’t overfit to one specific backbone pairing — a meaningful claim for a “bootstrapping” method, since the whole premise is “plug in better unimodal models later.”
• Emergent instruction-following without instruction data is a legitimately interesting empirical result: because the LLM is never touched, its instruction-following ability transfers “for free,” years before instruction-tuned multimodal models became standard practice.

Weaknesses and open questions

• The 32-query bottleneck is a strong architectural prior with unclear generality. It works well for the tasks tested (captioning, VQA, retrieval), which are relatively “gist”-oriented — but a fixed, small set of learned queries is a poor match for tasks needing fine-grained spatial detail (e.g., OCR of dense text in an image, precise counting, or grounding/localization). This is a plausible explanation for why later work (e.g., LLaVA-style approaches that pass a full grid of patch tokens rather than a compressed query bottleneck, or higher-resolution/tiling approaches in later multimodal LLMs) moved away from a hard 32-token bottleneck for tasks demanding fine visual detail.
• The in-context learning limitation is somewhat self-inflicted — it stems directly from the choice of single-pair pre-training data, not from anything fundamental about the Q-Former architecture. The paper is candid about this, but it does mean the “BLIP-2 can’t do in-context learning” framing is really “BLIP-2 wasn’t trained on data that supports in-context learning,” a distinction that matters when comparing architectures vs. comparing training recipes.
• Two-stage training is an added pipeline complexity. Compared to Flamingo’s single end-to-end generative objective (train the Perceiver Resampler + gated cross-attention directly), BLIP-2 requires a full separate pre-training stage (Stage 1, 250k steps) before the LLM is even introduced. The ablation shows this stage is important, but it also means the recipe has more moving parts, more hyperparameters, and a longer overall pre-training pipeline to reproduce correctly.
• OK-VQA result is a useful negative result the paper handles honestly (attributing the gap to world-knowledge capacity rather than claiming victory everywhere), but it also flags a real limitation: a frozen LLM bottlenecked through 32 visual tokens may not have a good pathway to combine “knowledge recall” with “visual grounding” for tasks that need both simultaneously.
• No open-ended generation quality evaluation (e.g., human preference studies, hallucination rate measurement) is reported in the main paper beyond the qualitative examples and a brief “incorrect output” appendix figure — quantifying generation quality/hallucination was mostly left to follow-up work (e.g., InstructBLIP added quantitative instruction-following and hallucination-adjacent evaluations).

Where this sits in the broader trajectory

BLIP-2’s Q-Former + frozen-backbone-bootstrapping pattern was highly influential: InstructBLIP (Dai et al., 2023) extended BLIP-2 by instruction-tuning the Q-Former (making it instruction-aware rather than purely question-conditioned), and MiniGPT-4 reused a BLIP-2-style vision encoder + Q-Former pipeline connected to Vicuna. The broader multimodal-LLM wave that followed (LLaVA, Qwen-VL, and others) largely kept “frozen or lightly-tuned vision encoder + frozen-or-tuned LLM + trainable bridge” as the dominant paradigm, though many subsequent designs replaced the fixed 32-query resampler with a simpler linear/MLP projection over all patch tokens (trading compute efficiency for finer visual detail) — a design choice directly informed by exactly the bottleneck trade-off discussed above.

Using the official implementation

BLIP-2 is implemented in Salesforce’s LAVIS library (github.com/salesforce/LAVIS) and is also available through Hugging Face transformers (Blip2Model, Blip2ForConditionalGeneration). For most practical use (inference, fine-tuning on a downstream task), you do not need to reproduce pre-training — pre-trained checkpoints for ViT-L/ViT-g × OPT-2.7B/6.7B/FlanT5-XL/FlanT5-XXL combinations are released.

A minimal inference sketch (conceptual, matching the paper’s architecture):

from transformers import Blip2Processor, Blip2ForConditionalGeneration
import torch
from PIL import Image

processor = Blip2Processor.from_pretrained("Salesforce/blip2-flan-t5-xl")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-flan-t5-xl", torch_dtype=torch.float16
).to("cuda")

image = Image.open("example.jpg").convert("RGB")
prompt = "Question: what is unusual about this image? Short answer:"

inputs = processor(images=image, text=prompt, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(out[0], skip_special_tokens=True))

This mirrors the paper’s prompting recipe exactly: the FlanT5 variant uses "Question: {} Short answer:", beam search is recommended (width 5) for VQA-style short answers, and a negative length penalty helps keep answers concise.

If you need to reproduce or adapt the pre-training pipeline

1. Pick frozen backbones first, and don’t skip Stage 1. The single most load-bearing empirical result in the paper is that skipping representation learning and going straight to generative pre-training severely hurts performance. If you’re adapting this recipe to new backbones (e.g., a newer vision encoder or LLM), budget for the full two-stage schedule rather than trying to shortcut to Stage 2 alone.
2. Reuse BERT-base initialization for the Q-Former rather than training the query-image cross-attention machinery from scratch — the paper does this and it’s a cheap way to inherit reasonable self-attention behavior for the text-transformer half of the module.
3. Match the attention-masking pattern to the objective, not just the loss function: unimodal mask for ITC, causal multimodal (UniLM-style) mask for ITG, and bidirectional mask for ITM. Getting this wrong silently defeats the purpose of each objective (e.g., without the unimodal mask, ITC would leak text information into the visual queries and stop being a real contrastive task).
4. Condition Q-Former on the question/prompt text for downstream VQA-style fine-tuning rather than only ever feeding it raw images — letting the query self-attention see the question tokens is what lets Q-Former’s cross-attention selectively focus on question-relevant image regions.
5. Mind the trade-off in bottleneck size. 32 queries of dimension 768 works well for the tasks BLIP-2 targets; if your downstream task needs fine-grained spatial precision (dense OCR, counting, grounding), consider whether a fixed small-query bottleneck is the right choice, or whether a design that preserves more patch-level detail (at higher compute cost) is more appropriate — this is exactly the trade-off later multimodal LLM architectures navigated differently.
6. FP16/BFloat16 frozen backbones are safe. The paper reports no measurable degradation from casting frozen ViT/LLM weights to lower precision during pre-training, which is a meaningful practical cost-saver if you’re compute constrained.