Visual Instruction Tuning (LLaVA)

Abstract

Instruction tuning large language models (LLMs) using machine-generated instruction-following data has been shown to improve zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and an LLM for general-purpose visual and language understanding. To facilitate future research on visual instruction following, we construct two evaluation benchmarks with diverse and challenging application-oriented tasks. Our experiments show that LLaVA demonstrates impressive multimodal chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model, and code publicly available.

TL;DR — Key Contributions

The paper makes four concrete contributions:

1. Multimodal instruction-following data. A pipeline that reformats existing image-caption pairs into instruction-following conversations by prompting text-only GPT-4/ChatGPT with symbolic (textual) representations of images — captions and bounding boxes — rather than the images themselves.
2. A large multimodal model (LMM): LLaVA. A simple architecture connecting a frozen CLIP ViT-L/14 vision encoder to the Vicuna LLM via a single trainable linear projection layer, trained end-to-end on the generated data.
3. Two new evaluation benchmarks: LLaVA-Bench (COCO) and LLaVA-Bench (In-the-Wild). Designed to stress-test multimodal instruction-following and chat ability, scored automatically using GPT-4 as a judge.
4. Open-source release of the 158K-sample instruction dataset, model weights, training code, and a chat demo.

The headline numbers: 85.1% relative score vs. text-only GPT-4 on the synthetic instruction-following benchmark, and a new state-of-the-art 92.53% accuracy on ScienceQA when LLaVA’s predictions are ensembled with GPT-4 acting as a judge.

1. Introduction

Humans understand the world through multiple channels — vision and language chief among them — because each channel captures concepts the other cannot as efficiently. AI research has long pursued a general-purpose assistant that can follow multimodal vision-and-language instructions to complete real-world tasks.

The paper frames prior progress along two separate tracks that, until this work, had not been fully merged:

• Vision-only foundation models. A wave of language-augmented vision models (CLIP, GLIP, BLIP, DALL-E 2, Latent Diffusion, Imagen, GLIGEN, etc.) achieved strong open-world performance on classification, detection, segmentation, captioning, and generation. However, each task is typically solved by a separate model with a fixed interface — language is used only to describe content, not to flexibly instruct the system, so interactivity and adaptability to user intent are limited.
• Instruction-tuned LLMs. ChatGPT and GPT-4 demonstrated that language can serve as a universal interface: task instructions expressed in natural language can steer a single end-to-end model to solve arbitrary tasks. Open-source efforts — LLaMA, Alpaca, Vicuna, GPT-4-LLM — replicated this instruction-following ability using machine-generated instruction data, but remained strictly text-only.

LLaVA’s core idea is to extend instruction tuning into the language-image multimodal space — the first attempt of its kind — to move toward a general-purpose visual assistant. The four contributions listed in the TL;DR above are given as the paper’s explicit list of contributions.

2. Related Work

The paper positions itself against two bodies of work:

Multimodal instruction-following agents. Existing systems fall into two categories:

• (i) End-to-end trained, task-specific models — e.g., vision-and-language navigation agents, Habitat embodied agents, and InstructPix2Pix (which edits images given a natural-language instruction). Each is built and trained for one narrow task.
• (ii) Tool-orchestration systems that coordinate multiple existing models via an LLM or a framework like LangChain — e.g., Visual ChatGPT, X-GPT, MM-REACT, VisProg, ViperGPT. These “glue” pretrained vision tools together using an LLM as a controller/orchestrator, rather than training one unified model end-to-end.

LLaVA instead pursues an end-to-end trained, single multimodal model that generalizes across multiple tasks — distinct from both categories.

Instruction tuning. In NLP, instruction tuning (InstructGPT/ChatGPT, FLAN-T5, FLAN-PaLM, OPT-IML) reliably improves zero-/few-shot generalization of LLMs by fine-tuning on instruction-formatted data. The paper argues it is natural to import this idea into vision-language modeling. It notes prior multimodal models — Flamingo (described as “the GPT-3 moment” for multimodal, due to strong zero-shot/in-context learning), BLIP-2, FROMAGe, KOSMOS-1, PaLM-E (for embodied AI), OpenFlamingo, and LLaMA-Adapter — as precedents that connect vision and language models, but observes that none of them were explicitly tuned on vision-language instruction-following data, so their multimodal task performance lags behind their language-only performance. LLaVA is presented as filling exactly this gap.

The paper also draws a sharp distinction between visual instruction tuning (this paper — improving a model’s ability to follow instructions) and visual prompt tuning (a parameter-efficient adaptation technique, unrelated in goal).

3. GPT-Assisted Visual Instruction Data Generation

This is arguably the paper’s most novel methodological contribution: a recipe for manufacturing multimodal instruction data without any new human annotation, by exploiting a text-only GPT-4/ChatGPT as a “teacher.”

Human: Xq Xv <STOP>
Assistant: Xc <STOP>

flowchart TD
    A["Image X_v + 5 human captions X_c"] --> B["Symbolic text encoding"]
    B --> C["Captions\n(scene description)"]
    B --> D["Bounding boxes\n(object + normalized coords)"]
    C --> E["Few-shot prompt to GPT-4\n(seed examples via in-context learning)"]
    D --> E
    E --> F1["Conversation data\n58K samples"]
    E --> F2["Detailed description data\n23K samples"]
    E --> F3["Complex reasoning data\n77K samples"]
    F1 --> G["LLaVA-Instruct-158K"]
    F2 --> G
    F3 --> G

4. Visual Instruction Tuning

flowchart LR
    subgraph Vision["Vision Pathway"]
        Xv["Image X_v"] --> VE["Vision Encoder g(·)\n(CLIP ViT-L/14, frozen)"]
        VE --> Zv["Visual features Z_v"]
        Zv --> Proj["Projection W\n(trainable linear layer)"]
        Proj --> Hv["Visual tokens H_v"]
    end

    subgraph Language["Language Pathway"]
        Xq["Language instruction X_q"] --> Hq["Instruction tokens H_q"]
    end

    Hv --> LLM["Language Model f_φ(·)\n(Vicuna)"]
    Hq --> LLM
    LLM --> Xa["Language response X_a"]

X_system-message <STOP>
Human: X¹_instruct <STOP>  Assistant: X¹_a <STOP>
Human: X²_instruct <STOP>  Assistant: X²_a <STOP>  ⋯

flowchart TD
    Start(("Pretrained\nCLIP ViT-L/14 + Vicuna")) --> S1

    subgraph S1["Stage 1 — Pre-training for Feature Alignment"]
        direction TB
        A1["CC3M filtered to CC-595K\n(image, caption) pairs"] --> A2["Naive expansion:\nHuman: brief-describe Xq Xv → Assistant: Xc"]
        A2 --> A3["Freeze vision encoder AND LLM\nTrain ONLY projection matrix W"]
        A3 --> A4["Result: visual tokens H_v aligned\nwith frozen LLM word-embedding space\n(a visual tokenizer for the LLM)"]
    end

    S1 --> S2

    subgraph S2["Stage 2 — Fine-tuning End-to-End"]
        direction TB
        B1["Freeze vision encoder\n(still frozen)"] --> B2["Update BOTH W and\nLLM weights φ"]
        B2 --> B3{"Use case"}
        B3 -->|"Multimodal Chatbot"| B4["Fine-tune on\nLLaVA-Instruct-158K\n(conversation multi-turn,\ndetail + reasoning single-turn)"]
        B3 -->|"Science QA"| B5["Fine-tune on ScienceQA\n(single-turn: question+context → reasoning+answer)"]
    end

    S2 --> End(("LLaVA"))

5. Experiments

All models are trained with 8× A100 GPUs, following Vicuna’s hyperparameters. Pre-training on CC-595K runs 1 epoch (learning rate 2e-3, batch size 128); fine-tuning on LLaVA-Instruct-158K runs 3 epochs (learning rate 2e-5, batch size 32). Full hyperparameters are in Appendix C.

flowchart TD
    Q["ScienceQA question"] --> L["LLaVA prediction"]
    Q --> G["Text-only GPT-4 prediction"]
    L --> C{"GPT-4 failed to answer\n(insufficient context)?"}
    G --> C
    C -->|"Yes"| UseLLaVA["Use LLaVA's answer\n(GPT-4 complement scheme) → 90.97%"]
    C -->|"No, but answers differ"| Judge["Re-prompt GPT-4 as a JUDGE\nwith both Q, both answers"]
    Judge --> Final["GPT-4 gives final answer\n(GPT-4 judge scheme) → 92.53% new SoTA"]
    C -->|"No, and answers agree"| Agree["Keep the (shared) answer"]

6. Conclusion

The paper’s own summary: it demonstrates the effectiveness of visual instruction tuning, presents an automatic pipeline for generating language-image instruction data, and trains LLaVA — achieving new SoTA on ScienceQA when fine-tuned there, and strong visual-chat ability when fine-tuned on multimodal chat data. It also contributes the first benchmark suite explicitly targeting multimodal instruction-following capability. The authors describe this as an initial step, focused on real-life tasks, and point readers to a follow-up paper — “Improved Baselines with Visual Instruction Tuning” (LLaVA-1.5) — for stronger quantitative results on academic benchmarks.

Appendix A. Broader Impact

The authors discuss risks and mitigations specific to releasing a general-purpose visual assistant:

• Malicious input: mitigated via (1) the OpenAI Filter API on text instructions, and (2) an NSFW filter on uploaded images.
• Hallucination: like text-only LLMs, LLaVA can generate outputs not grounded in the actual input — a particular concern in high-stakes domains like medicine.
• Biases: inherited from both the vision encoder (CLIP) and the language decoder (LLaMA/Vicuna).
• Energy consumption: not a major concern at LLaVA’s scale (small pretraining set), but flagged as a future concern if scaled up (e.g., to a 65B-parameter LLaMA backbone).
• Evaluation complexities: their GPT-4-based evaluation is consistent within their study, but its generality/robustness across other settings is untested; aspects like hallucination degree and fine-grained visual understanding are not yet well covered by their benchmarks.

The authors conclude that the research-community benefits of releasing LLaVA outweigh these risks, since open release enables collaborative improvement and mitigation.

Appendix B. More Results

Additional qualitative demonstrations of emergent behavior (Figures 2–6 in the paper; not independently verified as directly linkable image assets — see the paper’s appendix online for the original screenshots):

• Figure 2 — HTML/JS/CSS generation from a sketch. Given a simplified hand-drawn mockup of a “joke website,” LLaVA generates working HTML/JS/CSS for an interactive page (reveal a punchline on button click), with only a minor bug — echoing a similar live demo from OpenAI’s original GPT-4 announcement.
• Figure 3 — Instruction-following without explicit recognition prompts. LLaVA answers “what should I be cautious about here?” for a lake/pier photo, and later writes a full travel-blog-style post about the same image on request — showing it can flexibly follow varied instructions grounded in one image, without being told to “first describe the image.”
• Figure 4 — Relating visual content to pretrained world knowledge. Shown a sunset photo of a couple on a boat, LLaVA connects it to Titanic (Jack and Rose) and recalls plot details from its language-model pretraining — demonstrating that visual grounding can trigger downstream textual knowledge retrieval.
• Figure 5 — Recognizing famous artwork / humor. LLaVA identifies a comedic dog-in-a-dress painting as riffing on the Mona Lisa, and — in a fresh conversation — correctly identifies an actual Mona Lisa image and its painter, Leonardo da Vinci.
• Figure 6 — Zero-shot celebrity recognition (Elon Musk). LLaVA recognizes Elon Musk in both a plain headshot and a humorous “doge” meme edit, despite Musk never appearing in either of LLaVA’s own training stages — attributed to the frozen CLIP encoder having likely seen such images during its own (separate) pretraining, with the finding highlighted as surprising evidence of generalization to unseen visual concepts.

Table 9 (chicken-nugget-map meme) further compares LLaVA, GPT-4, BLIP-2, and OpenFlamingo on humor explanation — LLaVA and GPT-4 both explain the joke; BLIP-2 and OpenFlamingo fail to engage with the instruction at all and default to captioning.

Table 10 walks through a worked ScienceQA example (a rocking-chair material question) showing exactly how the “GPT-4 as judge” scheme corrects an LLaVA hallucination (LLaVA claims the chair is partly “silk,” which the judge correctly overrides in favor of “wood,” reasoning from general material-science knowledge even without seeing the image).

Appendix C. Training Details

Appendix D. Assets

The paper lists the following released assets (originally under an anonymized GitHub link for review purposes, later published at the LLaVA GitHub repo):

1. Source code
2. README / setup instructions
3. Instructions to launch the demo
4. All GPT-4 prompts and few-shot examples used for data generation
5. LLaVA-Instruct-158K dataset
6. LLaVA-Bench (COCO) and LLaVA-Bench (In-the-Wild)
7. Model checkpoints (noted as ~25GB compressed — exceeding GitHub LFS’s 5GB limit at submission time, hence released separately)

Appendix E. Data

CC3M filtering methodology (→ CC-595K): noun phrases are extracted from every CC3M caption via spaCy, and their frequencies counted across the whole dataset.

• Noun phrases occurring fewer than 3 times are dropped (too rare / idiosyncratic to be useful concept coverage).
• For noun phrases occurring more than 100 times, only a random subset of 100 captions containing that phrase is kept (to avoid massive redundancy for common concepts).
• Captions are added starting from the least-frequent remaining noun phrases, building up the final pool.
• This filtering reduces the noun-phrase vocabulary from 108,182 unique noun phrases (full CC3M) down to 31,423 unique noun phrases (filtered), while shrinking the pair count to ~595K — the paper’s own summary is that the filtered set trades off total pair volume for better balance of concept coverage (avoiding the raw dataset’s long tail of near-duplicate high-frequency captions).

(Figure 7 in the paper is a log-log plot of noun-phrase frequency before/after filtering; not independently linked here — see the paper’s appendix online.)

flowchart LR
    A["CC3M\n~3.3M image-caption pairs\n108,182 unique noun phrases"] --> B["spaCy noun-phrase extraction\n+ frequency counting"]
    B --> C{"Frequency check"}
    C -->|"freq < 3"| D["Drop\n(too rare)"]
    C -->|"freq > 100"| E["Randomly keep 100 captions\n(cap over-represented concepts)"]
    C -->|"3 ≤ freq ≤ 100"| F["Keep all captions"]
    E --> G["CC-595K\n~595,000 pairs\n31,423 unique noun phrases"]
    F --> G

Appendix F. Prompts

The paper documents its exact GPT-4 prompting template (Table 13). The system prompt used to generate the “conversation”-type data reads (reproduced verbatim):

“You are an AI visual assistant, and you are seeing a single image. What you see are provided with five sentences, describing the same image you are looking at. Answer all questions as you are seeing the image.”

“Design a conversation between you and a person asking about this photo. The answers should be in a tone that a visual AI assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers.”

“Include questions asking about the visual content of the image, including the object types, counting the objects, object actions, object locations, relative positions between objects, etc. Only include questions that have definite answers: (1) one can see the content in the image that the question asks about and can answer confidently; (2) one can determine confidently from the image that it is not in the image. Do not ask any question that cannot be answered confidently.”

“Also include complex questions that are relevant to the content in the image, for example, asking about background knowledge of the objects in the image, asking to discuss about events happening in the image, etc. Again, do not ask about uncertain details. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary.”

This system message is followed programmatically by a handful of few-shot in-context examples (Tables 15 and 16 in the paper give two full worked examples — a fire-hydrant-in-snow photo, and a skier photo — each showing 3–4 turns of Q&A the model should imitate in style and depth), and finally the actual query (captions + boxes for the target image) is appended as the final user turn. The paper notes that prompts for the other two data types (detailed description, complex reasoning) follow the same pattern but are omitted from the main text for space; readers are pointed to the released codebase.

Glossary of Symbols

Key Takeaways

The central methodological insight of this paper is that you can bootstrap high-quality multimodal instruction data purely from a text-only LLM, by describing images symbolically (captions + boxes) rather than requiring a multimodal teacher model — this sidesteps the chicken-and-egg problem of needing a good multimodal model to generate training data for a good multimodal model. Architecturally, the paper shows that a deliberately minimal design (frozen CLIP + frozen/then-unfrozen Vicuna + a single linear layer) is sufficient to reach near-SoTA results when the training data is good, which is the paper’s implicit thesis: data quality and instruction diversity, not architectural complexity, were the main bottleneck for multimodal instruction-following at the time. The two-stage training recipe (align first, then fine-tune end-to-end) and the “GPT-4 as judge” ensembling trick are both reused in numerous follow-up works, including the paper’s own sequel, LLaVA-1.5.

Selected References

(Full reference list has 63 entries in the original paper; the most load-bearing ones for understanding this paper are reproduced below. See the arXiv listing for the complete bibliography.)

• Chiang et al., “Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality,” 2023.
• Radford et al., “Learning transferable visual models from natural language supervision” (CLIP), arXiv:2103.00020, 2021.
• Touvron et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023.
• Taori et al., “Stanford Alpaca: An instruction-following LLaMA model,” 2023.
• Alayrac et al., “Flamingo: a visual language model for few-shot learning,” arXiv:2204.14198, 2022.
• Li et al., “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv:2301.12597, 2023.
• Lu et al., “Learn to explain: Multimodal reasoning via thought chains for science question answering” (ScienceQA), NeurIPS 2022.
• Zhang et al., “Multimodal chain-of-thought reasoning in language models” (MM-CoT), arXiv:2302.00923, 2023.
• OpenAI, “GPT-4 Technical Report,” 2023.
• Liu, Li, Li, Lee, “Improved Baselines with Visual Instruction Tuning” (LLaVA-1.5), 2023 — the direct follow-up paper referenced in the Conclusion.

Guide prepared from a full read of the arXiv v2 (11 Dec 2023) PDF text extraction of 2304.08485, cross-referenced against the arXiv/ar5iv HTML renderings for figure asset links. All tables and quoted passages are reproduced/paraphrased from the original paper; equations and the CC3M-filtering / GPT-4-ensembling diagrams are original Mermaid recreations synthesized from the paper’s prose descriptions, not literal copies of paper figures (only Figure 1 is a literal figure reproduction, linked directly from arXiv’s rendering assets).