Learning Transferable Visual Models From Natural Language Supervision (CLIP)

TL;DR

CLIP throws out the idea that a computer vision model must be trained to predict a fixed, hand-labeled list of categories (the “1000 ImageNet classes” paradigm). Instead, it trains an image encoder and a text encoder jointly, from scratch, on 400 million (image, text) pairs scraped from the public internet, using a single proxy task: given a batch of images and a batch of text snippets, figure out which image goes with which text. This is a contrastive objective, not a generative or predictive one — the model never has to generate or reconstruct the exact caption, only to rank the correct pairing above the incorrect ones.

Because the text encoder learns to map arbitrary natural-language descriptions into the same embedding space as images, at test time you can build a classifier for any set of categories just by describing them in words (e.g., “a photo of a dog”, “a photo of a cat”) — no additional labeled training data, no fine-tuning, no new output head. This is what the paper calls zero-shot transfer, and it is CLIP’s central contribution: a single, generically pre-trained model that is benchmarked across more than 30 existing computer vision datasets and is frequently competitive with fully supervised, task-specific baselines. The headline result is that zero-shot CLIP matches the accuracy of the original supervised ResNet-50 on ImageNet without using any of ImageNet’s 1.28 million training labels.

The paper’s contribution is therefore threefold: (a) a dataset-construction and training recipe that makes natural-language supervision computationally practical at scale (400M pairs, contrastive objective, large batch size); (b) an extensive empirical study of zero-shot transfer across dozens of tasks (OCR, action recognition, geo-localization, fine-grained classification); and (c) a careful analysis of what this buys you — better robustness to distribution shift, but still real limitations on abstract/counting tasks, fine-grained tasks, truly out-of-distribution data (e.g., MNIST), and social bias.

Introduction and motivating work

The approach

# image_encoder - ResNet or Vision Transformer
# text_encoder   - CBOW or Text Transformer
# I[n, h, w, c]  - minibatch of aligned images
# T[n, l]        - minibatch of aligned texts
# W_i[d_i, d_e]  - learned proj of image to embed
# W_t[d_t, d_e]  - learned proj of text to embed
# t              - learned temperature parameter

# extract feature representations of each modality
I_f = image_encoder(I)          # [n, d_i]
T_f = text_encoder(T)           # [n, d_t]

# joint multimodal embedding [n, d_e]
I_e = l2_normalize(np.dot(I_f, W_i), axis=1)
T_e = l2_normalize(np.dot(T_f, W_t), axis=1)

# scaled pairwise cosine similarities [n, n]
logits = np.dot(I_e, T_e.T) * np.exp(t)

# symmetric loss function
labels = np.arange(n)
loss_i = cross_entropy_loss(logits, labels, axis=0)
loss_t = cross_entropy_loss(logits, labels, axis=1)
loss   = (loss_i + loss_t) / 2

Experiments

Comparison to human performance

To contextualize task difficulty, five humans each labeled all 3,669 test images of Oxford-IIIT Pets (37 cat/dog breeds), in zero-shot, one-shot, and two-shot settings (with the “shots” being example images per breed, no internet lookups allowed).

The striking pattern: humans jump from 54% → 76% accuracy after just one example per class, with almost all of that gain concentrated in images the human was previously uncertain about — i.e., humans “know what they don’t know” and can resolve that uncertainty from a single example. CLIP’s few-shot methods show no such jump (in fact, the paper notes few-shot linear probes on CLIP features can underperform zero-shot CLIP, a counter-intuitive result revisited in the Limitations section). This exposes a real gap between human sample-efficient learning (which appears to integrate strong priors) and current few-shot machine learning methods, which mostly do not make effective use of prior knowledge. The hardest classes for CLIP also tend to be the hardest for humans, suggesting shared sources of difficulty — likely label noise and genuinely out-of-distribution or ambiguous images.

Data overlap analysis

A natural worry with a 400M-image internet-scraped pre-training set: some evaluation-set images might have leaked into training, inflating reported zero-shot accuracy. Rather than trying to scrub all possible overlaps up front (which would require knowing every future eval dataset in advance, and would make adding new evals expensive), the authors instead measured and quantified the effect:

1. Run a duplicate detector (detailed in the paper’s Appendix C) per eval dataset; manually tune a per-dataset similarity threshold for high precision.
2. Split each eval dataset into Overlap (near-duplicate found in training data) vs. Clean (no duplicate found) vs. All (unmodified).
3. Compute zero-shot RN50x64 accuracy on each split; report All − Clean as the estimated accuracy inflation from contamination, with binomial significance testing.

Result: across 35 datasets, 9 have zero detected overlap (synthetic/specialized datasets like MNIST, GTSRB, CLEVR, or ones guaranteed clean by publication date, like ObjectNet and Hateful Memes) — evidence the detector’s false-positive rate is low. Median overlap 2.2%, mean 3.2%. Overall accuracy is rarely shifted by more than 0.1%; only 7 of 35 datasets exceed that, and only 2 remain statistically significant after Bonferroni correction. Largest single effect: +0.6% on Birdsnap (12.1% overlap). Largest overlap by volume: Country211 at 21.5% (because Country211 is built from YFCC100M, a subset of which is in WIT) — yet this only inflates accuracy by 0.2%, because the accompanying text of overlapping images rarely mentions the geo-localization-relevant information the eval actually measures. The authors note some confounds (e.g., Kinetics-700’s apparent −20% “overlap” effect turns out to be duplicate all-black transition frames, not genuine leakage) and conclude the effect of train/test overlap on CLIP’s headline numbers is small and mostly not statistically significant, consistent with similar analyses in Mahajan et al. (2018) and Kolesnikov et al. (2019).

Limitations

The paper is unusually candid about where CLIP falls short. Consolidated list:

• Zero-shot CLIP is not close to overall SOTA. On datasets with training splits, it’s roughly competitive with a plain supervised linear classifier on ResNet-50 features — a baseline that, in 2021, was already well below task-specific state of the art. The authors estimate roughly a 1000× compute increase would be needed for zero-shot CLIP to reach overall SOTA — “infeasible to train with current hardware.”
• Weak on abstract/systematic tasks and fine-grained classification. Poor on counting (CLEVRCounts), distance estimation (KITTI Distance), and fine-grained categories like car models, aircraft variants, flower species — near-random on some novel tasks unlikely to be represented in the pre-training data.
• Poor generalization to truly out-of-distribution data, even when the task itself is one CLIP is otherwise good at. The canonical example: CLIP learns a strong OCR-like representation for digitally-rendered text (it does well on “Rendered SST2,” synthetic sentences turned into images), but only achieves 88% on handwritten MNIST digits — a simple logistic regression on raw pixels beats it. Nearest-neighbor search confirms almost nothing resembling MNIST digits exists in CLIP’s training data. This is read as evidence CLIP does not solve deep learning’s brittle out-of-distribution generalization problem — it just tries to make more of the input space “in-distribution” by using a huge, varied dataset, which is a strategy that can still be violated (as MNIST shows).
• Limited to a closed set of concepts at inference time, unlike a genuinely generative approach (e.g., image captioning) that could produce novel outputs — but the authors found the generative/captioning baseline they tried far less compute-efficient (see the Approach and Experiments sections above), so this is a deliberate trade-off, not an oversight. Suggested future directions: jointly training a contrastive + generative objective, or searching over natural-language explanations at inference time (cf. Andreas et al., 2017).
• Doesn’t fix data inefficiency, just outruns it with data volume. If every one of the ~12.8 billion image-views seen across 32 training epochs were shown at one per second, that’s 405 years. Combining CLIP with self-supervision/self-training methods is suggested as a route to genuine data-efficiency gains.
• Methodological caveats: despite the “zero-shot” framing, the authors repeatedly consulted full validation sets during development — an advantage a genuinely zero-shot deployment wouldn’t have. Also, the primary 27-dataset evaluation suite was assembled somewhat organically alongside CLIP’s own development, raising co-adaptation/selection-bias concerns; a purpose-built zero-shot benchmark is suggested as future work.
• Social bias, inherited from unfiltered, uncurated internet image–text pairs (see Broader Impacts, next section).
• Prompt sensitivity. Zero-shot accuracy can be sensitive to exact wording/phrasing of the class-description prompt, sometimes requiring “prompt engineering” trial and error to get right.
• Counter-intuitive zero-shot vs. few-shot drop. Because CLIP does not directly optimize for few-shot performance — the paper falls back to fitting linear probes on top of frozen CLIP features for few-shot evaluation — performance can decrease when moving from zero-shot to few-shot (adding a small number of labeled examples), the opposite of the human pattern documented above.

Broader impacts

(This section’s specifics are drawn primarily from OpenAI’s companion blog post for this paper, since the raw PDF/HTML extraction available in this environment was truncated before reaching the paper’s own Section 7 / Appendix F. The direction and substance are consistent with how this section of the paper is widely cited elsewhere.)

Because CLIP lets anyone define an arbitrary classifier just by naming categories in text, the choice of category labels itself becomes a source of model behavior and potential harm — there’s no dataset-curation step to catch it. Two studies illustrate this:

• Bias probing with FairFace. When CLIP is given a label set that mixes FairFace demographic categories with a handful of pejorative terms (e.g., “criminal,” “animal”), it classified images of people aged 0–20 into an egregious/derogatory category at a rate of ~32.3%. Simply adding “child” as an available class dropped that misclassification rate to ~8.7% — a small change in the label vocabulary sharply changes model behavior, which the paper treats as both a demonstration of risk and a (partial, imperfect) mitigation.
• Surveillance-relevant capability testing. Since CLIP needs no task-specific training data, it can be pointed at sensitive tasks with unusually little friction. The authors tested this directly on celebrity identification: CLIP reaches 59.2% top-1 accuracy picking the right identity out of 100 candidates, and 43.3% out of 1,000 candidates. Framed against dedicated, production-grade facial recognition systems, this is not competitive — but the fact that a general-purpose, task-agnostic model gets non-trivial accuracy on an unrequested, ethically fraught task at all is presented as exactly the kind of risk that needs characterizing before deployment, not after.

The paper’s own framing (echoed in the blog) is that flexibility is a double-edged sword: the same lack of a fixed, curated label taxonomy that makes CLIP broadly useful also means bias and misuse-potential show up wherever a user chooses to point the text encoder, rather than being contained by a dataset-curation process the model creators control. The authors present this as motivation for follow-on research into characterizing capabilities, shortcomings, and biases of this model class rather than as a solved problem.

Related work

CLIP sits at the intersection of three research threads the paper explicitly connects itself to:

• Zero-data / zero-shot learning, dating back over a decade in computer vision as a way to generalize to unseen object categories (Larochelle et al., 2008; Lampert et al., 2009), and more recently reframed (as this paper does) as a way to measure task-learning rather than just category generalization — mirroring the shift GPT-2/GPT-3 made in NLP.
• Natural language as a flexible prediction space. Early proofs of concept: Socher et al. (2013, Stanford) trained a CIFAR-10 model to predict into a word-vector embedding space and showed it could name two never-seen classes; DeVISE (Frome et al., 2013) scaled a similar idea to ImageNet-scale fine-tuning. Visual N-Grams (Li et al., 2017) is singled out as the most direct ancestor — the first to attempt zero-shot transfer to existing image classification benchmarks using a generically pre-trained model, at 11.5% ImageNet accuracy.
• Modern architectures for text-image representation learning: VirTex (autoregressive captioning), ICMLM (masked language modeling), and ConVIRT (contrastive learning, in the medical-imaging domain) — all recent, small-scale (100–200K image) proofs of concept that CLIP scales up by roughly 2,000–4,000×.
• More broadly, CLIP’s efficiency choices connect to the contrastive representation learning literature (SimCLR/Chen et al. 2020a,b, MoCo/Chen et al. 2020d, BYOL/Grill et al. 2020) and to compute-scaling-law work (Hestness et al., 2017; Kaplan et al., 2020) that CLIP’s own Figure 9 scaling result echoes.

Conclusion

The paper’s own framing: task-agnostic, internet-scale pre-training — the recipe that drove NLP’s recent breakthroughs (GPT-family models) — transfers to computer vision when natural language is used as the supervisory signal instead of hand-curated labels. Like GPT models, CLIP picks up a surprisingly wide variety of tasks “for free” during pre-training (OCR, geo-localization, action recognition, and more), demonstrable purely via zero-shot transfer, with no task-specific fine-tuning. The finding that zero-shot evaluation appears to be a more representative measure of real-world model capability than in-distribution ImageNet benchmarking (because a zero-shot model cannot overfit to a benchmark it was never trained or tuned against) is presented as evidence that the field should weight zero-shot/broad-distribution evaluation more heavily going forward — not just for vision, but potentially, the authors suggest, for other modalities as well.

Appendix highlights

The full paper includes six appendices (A–F) covering material this guide summarizes only at a high level, since the raw text extraction available in this environment did not reach them:

• Appendix A — Linear-probe evaluation details. Full protocol and dataset list for the representation-learning study: the roughly 27-dataset suite mentioned throughout includes (among others named in the main text) ImageNet, CIFAR-10, CIFAR-100, STL-10, Food101, StanfordCars, FGVCAircraft, OxfordPets, Flowers102, Caltech101, SUN397, DTD, EuroSAT, RESISC45, GTSRB, KITTI Distance, Country211, PatchCamelyon (PCam), CLEVRCounts, HatefulMemes, SST2 (rendered), FER2013, Birdsnap, UCF101, Kinetics700, PascalVOC2007, and MNIST — plus aYahoo and SUN for the smaller Visual-N-Grams comparison, and the 7 natural-distribution-shift sets (ImageNetV2, ImageNet-A, ImageNet-R, ImageNet Sketch, ObjectNet, Youtube-BB, ImageNet-Vid) used in the robustness study.
• Appendix B — Extended per-dataset results tables for both zero-shot and linear-probe evaluations, underlying the dataset-delta figures above.
• Appendix C — Duplicate detector methodology underlying the data-overlap analysis above.
• Appendix D — Dataset ablation, studying performance as a function of pre-training dataset size/composition.
• Appendix E — OCR performance deep dive, including the “Rendered SST2” digitally-rendered-text benchmark referenced in the Limitations section.
• Appendix F — Broader impacts, extended. A more detailed treatment of the bias and surveillance-capability analyses summarized above, including additional demographic and label-set sensitivity experiments.

If you need the exact figures/tables from these appendices, the most reliable path is to open the PDF directly (https://arxiv.org/pdf/2103.00020) or the HTML mirror at https://ar5iv.labs.arxiv.org/html/2103.00020, since this guide’s source extraction was cut off before those sections.

Glossary

• Zero-shot transfer — applying a model to a task/dataset it was never trained or fine-tuned on, using only a natural-language description of the target categories.
• Contrastive objective — a training signal that rewards correctly ranking a matching pair above non-matching pairs, rather than reconstructing/predicting exact content (contrast with a generative/captioning objective).
• In-batch negatives — for a batch of \(N\) pairs, every non-matching combination of image \(i\) and text \(j \neq i\) is treated as a negative example “for free,” without needing separately mined negatives.
• InfoNCE / N-pair loss — the family of contrastive losses (Sohn, 2016; Oord et al., 2018) that CLIP’s symmetric cross-entropy loss belongs to.
• Linear probe — freezing a pre-trained encoder and training only a linear (logistic regression) classifier on top of its output features, used here as the paper’s primary representation-learning evaluation method.
• Effective robustness vs. relative robustness (Taori et al., 2020) — effective robustness is accuracy gain under distribution shift beyond what’s predicted by the in-distribution/out-of-distribution accuracy relationship observed across many models; relative robustness is any accuracy gain at all.
• Hypernetwork (Ha et al., 2016) — a network that generates the weights of another network; CLIP’s text encoder functions as a hypernetwork generating a zero-shot linear classifier’s weight vector from a text description.
• Prompt engineering / ensembling — customizing (and averaging over multiple variants of) the natural-language template used to describe classes, to close the gap between the pre-training text distribution (full sentences) and the bare category names typically provided by classification datasets.

References

The in-text citations transcribed from the extracted portion of the paper (author/year as they appear; full bibliographic details can be found in the paper’s own reference list, https://arxiv.org/pdf/2103.00020):

• Radford, A. et al. (2018). Improving Language Understanding by Generative Pre-Training (GPT-1).
• Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2).
• Brown, T. et al. (2020). Language Models are Few-Shot Learners (GPT-3).
• Devlin, J. et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• Raffel, C. et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5).
• Deng, J. et al. (2009). ImageNet: A Large-Scale Hierarchical Image Database.
• Mori, Y. et al. (1999). Content-based image retrieval via caption word prediction.
• Quattoni, A. et al. (2007). Learning visual representations via manifold learning in classifier weight space.
• Srivastava, N. & Salakhutdinov, R. (2012). Multimodal learning with Deep Boltzmann Machines.
• Joulin, A. et al. (2016). Learning visual features from large weakly supervised data (YFCC100M).
• Thomee, B. et al. (2016). YFCC100M: The New Data in Multimedia Research.
• Li, A. et al. (2017). Learning Visual N-Grams from Web Data.
• Desai, K. & Johnson, J. (2020). VirTex: Learning Visual Representations from Textual Annotations.
• Bulent Sariyildiz, M. et al. (2020). ICMLM: Learning Visual Representations with Caption Annotations.
• Zhang, Y. et al. (2020). ConVIRT: Contrastive Learning of Medical Visual Representations from Paired Images and Text.
• Mahajan, D. et al. (2018). Exploring the Limits of Weakly Supervised Pretraining (Instagram hashtags).
• Kolesnikov, A. et al. (2019). Big Transfer (BiT): General Visual Representation Learning.
• Dosovitskiy, A. et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT).
• Xie, Q. et al. (2020). Self-training with Noisy Student improves ImageNet classification.
• Sohn, K. (2016). Improved Deep Metric Learning with Multi-class N-pair Loss Objective.
• Oord, A. van den, Li, Y., & Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding (InfoNCE).
• Bachman, P., Hjelm, R. D., & Buchwalter, W. (2019). Learning Representations by Maximizing Mutual Information Across Views.
• Chen, T. et al. (2020a, 2020b). SimCLR: A Simple Framework for Contrastive Learning of Visual Representations (and v2, 2020c).
• Chen, X. et al. (2020d). Improved Baselines with Momentum Contrastive Learning (MoCo v2).
• Grill, J.-B. et al. (2020). Bootstrap Your Own Latent (BYOL).
• He, K. et al. (2016a). Deep Residual Learning for Image Recognition (ResNet); (2016b) original ResNet models; (2019) Bag of Tricks for Image Classification with Convolutional Neural Networks (ResNet-D).
• Zhang, R. (2019). Making Convolutional Networks Shift-Invariant Again (anti-aliased blur pooling).
• Tan, M. & Le, Q. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.
• Vaswani, A. et al. (2017). Attention Is All You Need.
• Sennrich, R., Haddow, B., & Birch, A. (2015). Neural Machine Translation of Rare Words with Subword Units (BPE).
• Kingma, D. & Ba, J. (2014). Adam: A Method for Stochastic Optimization.
• Loshchilov, I. & Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Warm Restarts (cosine schedule); (2017) Decoupled Weight Decay Regularization (AdamW).
• Wu, Z. et al. (2018). Unsupervised Feature Learning via Non-Parametric Instance Discrimination.
• Touvron, H. et al. (2019). Fixing the train-test resolution discrepancy (FixRes).
• Micikevicius, P. et al. (2017). Mixed Precision Training.
• Griewank, A. & Walther, A. (2000); Chen, T. et al. (2016). Gradient checkpointing.
• Dhariwal, P. et al. (2020). Half-precision Adam statistics (Jukebox).
• Kornblith, S., Shlens, J., & Le, Q. (2019). Do Better ImageNet Models Transfer Better?
• Zhai, X. et al. (2019). A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (VTAB).
• Recht, B. et al. (2019). Do ImageNet Classifiers Generalize to ImageNet? (ImageNetV2).
• Barbu, A. et al. (2019). ObjectNet: A Large-scale Bias-controlled Dataset.
• Hendrycks, D. et al. (2019). Natural Adversarial Examples (ImageNet-A); (2020a) The Many Faces of Robustness (ImageNet-R); (2020b) pre-training and sentiment robustness.
• Wang, H. et al. (2019). Learning Robust Global Representations by Penalizing Local Predictive Power (ImageNet-Sketch).
• Shankar, V. et al. (2019). Distribution shift in Youtube-BB / ImageNet-Vid.
• Taori, R. et al. (2020). Measuring Robustness to Natural Distribution Shifts in Image Classification.
• Hendrycks, D. & Dietterich, T. (2019). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (ImageNet-C).
• Geirhos, R. et al. (2018). ImageNet-trained CNNs are biased towards texture (Stylized ImageNet); (2020) Shortcut Learning in Deep Neural Networks.
• Parkhi, O. et al. (2012). Cats and Dogs (Oxford-IIIT Pets).
• Coates, A., Ng, A., & Lee, H. (2011). An Analysis of Single-Layer Networks in Unsupervised Feature Learning (STL-10).
• Stallkamp, J. et al. (2011). The German Traffic Sign Recognition Benchmark (GTSRB).
• Tian, Y., Krishnan, D., & Isola, P. (2019). Contrastive Multiview Coding; Tian, Y. et al. (2020). Rethinking Few-Shot Image Classification.
• Ha, D., Dai, A., & Le, Q. (2016). HyperNetworks.
• Lei Ba, J., Swersky, K., & Salakhutdinov, R. (2015). Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions.
• Elhoseiny, M., Saleh, B., & Elgammal, A. (2013). Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions.
• Larochelle, H., Erhan, D., & Bengio, Y. (2008). Zero-data Learning of New Tasks.
• Lampert, C., Nickisch, H., & Harmeling, S. (2009). Learning to Detect Unseen Object Classes by Between-Class Attribute Transfer.
• Liu, P. et al. (2018). Emergent task-learning in Wikipedia-generating language models.
• Hestness, J. et al. (2017); Kaplan, J. et al. (2020). Neural scaling laws.
• Andreas, J., Klein, D., & Levine, S. (2017). Learning with Latent Language.
• Bhargava, S. & Forsyth, D. (2019). Exposing and correcting the gender bias in image captioning datasets.
• D’Amour, A. et al. (2020). Underspecification Presents Challenges for Credibility in Modern Machine Learning.
• Locatello, F. et al. (2020). Disentangled representation learning, critique.
• Yogatama, D. et al. (2019); Linzen, T. (2020). Broad-coverage evaluation for NLP generalization.
• Miller, J. et al. (2020). Robustness of question-answering models to distribution shift.
• Oliver, A. et al. (2018). Realistic Evaluation of Deep Semi-Supervised Learning Algorithms.
• Lake, B. et al. (2016). Building Machines That Learn and Think Like People.
• Frome, A. et al. (2013). DeViSE: A Deep Visual-Semantic Embedding Model.
• Socher, R. et al. (2013). Zero-shot learning through cross-modal transfer (Stanford CIFAR-10 word-embedding proof of concept).

Primary sources used to build this guide:

1. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020. https://arxiv.org/abs/2103.00020
2. OpenAI (2021). CLIP: Connecting text and images. https://openai.com/research/clip (used to reconstruct the paper’s Broader Impacts and Conclusion sections, which were beyond the extraction window of the primary PDF source in this environment).
3. Code and pre-trained weights: https://github.com/OpenAI/CLIP