Learning Transferable Visual Models From Natural Language Supervision

Reading guide

Problem and motivation

State-of-the-art vision models traditionally rely on crowd-labeled, closed-set classification benchmarks such as ImageNet. This fixed-label paradigm restricts their usability: recognizing any concept outside the target vocabulary requires collecting and annotating new data. Conversely, natural language processing had successfully shifted toward task-agnostic pre-training directly on web-scale text (e.g., GPT-3), producing models capable of zero-shot generalization via text-to-text interfaces. The authors investigate whether scaling natural language supervision on web-scale data can yield visual representations that generalize broadly across computer vision tasks without dataset-specific supervised fine-tuning.

Key ideas

• Learning from Natural Language Supervision: Treating web-scraped (image, text) pairs as training signals connects visual features directly to rich natural language semantics, bypassing 1-of-N discrete class labeling.
• Contrastive Learning for Efficiency: Predicting exact text tokens (generative captioning) is computationally wasteful. Formulating the objective as a contrastive matching task—distinguishing true (image, text) pairs from incorrect pairs across a minibatch—proves 4x more sample-efficient than bag-of-words prediction and far faster than generative modeling.
• Zero-Shot Classification as Dynamic Linear Classifiers: Target dataset classes are embedded through the text encoder into feature vectors. These vectors serve as weights for a zero-shot linear classifier against the image embeddings.
• Prompt Engineering and Ensembling: Zero-shot performance is sensitive to prompt syntax and context. Wrapping class names in full sentences (e.g., "A photo of a {label}.") and ensembling across multiple prompt variations improves zero-shot transfer by roughly 5 points on ImageNet.

Method (with the key equations)

CLIP constructs a multi-modal embedding space by jointly training an image encoder $f_{\text{image}}$ and a text encoder $f_{\text{text}}$ on a dataset of 400 million internet-scraped (image, text) pairs (WIT).

1. Feature Extraction and Projection: Given a minibatch of $N$ image-text pairs $(I_i,T_i)$ for $i\in \{1,\dots ,N\}$, each encoder maps inputs to a joint dimension $d$ with linear projections $W_i$ and $W_t$: $$I_i^e=\frac{f_{\text{image}}(I_i)W_i}{\Vert f_{\text{image}}(I_i)W_i{\Vert }_2},T_j^e=\frac{f_{\text{text}}(T_j)W_t}{\Vert f_{\text{text}}(T_j)W_t{\Vert }_2}$$

2. Similarity Matrix: Logits are computed via scaled cosine similarity: $$S_{i,j}=\frac{⟨I_i^e,T_j^e⟩}{\tau }$$ where $\tau$ is a learnable temperature parameter initialized to 0.07, clipped to prevent scaling logits beyond 100.

3. Symmetric Cross-Entropy Loss: The total loss is the average of image-to-text and text-to-image classification losses over the $N\times N$ pairing matrix: $$\mathcal{L}=\frac{1}{2}\left(-\frac{1}{N}\sum _{i=1}^N\log \frac{\exp (S_{i,i})}{{\sum }_{j=1}^N\exp (S_{i,j})}-\frac{1}{N}\sum _{j=1}^N\log \frac{\exp (S_{j,j})}{{\sum }_{i=1}^N\exp (S_{i,j})}\right)$$

4. Zero-Shot Inference: For a dataset with $K$ classes, text templates (e.g., "a photo of a {class_name}") are encoded into $w_k=T_k^e$. For test image $x$, predicted probabilities are: $$p(y=k\mid x)=\frac{\exp (⟨I_x^e,w_k⟩/\tau )}{{\sum }_{j=1}^K\exp (⟨I_x^e,w_j⟩/\tau )}$$

Results (with the most important numbers)

• Zero-Shot ImageNet: CLIP (ViT-L/14@336px) reaches 76.2% top-1 accuracy (and 95% top-5 accuracy) zero-shot on ImageNet, matching the original supervised ResNet-50 without using its 1.28 million training images.
• Zero-Shot vs. Supervised Baseline: Evaluated across 27 downstream datasets, zero-shot CLIP outperforms a fully supervised linear probe on top of standard ResNet-50 features on 16 of the 27 datasets.
• Few-Shot Comparison: Zero-shot CLIP matches the average performance of a 4-shot linear classifier trained on the exact same CLIP feature representations, and rivals a 16-shot BiT-M ResNet-152x2 model.
• Linear Probe Representation Quality: When evaluating frozen representations with supervised linear probes, CLIP's best model outperforms Noisy Student EfficientNet-L2 on 21 of 27 datasets, excelling in OCR, video action recognition (Kinetics-700: +14.5% over ResNet-50), and geo-localization.
• Distribution Shift Robustness: Across 7 natural distribution shift datasets (ImageNetV2, ImageNet-R, ImageNet-A, ObjectNet, etc.), zero-shot CLIP reduces the drop between in-distribution and out-of-distribution accuracy by up to 75% compared to standard ImageNet-trained models.

Limitations and open questions

• Abstract, Specialized, and Systematic Tasks: Zero-shot CLIP struggles severely with counting (CLEVRCounts), fine-grained tasks with narrow distinctions (FGVCAircraft, Flowers102), and domain-specific tasks outside web text commonality (tumor detection on PatchCamelyon; distance estimation on KITTI).
• Truly Out-of-Distribution Data: On MNIST handwritten digits, zero-shot CLIP achieves only 88% accuracy—underperforming logistic regression on raw pixels—demonstrating that CLIP circumvents rather than solves out-of-distribution brittleness.
• Compute Requirements: Reaching overall supervised state-of-the-art across all benchmarks purely zero-shot is estimated to require an estimated 1000x increase in compute with existing scaling trends.
• Social Biases and Denigration: Probing with the FairFace benchmark reveals significant disparate failure modes, including non-human and crime-related misclassifications skewed toward Black individuals and youth, which vary heavily with class label design and thresholding.