Segment Anything

Reading guide

Problem and motivation

Foundation models pre-trained on web-scale data have revolutionized NLP and vision-language alignment (e.g., CLIP), but general computer vision lacks a foundational system capable of broad zero-shot generalization across diverse segmentation tasks. Existing datasets for image segmentation are small compared to web-scale image-text corpora, and conventional segmentation architectures are typically tied to fixed semantic classes or specialized interactive setups.

To build a vision foundation model for segmentation, the authors simultaneously address three core elements:

1. Task: What objective allows general zero-shot transfer?
2. Model: What architecture supports flexible prompting in real time while resolving inherent visual ambiguity?
3. Data: How can web-scale segmentation masks be acquired when internet data lacks dense mask annotations?

Key ideas

• Promptable Segmentation Task: The model must return a valid segmentation mask given any prompt—such as foreground/background points, bounding boxes, dense masks, or free-form text. When a prompt is ambiguous (e.g., a single point on a shirt), the model must output multiple plausible hierarchical masks (subpart, part, whole).
• Asymmetric Amortized Architecture (SAM): A compute-heavy image encoder runs once per image to produce an embedding, while a lightweight prompt encoder and two-layer Transformer decoder evaluate prompts in $\sim 50\text{ ms}$ on a CPU inside a web browser.
• Flywheel Data Engine: A three-stage annotation pipeline:
  1. Assisted-manual: SAM aids annotators in a browser, dropping annotation time from 34s to 14s per mask (collecting 4.3M masks).
  2. Semi-automatic: SAM detects confident masks to guide annotators toward unannotated, less prominent objects (collecting 5.9M additional masks).
  3. Fully automatic: SAM is prompted on a uniform $32\times 32$ point grid plus zoomed-in image crops, generating the 1.1 billion masks in SA-1B.

Method

Model Architecture

SAM consists of three components:

1. Image Encoder: An MAE pre-trained Vision Transformer (ViT-H/16 default) adapted with windowed attention and four global attention blocks, processing $1024\times 1024$ inputs to produce a $64\times 64$ embedding with 256 channels.
2. Prompt Encoder:
   • Sparse prompts: Points and bounding boxes are mapped to 256-dimensional vectors using positional encodings summed with learned type embeddings. Free-form text is encoded using CLIP's text encoder.
   • Dense prompts: Input masks (at $4\times$ lower resolution) are downscaled via $2\times 2$ stride-2 convolutions and added element-wise to the image embedding.
3. Mask Decoder: A two-layer modified Transformer decoder. It updates prompt tokens and image embeddings through alternating prompt self-attention and bidirectional cross-attention (tokens-to-image and image-to-tokens). The image features are upscaled $4\times$ via transposed convolutions, and an MLP projects the output token to produce a dynamic linear classifier applied point-wise to the upscaled features.

Ambiguity-Aware Output and Loss

A single prompt often describes multiple nested structures. SAM addresses ambiguity by predicting 3 masks simultaneously with corresponding estimated IoU scores. During training, backpropagation applies only to the minimum loss across the candidates: $${\mathcal{L}}_{\text{mask}}=\underset{i\in \{1,2,3\}}{\min }\left(20\cdot {\mathcal{L}}_{\text{focal}}(m_i,y)+{\mathcal{L}}_{\text{dice}}(m_i,y)\right)$$ An auxiliary head minimizes the mean-squared-error between predicted IoU and ground truth IoU: $${\mathcal{L}}_{\text{IoU}}=\Vert {\widehat{\text{IoU}}}_i-\text{IoU}(m_i,y){\Vert }^2$$ When multiple prompts are provided, SAM outputs a single unambiguous mask using a fourth dedicated output token.

Results

• Zero-Shot Point-to-Mask Transfer: Across 23 diverse segmentation datasets (including egocentric, underwater, aerial, and microscopy imagery), SAM outperforms RITM on 16 of 23 datasets using a single center point prompt (with deltas reaching $+46.9$ IoU). When ambiguity is resolved via an oracle selector over SAM's 3 predictions, SAM outperforms RITM on all 23 datasets. Human quality ratings consistently rate SAM significantly higher (between 7 and 9 out of 10).
• Zero-Shot Edge Detection (BSDS500): Evaluated zero-shot using a $16\times 16$ grid with Sobel filtering and edge NMS, SAM achieves an ODS of 0.768 and an R50 of 0.928, performing competitively with supervised models like HED (ODS 0.788, R50 0.923).
• Zero-Shot Object Proposals (LVIS v1): SAM scores 59.3 AR@1000 without class supervision, outperforming a fully-supervised ViTDet-H baseline on medium objects (81.6 vs. 80.8), large objects (86.9 vs. 87.0), and rare categories (65.8 vs. 58.3).
• Zero-Shot Instance Segmentation: When prompted with ViTDet bounding boxes, SAM reaches 46.5 AP on COCO and 44.7 AP on LVIS. In human evaluations, annotators consistently rate SAM masks as cleaner and higher quality than fully supervised ViTDet outputs.

Limitations and open questions

• Fine details and extreme zoom: SAM can miss ultra-fine structures or produce small disconnected components compared to compute-heavy interactive methods that explicitly zoom into crops.
• Full-pipeline latency: While prompt decoding is real-time ($\sim 50\text{ ms}$), running the heavy ViT-H image encoder is not real-time.
• Semantic understanding: SAM yields class-agnostic masks; it does not natively provide semantic or panoptic category labels without composable exterior classifiers.
• Text-to-mask prompting: The zero-shot CLIP-aligned text prompting pipeline remains preliminary and sensitive compared to geometric prompts.