Segment Anything (SAM)

Definition

Borrowing from NLP’s next-token-prediction paradigm — one objective that serves both pre-training and downstream transfer — the authors define promptable segmentation:

Given any prompt, return a valid segmentation mask.

A prompt specifies what to segment: a set of foreground/background points, a rough box, a coarse mask, free-form text, or any combination. Validity is the crucial subtlety: when a prompt is ambiguous (a single point on a shirt could indicate the shirt or the person), the output need only be a reasonable mask for at least one plausible object — much as a language model should still return a coherent response to an ambiguous query.

Pre-training

The task implies a natural pre-training algorithm adapted from interactive segmentation: for each training mask, simulate a sequence of prompts and compare the model’s predictions to ground truth across the sequence. The key departure from classic interactive segmentation is the objective — rather than aiming to eventually produce a good mask after enough user input, SAM must always produce a valid mask for any prompt, including ambiguous ones. This property is what later makes fully automatic annotation feasible.

Zero-shot transfer via prompting

Because the model can respond appropriately to any prompt, downstream tasks reduce to engineering the right prompt. For example, given a cat bounding-box detector, cat instance segmentation is solved by feeding each detected box to SAM as a prompt. This is task generalization — importantly different from multi-task systems, which perform a fixed set of tasks whose training and test tasks coincide. SAM instead acts as a composable component inside larger systems, potentially serving tasks unknown at design time.

flowchart LR
    subgraph Prompts["Prompt types"]
        P1["Foreground / background points"]
        P2["Bounding box"]
        P3["Coarse mask"]
        P4["Free-form text"]
    end
    Prompts --> SAM["SAM<br/>(promptable segmentation)"]
    SAM --> V["Valid mask(s)<br/>for at least one plausible object"]

    subgraph Downstream["Downstream tasks via prompt engineering"]
        D1["Edge detection"]
        D2["Object proposals"]
        D3["Instance segmentation<br/>(detector box → SAM)"]
        D4["Text-to-mask"]
    end
    V --> Downstream

Design constraints

Three requirements flow from the task and from the goal of real-world interactive use:

• Flexible prompts — points, boxes, masks, and text must all be accepted.
• Amortized real-time — masks must be computed fast enough for interactive prompting.
• Ambiguity-aware — a single ambiguous prompt should yield multiple valid masks.

A surprisingly simple three-part design satisfies all three.

Architecture

flowchart LR
    IMG["Input image"] --> IE["Image encoder<br/>(MAE-pretrained ViT-H,<br/>adapted for high-res)"]
    IE --> EMB["Image embedding<br/>(computed once per image)"]

    subgraph PE["Prompt encoder"]
        SPARSE["Sparse: points, box, text<br/>positional enc. + learned emb.<br/>(text via CLIP encoder)"]
        DENSE["Dense: mask<br/>convolution, summed with image emb."]
    end

    EMB --> DEC
    PE --> DEC["Mask decoder<br/>(2x modified Transformer blocks:<br/>prompt self-attn + bidirectional cross-attn)"]
    DEC --> UP["Upsample + dynamic linear classifier (MLP)"]
    UP --> MASKS["3 output masks<br/>+ IoU confidence per mask"]

Image encoder. An MAE-pretrained Vision Transformer (ViT), minimally adapted to process high-resolution inputs. It runs once per image; its output embedding is cached and reused for every subsequent prompt — the crux of the “amortized real-time” property.

Prompt encoder. Two families:

• Sparse prompts (points, boxes, text): points and boxes are represented by positional encodings summed with learned type-specific embeddings; free-form text uses an off-the-shelf CLIP text encoder.
• Dense prompts (masks): embedded via convolutions and summed element-wise with the image embedding.

Mask decoder. A lightweight, modified Transformer decoder that maps the image embedding, prompt embeddings, and a learned output token to a mask. Each of its two blocks applies prompt self-attention plus cross-attention in both directions (prompt→image and image→prompt), updating all embeddings. The image embedding is then upsampled, and an MLP turns the output token into a dynamic linear classifier that computes a foreground probability at every location.

Resolving ambiguity

With a single output, the model would average conflicting valid masks under ambiguity. Instead SAM predicts three masks per prompt — empirically sufficient, since nested masks are typically at most three deep (whole → part → subpart). During training, loss is backpropagated only through the minimum-loss mask (a “hindsight”/multiple-choice loss). Each mask also gets a predicted confidence score (estimated IoU) used to rank outputs.

Efficiency

Given a precomputed image embedding, the prompt encoder + mask decoder run in ~50 ms in a web browser on CPU, enabling seamless interactive prompting. The heavy ViT image encoder is the expensive part, but its cost is amortized across all prompts on that image.

Losses and training

Mask supervision combines focal loss and dice loss. Training simulates an interactive setup by randomly sampling prompts over 11 rounds per mask, so SAM integrates directly into the data engine’s annotation loop. Text prompts are handled separately (see Section 1.9.5).

Stage 1 — Assisted-manual

Professional annotators label masks by clicking foreground/background points in a browser tool powered by SAM (using precomputed embeddings for real-time interaction), refining with pixel-precise brush/eraser tools. No semantic constraints — annotators freely label both “stuff” and “things,” labeling in order of prominence and moving on after ~30 s per mask. SAM was initially trained on public datasets, then retrained only on newly collected masks; over the stage the encoder scaled ViT-B → ViT-H and the model was retrained 6 times. Average time per mask fell from 34 s to 14 s (≈6.5× faster than COCO polygon annotation), and masks/image rose from ~20 to ~44. Yield: 4.3M masks from 120k images.

Stage 2 — Semi-automatic

Goal: increase mask diversity. A generic “object” bounding-box detector trained on stage-1 masks pre-fills confident masks; annotators then focus on the remaining, less prominent objects. The model was retrained 5 times; time/mask returned to ~34 s (harder objects), and masks/image rose 44 → 72 (including automatic masks). Added 5.9M masks in 180k images (10.2M total).

Stage 3 — Fully automatic

Now feasible because (a) enough diverse masks had been collected and (b) the model was ambiguity-aware. SAM is prompted with a 32×32 regular grid of points; each point yields masks for whole/part/subpart. Masks are filtered by predicted IoU (confidence) and stability (thresholding the probability map at 0.5±δ yields similar masks), then de-duplicated with NMS; zoomed-in crops improve small masks. Applied to all 11M images, producing 1.1B masks.

Single-point valid-mask segmentation

The hardest, most ambiguous prompt: one foreground point. Because ground truth rarely enumerates all valid masks, automatic mIoU is supplemented by a human study (annotators rate masks 1–10). Points are sampled from the mask “center” (distance-transform maximum); SAM reports its most confident mask by default. The main baseline is RITM, a strong interactive segmenter.

Results. SAM beats RITM on 16 of 23 datasets by automatic mIoU (by up to ~47 IoU). With an “oracle” that picks the best of SAM’s 3 masks, SAM beats RITM on all 23. In the human study, annotators consistently rate SAM’s masks substantially higher than RITM (mean ratings ~7–9); an ambiguity-unaware, single-mask ablation scores lower than full SAM but still above RITM. Notably, on datasets where SAM loses on automatic metrics (e.g., DRAM, IBD), humans still rate it higher — illustrating the unreliability of automatic single-point metrics under ambiguity. As prompts grow from 1→9 points, methods converge (SAM is not tuned for the very-high-IoU regime); with random (rather than center) point sampling, SAM’s advantage grows.

Zero-shot edge detection

Approach. A simplified auto-mask pipeline: prompt SAM with a 16×16 grid (768 masks, 3/point), NMS, then compute edges via Sobel filtering of unthresholded mask probability maps + lightweight edge NMS. SAM never saw BSDS500 images or edge labels.

Results (BSDS500). SAM produces reasonable edge maps and outperforms classic zero-shot methods (Sobel, Canny, Felzenszwalb–Huttenlocher). It trails specialized learned detectors (HED, EDETR) but is competitive given it was never trained for edges. SAM tends to predict more edges than the ground truth annotates (it segments things not labeled in BSDS).

Zero-shot object proposals

Approach. Run the full automatic mask generator; treat masks as object proposals. Metric: average recall (AR) on LVIS v1 (chosen for its large vocabulary). Baseline: a strong ViTDet-H detector’s proposal stage.

Results. SAM is competitive with the strong supervised baseline. It outperforms on medium/large objects and on rare/common categories, but is worse on small and frequent objects, where ViTDet — trained specifically on LVIS — can exploit dataset-specific biases SAM never learned.

Zero-shot instance segmentation

Approach. A textbook composition: feed an object detector’s boxes (ViTDet) to SAM as prompts to produce instance masks.

Results (COCO, LVIS). SAM’s mask AP is slightly below the fully supervised ViTDet on COCO. But because COCO ground-truth masks are relatively coarse, a human study rates SAM’s masks higher than ViTDet’s — SAM’s boundaries are often visibly better than the “ground truth.” On LVIS, whose masks are higher quality, the gap narrows.

flowchart LR
    IMG["Image"] --> DET["Object detector<br/>(e.g., ViTDet)"]
    DET --> BOX["Detected boxes"]
    BOX --> SAM["SAM<br/>(box prompt)"]
    IMG --> SAM
    SAM --> INST["Instance masks"]

Zero-shot text-to-mask (proof of concept)

Approach. A preliminary probe of free-form-text prompting. During training, each mask is paired with its CLIP image embedding as a prompt; because CLIP aligns image and text, at inference the model can be prompted with CLIP text embeddings instead — no explicit text supervision on masks required.

Results. SAM can segment from prompts like “a wheel” or “beaver tooth grille.” When text alone is imprecise, adding a point refines the result. The authors frame this as a proof of concept, not a robust text-to-mask system.

Foundation-model framing

SAM is positioned as a promptable, composable vision foundation model. Its value lies less in any single benchmark number and more in being a reusable component: prompt engineering and composition let one model serve many tasks — including ones unanticipated at training time — analogous to how CLIP serves as the alignment module inside DALL·E.

Limitations

• Fine structure & boundaries. SAM can miss thin structures, occasionally hallucinates small disconnected components, and produces less crisp boundaries than compute-intensive methods that “zoom in.”
• Interactive high-IoU regime. Dedicated interactive segmenters can outperform SAM when many points are provided; SAM is optimized for generality, not the very-high-IoU regime.
• Real-time caveat. Only the prompt encoder + mask decoder are real-time (~50 ms). With the heavy ViT-H image encoder, overall processing is not real-time.
• Text-to-mask is preliminary. Not robust; needs better modeling.
• Domain-specific tools may still beat SAM within their domains.
• Semantic/panoptic ambiguity. It is unclear how to design simple prompts for semantic and panoptic segmentation, and “what counts as an object” is inherently ambiguous.

Impact

By releasing SAM (Apache 2.0) and SA-1B, the authors aim to make promptable segmentation a building block for larger systems and to seed further vision-foundation-model research. SAM catalyzed a large ecosystem of downstream work (medical imaging, video, annotation tooling, and follow-ups such as HQ-SAM, MobileSAM, and SAM 2).