Depth Anything V2

TL;DR

Depth Anything V2 is a monocular depth estimation model — it predicts a depth map from a single RGB image, no stereo pair or LiDAR required. The paper’s core message is almost contrarian: don’t train on more labeled real-world depth data — train on almost none of it. Real depth-labeled datasets (from stereo cameras, structured-light sensors, or SfM/multi-view reconstruction) are inherently noisy: sensors mislabel mirrors, glass, thin structures like tree branches or basketball nets, and moving objects. Feeding a model that noise teaches it to be “average,” not sharp.

Instead, the authors build a three-stage recipe:

1. Train a large teacher model (DINOv2-Giant backbone) purely on synthetic images, which have perfect, noise-free depth ground truth (because it comes straight from a graphics engine).
2. Use that teacher to generate pseudo depth labels on 62 million unlabeled real images.
3. Train smaller student models (25M–1.3B parameters) purely on this pseudo-labeled real data — never touching noisy human-labeled real datasets at all.

This sidesteps synthetic data’s biggest weakness (limited scene diversity — a graphics engine only has so many rendered rooms and streets) by borrowing scene diversity from huge unlabeled real photo collections, while sidestepping real labeled data’s biggest weakness (sensor noise) by never using it. The result: models that are sharper and more robust than Depth Anything V1, more than 10× faster than diffusion-based competitors like Marigold, and — despite being much smaller and faster — more accurate on the authors’ new, cleaner benchmark. The paper also introduces DA-2K, a sparse but precisely-annotated evaluation benchmark built to expose exactly the failure modes (transparency, reflections, fine structures) that older, noisier benchmarks like NYU-D and KITTI cannot reliably measure.

1. Introduction & Motivation

Monocular depth estimation (MDE) — predicting a per-pixel depth map from a single 2D image — is foundational for robotics, AR/VR, 3D reconstruction, and autonomous driving. The field has moved from narrow, in-domain “metric depth” models (trained and tested on the same type of scene, e.g. only indoor rooms or only driving scenes) toward zero-shot relative depth estimation: models like MiDaS and Depth Anything V1 that generalize to any image and predict depth up to an unknown scale and shift (see the Glossary for “affine-invariant depth”).

The paper opens by stating its goal plainly: not to chase “fancy techniques,” but to surface a small number of practical findings that meaningfully improve on Depth Anything V1. Compared with V1, Depth Anything V2 produces “much finer and more robust depth predictions” via three practices, which form the spine of the entire paper:

1. Replace all labeled real images with synthetic images when training the teacher model.
2. Scale up the teacher model’s capacity (using the largest DINOv2 backbone, “DINOv2-G”).
3. Bridge to student models via large-scale pseudo-labeled real images rather than training students directly on labeled data or via classic feature-distillation.

The introduction also previews the paper’s second contribution: because existing depth benchmarks (NYU-D, KITTI, and similar) are themselves noisy and narrow in scene coverage, “performance on these benchmarks may not reflect real-world reliability.” So alongside the model, the authors build DA-2K, a new benchmark with sparse but precise, human-verified relative-depth annotations spanning eight diverse real-world scenario categories at high resolution.

Two families of prior approaches are contrasted throughout the paper:

• Discriminative models (regression-style, e.g. MiDaS, Depth Anything V1) — trained on large amounts of labeled data, fast at inference, but limited by the precision of that labeled data.
• Generative / diffusion-based models (e.g. Marigold, GeoWizard, DepthFM) — built on Stable Diffusion, trained mostly on synthetic data since diffusion priors need clean signal, capable of very fine detail, but slow at inference (diffusion sampling requires multiple denoising steps) and comparatively weaker on complex/diverse real scenes.

Depth Anything V2’s central bet is that you can get the fine detail of the diffusion camp and the speed and robustness of the discriminative camp, if you fix the training data pipeline rather than the architecture.

2. Background: How Depth Anything V1 Used Labeled Data

Depth Anything V1’s recipe was: use a modest amount of labeled real images plus a very large pool of unlabeled real images (turned into pseudo-labels by a teacher trained on the labeled set), which gave it strong “data-driven” generalization. This version 2 paper revisits that design and questions the wisdom of leaning on labeled real data at all, given how noisy real-world depth ground truth typically is.

Real depth labels come from three main acquisition methods, each with characteristic failure modes:

• Depth sensors (e.g. Kinect-style structured light / time-of-flight, used for datasets like NYU-D): fail on reflective and transparent surfaces (mirrors, glass) and struggle with fine/thin structures.
• Stereo cameras: produce noisy or missing disparity at depth discontinuities, textureless regions, and thin structures.
• SfM / multi-view reconstruction: only reconstructs static, well-textured, well-covered geometry — thin or moving objects and low-texture regions frequently drop out or are wrong.

A model trained directly on these labels inherits their blind spots — it learns that mirrors have “solid” depth, that thin branches or nets are invisible or blurred, because that’s literally what the training signal told it. This is the paper’s motivating diagnosis, illustrated in the source paper’s Figure 3 (noisy ground truth from sensor, stereo, and SfM sources, alongside the resulting prediction errors of models trained on them).

Figure 3 — “Various noise in GT depth labels and prediction errors in correspondingly trained models.” (source: ar5iv mirror).

3. The Problem With Synthetic Data

If real labels are noisy, the obvious fix is synthetic data: images rendered by a graphics engine come with perfect, dense, noise-free depth for free. So why hasn’t the field simply switched to synthetic data already? The paper identifies two structural problems:

1. Distribution shift. Synthetic images are “too ‘clean’ in color and ‘ordered’ in layout, while real images contain more randomness” in lighting, clutter, texture, and composition. A model trained only on synthetic renders can overfit to that cleanliness and fail to transfer to messy real photographs.

2. Restricted scene coverage. Synthetic datasets are “iteratively sampled from graphics engines with pre-defined fixed scene types” (the paper’s examples: “living room,” “street scene”). Even if the geometry within those scenes is pixel-perfect, the variety of scenes is inherently bounded by what artists have modeled — nothing like the boundless diversity of real-world photography (a monastery courtyard, a coral reef, a construction site, a concert stage).

The authors test this empirically: only the largest encoder (DINOv2-G) trained purely on synthetic data manages to transfer reasonably well to real images; smaller encoders (ViT-S/B/L) trained the same way transfer poorly. Even DINOv2-G, the best case, still shows systematic failure cases on scene types/objects underrepresented in the synthetic training distribution (paper’s Figure 6). This finding is important because it justifies two separate design choices later: (a) only the largest model is trusted as the synthetic-only teacher, and (b) smaller student models are not trained directly on synthetic data — they need a different bridge, which is the paper’s next section.

Figure 6 — “Failure cases of the most capable DINOv2-G model when purely trained on synthetic images.” (source: ar5iv mirror). >

Figure 5 — “Qualitative comparison of different vision encoders on synthetic-to-real transfer.” (source: ar5iv mirror).

4. Why Large-Scale Unlabeled Real Images Matter

The paper’s resolution to the synthetic-data dilemma is not to add real labeled data back in (that reintroduces sensor noise), but to add real unlabeled data, relabeled automatically by the synthetic-trained teacher. This gives three benefits, each explained in the paper:

• Bridging the domain gap. Pseudo-labeled real images act as “an intermediate learning target” — a way for the student to see real-world pixel statistics (real lighting, clutter, texture) while still being supervised by labels that trace back to clean synthetic ground truth (via the teacher), rather than by noisy sensor/stereo/SfM labels directly.
• Restoring scene coverage. Public unlabeled image collections (e.g. Places365, ImageNet-21K, Open Images, SA-1B) “can easily cover numerous distinct scenes” that a synthetic renderer never modeled, fixing synthetic data’s diversity problem without touching noisy labels.
• A safer form of knowledge transfer. Rather than distilling the teacher into students via feature matching or logit matching (classic knowledge distillation, which can be brittle when teacher and student differ hugely in capacity — here 1.3B → 25M parameters), the authors transfer knowledge at the prediction level: the teacher just produces depth maps, and the student learns from those maps as if they were ordinary training labels. The paper argues this is “safer” across such a large capacity gap.

This is the conceptual heart of the paper — synthetic data supplies precision, unlabeled real data (relabeled by a strong teacher) supplies diversity and realism, and combining them lets a small, fast student model inherit both properties without ever seeing a single noisy human-annotated real depth map.

5. Method: Depth Anything V2

flowchart TD
    subgraph Stage1["Stage 1 — Train the Teacher"]
        A["Synthetic images<br/>595K images, 5 datasets<br/>(BlendedMVS, Hypersim, IRS, TartanAir, VKITTI2)"] --> B["Teacher model<br/>DINOv2-G encoder + DPT decoder"]
        B --> C["Precise depth predictions<br/>(clean synthetic ground truth supervises training)"]
    end

    subgraph Stage2["Stage 2 — Pseudo-Label Real Images"]
        D["Unlabeled real images<br/>62M images, 8 datasets<br/>(SA-1B, Open Images, ImageNet-21K, Places365,<br/>BDD100K, LSUN, Google Landmarks, Objects365)"] --> E["Trained teacher<br/>(frozen, from Stage 1)"]
        E --> F["Pseudo depth labels<br/>on all 62M real images"]
    end

    subgraph Stage3["Stage 3 — Train Students"]
        F --> G["Student training<br/>(pseudo-labeled real images only)"]
        G --> H1["Student: ViT-S (~25M params)"]
        G --> H2["Student: ViT-B (~97M params)"]
        G --> H3["Student: ViT-L (~335M params)"]
        G --> H4["Student: ViT-G (~1.3B params)"]
    end

    C -.trained model feeds forward.-> E

    style Stage1 fill:#eef,stroke:#446
    style Stage2 fill:#efe,stroke:#464
    style Stage3 fill:#fee,stroke:#644

6. DA-2K: A New Evaluation Benchmark

flowchart TD
    A["Source images<br/>(keywords generated via GPT-4,<br/>images downloaded from Flickr)"] --> B{"Two annotation pipelines"}

    B --> P1S1["Pipeline 1: Automatic candidate mining"]
    P1S1 --> P1S2["Run SAM (Segment Anything Model)<br/>to predict object masks"]
    P1S2 --> P1S3["Sample two keypoints<br/>from mask boundaries"]
    P1S3 --> P1S4["Query 4 expert models for relative depth:<br/>Depth Anything V1, Marigold, GeoWizard, Depth Anything V2"]
    P1S4 --> P1S5{"Do experts agree?"}
    P1S5 -->|Yes, unanimous| P1S7["Accept as candidate label"]
    P1S5 -->|Disagreement| P1S6["Send to human annotator<br/>to resolve (or skip if truly ambiguous)"]
    P1S6 --> P1S7

    B --> P2S1["Pipeline 2: Manual hard-case mining"]
    P2S1 --> P2S2["Annotators manually identify<br/>challenging point pairs missed by automation<br/>(e.g. reflections, transparency, thin structures)"]
    P2S2 --> P1S7

    P1S7 --> V["Triple-check every annotation<br/>(independent verification passes)"]
    V --> Z["DA-2K benchmark<br/>1,000 images · 2,000 point-pair labels<br/>8 scenario categories"]

7. Experiments

8. Related Work

The paper situates itself within the evolution of monocular depth estimation:

• From in-domain metric depth to zero-shot relative depth. Early MDE models trained and tested within a single domain (indoor-only or driving-only). MiDaS was an early push toward zero-shot generalization by scaling up labeled training data (~2M images); later work pushed this further — Metric3D to ~8M labels, ZeroDepth to ~15M.
• Two architectural camps. Discriminative regression models (MiDaS-style, DINOv2-based like Depth Anything V1) versus generative diffusion-based models (Marigold, GeoWizard) built on Stable Diffusion. The paper notes SD-based methods “exclusively utilize synthetic images with complete depth information,” which is part of why they achieve fine detail — but most prior discriminative work stayed committed to real labeled data, which V2 argues is the wrong default.
• Positioning against Depth Anything V1. V1’s strength was data-driven scaling of unlabeled real images; V2 keeps that instinct but flips the labeled-data source to synthetic, combining what the paper calls a “data-driven (large-scale pseudo-labeled real images) and model-driven (scaling up the teacher model)” strategy.
• Knowledge transfer framing. The paper explicitly distinguishes its prediction-level pseudo-labeling approach from traditional feature-level or logit-level knowledge distillation, arguing the former is “safer” when the teacher and student differ enormously in size (1.3B teacher down to a 25M student).

9. Conclusion

The paper’s own summary of its contribution has three parts:

1. A more robust, fine-detail model family (25M–1.3B parameters) that avoids the precision/robustness trade-off seen in prior work, achieved via a training-data recipe (synthetic → pseudo-labeled real) rather than an architectural change.
2. Strong downstream transferability, validated by fine-tuning into metric depth models and (per the appendix) semantic segmentation.
3. DA-2K, a new “versatile evaluation benchmark with precise annotations and diverse scenes,” addressing the measurement gap that made it hard to even see V2’s advantages on older benchmarks.

The authors frame the release (models across all four scales, plus the DA-2K benchmark) as intended to support both future monocular depth research and broader multimodal AI systems that might consume depth as an input signal.

10. Appendix Highlights

11. Glossary

• Monocular depth estimation (MDE): Predicting a per-pixel depth map from a single 2D image (no stereo pair, no LiDAR).
• Affine-invariant depth: A depth prediction that is correct only up to an unknown global scale and shift — i.e., the model gets relative depth relationships right, but not absolute real-world distances, unless separately calibrated.
• Metric depth: Depth predictions in real physical units (e.g. meters), as opposed to affine-invariant/relative depth.
• AbsRel: Mean absolute relative error between predicted and ground-truth depth — lower is better.
• δ₁ (delta-1): The percentage of pixels where max(pred/gt, gt/pred) < 1.25 — i.e., prediction is within 25% of ground truth — higher is better.
• DINOv2: A family of self-supervised vision transformer encoders (from Meta AI) used here as the backbone for both teacher and student depth models.
• DPT (Dense Prediction Transformer): A transformer-based decoder architecture for turning encoder features into dense per-pixel outputs like depth maps.
• Scale- and shift-invariant loss (\(\mathcal{L}_{ssi}\)): A regression loss (from MiDaS) robust to the unknown affine ambiguity in relative depth prediction.
• Gradient matching loss (\(\mathcal{L}_{gm}\)): A multi-scale loss encouraging predicted depth gradients (edges) to match ground-truth gradients — drives sharpness at boundaries/thin structures.
• Pseudo-labeling: Using a trained model’s own predictions as training labels for further training (here: teacher’s depth predictions become student training targets).
• Knowledge distillation: Broadly, transferring knowledge from a large “teacher” model to a smaller “student” model; this paper uses prediction-level pseudo-labeling rather than classic feature/logit distillation.
• SAM (Segment Anything Model): A foundation model for promptable image segmentation, used here to generate candidate object masks for DA-2K’s keypoint sampling.
• ZoeDepth: A prior metric depth estimation pipeline the authors reuse (swapping in their own encoder) to produce metric depth models in §7.3.

12. Key Takeaways

• Don’t trust real depth labels blindly — sensors, stereo, and SfM all have structural blind spots (mirrors, glass, thin structures), and a model trained directly on them inherits those blind spots.
• Synthetic data gives you noise-free labels but bounded scene diversity; large-scale unlabeled real data, relabeled by a strong synthetic-trained teacher, gives you the missing diversity without reintroducing label noise.
• A big capacity gap between teacher and student (1.3B → 25M here) is better bridged by prediction-level pseudo-labeling than by classic feature/logit distillation.
• Filtering out the highest-loss (likely mislabeled) regions during training, and weighting a gradient-matching loss heavily, both directly target fine-detail sharpness.
• If your benchmark is noisy, your evaluation is measuring the wrong thing — DA-2K exists because conventional benchmarks structurally cannot reward exactly the improvements V2 makes.
• The resulting models are both more accurate on a cleaner benchmark and more than 10× faster than diffusion-based competitors, at model sizes as small as ~25M parameters — a rare case of not having to trade off speed against quality.