Rethinking Asymmetry in
Self-Supervised 3D Geometry Learning

Gyeongjin Kang¹Freeman Cheng²Botao Ye³Weijie Lyu²
Ming-Hsuan Yang²Eunbyung Park¹

¹Yonsei University   ²UC Merced   ³ETH Zurich

Reconstructions

One forward pass over 10 unposed images (DL3DV-trained model). Points are the model's predicted depth, lifted with its predicted cameras (purple). No ground truth is used, and points inconsistent across views are removed.

Method

Where should the asymmetry live?

A self-supervised objective needs an asymmetry between what the model sees and what it must explain; otherwise trivial solutions satisfy it. Prior pose-free methods place this asymmetry in the input space. E-RayZer splits camera and geometry prediction into two backbones so that geometry sees only context views, and NAS3R masks attention so that context views attend only among themselves.

When a held-out view lies between context views, the context views already describe the same scene. Hiding the held-out view from the network hides no geometry; it only removes correspondence. We instead place the asymmetry in the 3D representation: a shared backbone lets every view inform depth and pose, while the rendered scene is built from context-view Gaussians only, so a held-out view is never rendered from its own prediction.

Where the asymmetry is placed

Different view sets, same scene

Architecture

Left: all designs build the 3D scene from context (C) views. Separate backbones (a) and masked attention (b) restrict interaction with held-out (H) views, while ours (c) allows full cross-view attention and excludes held-out views only from the 3D representation. Top right: when held-out views lie between context views (the interpolation setting), the context views depict the same scene as the full view set. Bottom right: one estimator processes context and held-out views; a DPT head predicts depth and Gaussian attributes, a pose head predicts cameras, and context-view depths are unprojected and rendered at the held-out poses.

Beyond rendering: two alignment signals

Asymmetric feature alignment. A model sees a scene more completely from many views than from a few. We turn this gap into a training signal. An EMA teacher encodes all views, the student encodes a sparse subset, and the student's features are pulled toward the teacher's with a cosine loss. Matching features rather than depth maps keeps the teacher's depth and scale errors from being copied.

Semantic guidance. The rendering loss is local and pixel-wise. We additionally align the encoder features with those of a frozen vision foundation model, which supplies structure where photometric cues are weak, such as textureless or repetitive regions.

None of the three signals uses geometric annotation.

Results

All comparisons use 10 input views, released baseline checkpoints and the baselines' evaluation protocols. Pose is scored by relative pose accuracy (RPA, %) at 3° and 5°, and depth by AbsRel and δ<1.25 after per-view median alignment. Shaded cells mark the best per column.

Trained on RealEstate10K

Camera pose and multi-view depth with 10 input views, all models trained on RE10K. SPFSplat initializes its backbone from MASt3R, which is trained with 3D supervision, and takes ground-truth intrinsics as input.

Trained on DL3DV

Camera pose and multi-view depth with 10 input views, both models trained on DL3DV. E-RayZer decodes depth only for its context views, so for this comparison our depth head also decodes only the context views and both methods are scored on the same views.

Recovered camera trajectories (interactive)

Predicted cameras from 10 unposed views (color) against ground truth (white), after one Sim(3) alignment of the camera centers (Umeyama). APE is the RMSE of the aligned camera centers.

Analyses

Controlled experiments on DL3DV: all variants share the backbone, data and compute, and are evaluated on a fixed 48-frame span. Depth is compared against MoGe-2 pseudo ground truth, so these numbers are not directly comparable to the main tables.

Full interaction vs. input-space asymmetry

Asymmetric feature alignment

Context alignment analysis on DL3DV. Aligning depth maps directly does not help in this setting, while aligning features does.

Self-supervised pretraining for supervised fine-tuning

Same architecture and data throughout, analysis protocol. SSL alone, trained for half the iterations, is comparable to SL: it is more accurate at strict pose thresholds, while SL is better on some looser metrics. Fine-tuning the SSL model with the same supervision as SL surpasses both on every reported metric with fewer total iterations than SL.

Dynamic monocular videos

We also explore training on dynamic monocular videos from SpatialVID, keeping the architecture and objective unchanged, so the model still assumes a static scene. We compare a model trained on DL3DV alone with the same model trained on DL3DV and SpatialVID, evaluating camera pose on held-out SpatialVID videos grouped by the dataset's metadata (10 views over a 32-frame span).

RPA@5 / AUC@30 (%) with 10 input views. SpatialVID test sets have low or high ratios of dynamic content, or crowds, and use a 32-frame span. DL3DV uses a 48-frame span.

Without any architectural change, the model tends to assign low opacity to Gaussians on moving objects, so they are largely absent from renderings at held-out views while the static background is reconstructed.

Camera trajectories on dynamic videos (interactive)

Predicted cameras (color) against ground truth (white) on held-out SpatialVID videos, after one Sim(3) alignment of the camera centers.

BibTeX

@article{kang2026rethinking,
  title   = {Rethinking Asymmetry in Self-Supervised 3D Geometry Learning},
  author  = {Kang, Gyeongjin and Cheng, Freeman and Ye, Botao and Lyu, Weijie and
             Yang, Ming-Hsuan and Park, Eunbyung},
  journal = {arXiv preprint},
  year    = {2026}
}