Preprint · 2026

CDSeg A Renderable Gaussian Carrier for Image-to-3D Label Transfer

Image models know what to segment. CDSeg determines where those labels persist in 3D—then keeps them queryable as points, native Gaussians, or newly rendered views.

Wentao Sun1 Yiping Chen2 Zhengsen Xu3 Jonathan Li1 John S. Zelek1
1 University of Waterloo 2 Sun Yat-sen University 3 University of Calgary
Read on arXiv Code · Coming soon

CDSeg uses the renderer itself as the correspondence interface between image pixels and explicit 3D primitives.

The core idea

Move the label, not the model.

Instead of training a dedicated 3D network for every segmentation task, CDSeg reuses mature image-mask sources and writes their evidence into an explicit, renderable 3D carrier.

01

Image masks

Promptable, automatic-instance, semantic, or detector masks.

02

Gaussian carrier

Renderer-derived visibility associates pixels with explicit 3D primitives.

03

Persistent 3D labels

Return point indices, retain a labeled scene, or render labels to new views.

∅

No task-specific 3D training

The external image model decides what is labeled; CDSeg handles where the evidence is stored in 3D.

Ai,v,p

Visibility from rendering

Pixel–primitive associations follow projected support, depth order, and accumulated transmittance.

2D↔3D

One reusable interface

The same fusion logic connects ordinary point sets, optimized Gaussian scenes, and calibrated image views.

Two carrier modes

Use the 3D representation you already have.

Both modes share the same renderer-derived association, multi-view voting, and local filtering. They differ only in how the carrier is instantiated and where the final labels live.

Mode I Point-indexed output

Point-completed Gaussians

Complete every input point with Gaussian parameters while preserving its original index. No scene optimization, densification, or pruning is required.

  • One point → one Gaussian
  • Exact return to input point order
  • Designed for point clouds and LiDAR
pi = (x, y, z, r, g, b) → gi = (μi, Σi, αi, shi)
Mode II Renderable scene output

Native optimized Gaussians

Reuse the primitives of an existing Gaussian scene. Fused labels remain attached to the same representation used for rendering.

  • No point-to-scene conversion
  • Persistent labels on native primitives
  • Labels render directly to target views
Masksv → visibility + voting → Labels3D
External mask source CDSeg carrier interface Points · Gaussians · Views
Results across representations

One interface, four segmentation settings.

The experiments test what the interface enables: prompt-driven labeling, persistent scene-level instances, exact benchmark point indices, and transfer to sparse LiDAR geometry.

92.35% mIoU DesktopObjects-360

Promptable object segmentation

95.89% mIoU NeRDS-360

Automatic vehicle instances

65.77% mIoU ScanNet-v2

Full validation split

57.44% mIoU KITTI-360

Stationary-vehicle transfer

01 / Promptable

A single prompt becomes a persistent 3D selection.

SAM2 masks from ordered views are fused onto an optimized Gaussian scene, where the selected object remains available beyond the original camera.

02 / Automatic instances

View-wise detections become scene-level identities.

Tracked image masks are consolidated on native Gaussians across outdoor viewpoints.

03 / Semantics

Image annotations return to exact benchmark vertices.

Mode I preserves the ScanNet point index while lifting controlled semantic masks.

04 / Sparse LiDAR

Calibrated image masks write labels into driving geometry.

Point-completed Gaussians carry stationary-vehicle labels into accumulated KITTI-360 LiDAR points without reconstructing a separate scene.

3D labels remain renderable

Write once in 3D.
Read from another view.

Because every camera queries the same completed 3D labeling, each physical object retains its instance identity as projected shape, scale, and visibility change.

Consistent IDs across views Renderer-aligned occlusion No second 2D segmentation pass
Citation

Build on CDSeg.

If this work helps your research, please cite the preprint. The entry can be updated with the arXiv identifier after publication.

View on arXiv
BibTeX 
@article{sun2026cdseg,
  title   = {CDSeg: A Renderable Gaussian Carrier for
             Image-to-3D Label Transfer},
  author  = {Sun, Wentao and Chen, Yiping and Xu, Zhengsen
             and Li, Jonathan and Zelek, John S.},
  journal = {arXiv preprint arXiv:2608.05482},
  year    = {2026}
}