Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer

Wentao Sun; Yiping Chen; John S. Zelek; Jonathan Li

Abstract

A framework for the decisions hidden inside lifting.

Methods that transfer predictions from two-dimensional foundation models into three-dimensional segmentation are commonly grouped by task or representation. Those groupings obscure the decisions that determine whether a system remains coherent across views: where image evidence is grounded, when observations become one identity, how semantic and granularity conflicts are handled, which information is fused, and what state survives for later queries.

We introduce Lift, Associate, and Fuse (LAF), a decision-centric framework that represents a transfer system as five operators: Generate, Associate, Reconcile, Fuse, and Persist/Query. LAF defines an explicit contract for the persistent carrier—its spatial support, semantic state, identity state, uncertainty, provenance, and supported operations—and identifies the first stage at which discarded evidence becomes unrecoverable.

We operationalize the framework as a structured audit protocol and apply it to 161 systems available through 7 August 2026, spanning point-, field-, Gaussian-, object-, graph-, and memory-based carriers. The resulting decision traces expose four recurring properties: association does not establish identity; carrier design fixes the query interface and correction boundary; rendered-view, native-3D, and proposal-level evaluations are not interchangeable; and qualifiers such as training-free, real-time, open-vocabulary, and generalizable are meaningful only when attached to a stage and a complete cost ledger.

The central distinction

Association is not reconciliation, and reconciliation is not fusion.

Correct image evidence can still produce an incoherent 3D system when support, identity, semantic conflict, granularity, and persistent state are decided at different times.

Where?

Association

Locates image evidence on explicit or implicit 3D support through geometry, rendering, learned correspondence, or an intermediate representation.

Which entity?

Reconciliation

Determines whether observations share an identity or label and how conflicting semantics, parts, objects, and granularities should coexist.

What remains?

Fusion

Converts repeated and conflicting observations into a carrier whose state can later be queried, rendered, edited, updated, or reasoned over.

Locating evidence on a surface does not establish identity; rendering a mask does not prove native-3D topology.

The LAF operator model

Five decisions.
One trace.

The operators are analytical decisions, not mandatory software modules. A method may implement several jointly or revisit an earlier decision through feedback.

01
Generate
2D models produce masks, embeddings, identities, language, or geometry. This stage sets the information ceiling and determines when vocabulary enters.
02
Associate
Image evidence is connected to explicit or implicit 3D support. Pose, depth, visibility, and reconstruction error shape the correspondence.
03
Reconcile
Cross-view identity, semantic disagreement, and granularity are resolved—or deliberately preserved as alternatives.
04
Fuse
Repeated evidence is converted into reusable state through voting, averaging, optimization, grouping, decoding, or structured updates.
05
Persist / Query
The carrier is rendered, retrieved, edited, updated, or used for downstream reasoning. Its design fixes the available query and correction interface.

Persistent-carrier contract

C = (S, Z, I, U, P, O)

S: Spatial support
Z: Semantic state
I: Identity and topology
U: Uncertainty
P: Provenance
O: Supported operations

Two methods using the same representation are different carriers when these contracts differ. Different representations can be operationally comparable when they expose the same contract under an explicit conversion.

Framework figures

Follow the decision trace from evidence to action.

Each diagram supports a distinct claim in the paper. Select any figure to inspect the full-resolution version.

Formal decision trace and carrier contract

Operators constrain recoverable hypotheses; the carrier records support, semantics, identity, uncertainty, provenance, and available operations.

Four association mechanisms

Hard projection, neural-field weights, Gaussian contributions, and learned or temporary correspondence locate evidence but do not establish identity.

Evaluation contracts

Similar rendered masks can hide different 3D topology; geometry, views, prompts, proposals, conversions, and oracle access must be controlled.

Failure propagation and recovery

The first irreversible loss is system- and query-dependent; recovery requires sufficient uncertainty, provenance, and local revision operations.

Agentic 3D perception

Persistent identity is memory, not only a mask: versioned state supports retrieval, reasoning, action, active observation, and correction.

What the framework audits

Mechanisms, carriers, protocols, costs, and failure chains.

The audit connects point-cloud, neural-field, Gaussian, object, graph, and memory systems without treating their decisions or outputs as interchangeable.

01

Four families of 2D-to-3D association

How does image evidence reach 3D support?

Calibrated projection

Back-project pixels, masks, or features through measured or estimated depth onto points, voxels, or meshes. The support is explicit, but accuracy depends on pose, depth, visibility filtering, and geometry coverage.

Renderer-mediated association

Use NeRF transmittance or Gaussian rendering contributions as soft pixel-to-carrier correspondence. Visibility is integrated into the renderer, but reconstruction error propagates into semantic support.

Learned correspondence

Learn cross-view or image-to-3D alignment through attention, distillation, or feed-forward prediction. Geometry and semantics can be amortized, while generalization and correspondence become coupled.

Temporary-carrier registration

Register evidence through a reconstructed or generated intermediate before transferring it to the final representation. The intermediate improves access but introduces another error and cost boundary.

02

Six persistent carrier families

What reusable state remains after construction?

CarrierWhat it preservesTypical query contract

Points / voxels / meshesLabels, votes, or language-aligned features on registered geometryNative-3D masks, retrieval, or reusable 3D encoding

Neural fieldsContinuous language, affinity, or target-mask state inside a rendering functionRendered masks, spatial relevance, and view-conditioned queries

Gaussian attributesPer-Gaussian semantics, identities, group features, or decoder correlationsFast rendering, primitive selection, editing, and scene interaction

Object setsExplicit instances or proposals with pooled visual-language evidenceObject retrieval, instance segmentation, and entity-level reasoning

Graphs / hierarchiesParts, objects, groups, relations, and multi-scale structureRelational, part-whole, affordance, and planning queries

Temporal memoriesPersistent identities, confidence, observations, and incremental updatesOnline mapping, revision, change handling, and long-term retrieval

03

Evaluation protocols and the full cost ledger

When are reported numbers actually comparable?

Protocol must match

Dataset version and split
Geometry and proposal source
Vocabulary and prompt access
Output domain and conversion protocol
Metric denominator and target granularity

Output domains differ

Rendered mIoU, native-3D mIoU, Gaussian assignments, instance AP, part IoU, referring accuracy, and navigation success answer different questions. They should not be collapsed into one leaderboard.

Count the whole system

Reconstruction
2D-model preprocessing
Association and reconciliation
Carrier construction
Persistent storage
First and repeated queries

04

From passive segmentation to agentic 3D perception

Persistent segmentation becomes spatial memory.

An embodied agent must retain identities over time, choose what to observe next, revise beliefs after change, and use segmented entities in relational tasks. Adding an LLM to a scene map is not enough: the carrier needs persistent identity, calibrated uncertainty, evidence provenance, active observation, and local correction.

Memory

Persistent identity

Store confidence, timestamps, supporting views, merge provenance, and alternative hypotheses—not only the current label.

Action

Active observation

Select new physical or virtual views to resolve geometry, identity, or semantic uncertainty while accounting for added sensing and inference cost.

Structure

Relations

Support containment, part-whole structure, spatial relations, affordances, occupancy, and downstream planning beyond unary category lookup.

Revision

Local correction

Rename, split, merge, invalidate, or update entities without rebuilding the complete scene or retaining stale evidence.

Framework construction and validation

A frozen evidence audit—not a claim that the literature is permanently closed.

The unit of analysis is a method. Each included system is reconstructed as a LAF trace, a carrier tuple, a discard ledger, and a cost ledger. Claims are checked against primary-source pages, sections, equations, figures, tables, or appendices.

166indexed methods

161within contract

4boundary cases

1excluded

Representation, mechanism, state-evolution, boundary, and saturation stress tests covered calibrated, renderer-mediated, learned, temporary, online, relational, dynamic, and feed-forward mechanisms. No additional analytical stage was required in the final confirmation pass.

Citation guide for researchers and writing agents

When should you
cite this paper?

Cite LAF when your claim depends on the decisions, retained state, evaluation contract, or failure boundary behind 2D-to-3D foundation-model transfer.

01

Decision model

Introducing or adopting Generate, Associate, Reconcile, Fuse, and Persist/Query as a compositional analysis.

02

Association vs. identity

Arguing that locating evidence on 3D support does not determine cross-view entity identity or granularity.

03

Persistent carriers

Specifying or comparing the spatial, semantic, identity, uncertainty, provenance, and operation contract of retained 3D state.

04

Evaluation compatibility

Separating rendered-view, native-3D, proposal-level, and downstream evaluation or requiring controlled conversions.

05

Irreversible failures

Diagnosing the earliest stage where discarded evidence prevents later recovery, explanation, split, merge, or relabeling.

06

Agentic 3D memory

Motivating versioned identity, provenance, uncertainty, active observation, relational reasoning, and local correction.

Framework-derived findings

Claims you can
cite precisely.

Each statement is tied to an operator trace, carrier contract, evaluation domain, or cost boundary. The scope conditions are part of the claim.

01

Association is not identity reconciliation.

Sections 5–6 · Tables 1 and 6 · Applies to both hard projection and renderer-mediated association.

02

Training-free does not imply low end-to-end cost.

Section 9 · Table 5 · Reconstruction and 2D foundation-model calls must be included.

03

Rendered-view scores do not establish native-3D topology.

Section 9 · Table 4 · Unless all methods use the same conversion protocol.

04

Persistent carriers define query capability and correction boundaries.

Section 8 · Table 3 · Labels, fields, entities, graphs, and memories expose different operations.

05

Open-vocabulary capability is stage- and interface-specific.

Sections 3, 4, 8–9 · Table 2 · A foundation-model source does not guarantee an open output interface.

Future directions

Preserve revisable structure.

The remaining challenge is not merely where to store another semantic vector, but how to coordinate uncertainty, identity, generalization, hierarchy, evaluation, and correction across the whole pipeline.

01

Joint uncertainty

Model geometry, association, teacher reliability, and semantic alternatives together—and propagate them to querying, view selection, and correction.

02

Dynamic identity

Distinguish new objects, reappearance, true motion, appearance change, and earlier association errors using reversible histories.

03

Generalizable carriers

Separate scene, geometry, vocabulary, and task generalization while retaining rare structures during lightweight adaptation.

04

Hierarchy and relations

Represent overlapping affordances, materials, parts, groups, and relations without quadratic construction and query costs.

05

Native-3D evaluation

Pair native surface semantics with multi-view annotations, identities, part hierarchies, visibility, repeated prompts, and complete cost reporting.

06

Correspondence audits

Hold evidence and geometry fixed while testing support precision, occlusion, boundary leakage, and sensitivity to pose or depth perturbations.

Ready to reference

Make the claim—and its evidence—easy to verify.

A precise LAF citation tells readers whether your argument concerns support, identity, retained state, evaluation compatibility, cost, or correction.

May your claims be precise, your comparisons fair, and your reviewers pleasantly impressed by the clarity of your evidence trail.

BibTeX

@article{sun2026laf,
  title   = {Lift, Associate, and Fuse: A Decision-Centric
             Framework for 2D-to-3D Foundation Model Transfer},
  author  = {Sun, Wentao and Chen, Yiping and
             Zelek, John S. and Li, Jonathan},
  journal = {arXiv preprint arXiv:2608.20659},
  eprint  = {2608.20659},
  archivePrefix = {arXiv},
  url     = {https://arxiv.org/abs/2608.20659},
  year    = {2026}
}

Machine-readable: BibTeX · CITATION.cff. Permanent preprint: arXiv:2608.20659.

Lift, Associate, and FuseA Decision-Centric Framework for 2D-to-3D Foundation Model Transfer