Agentic RSR

Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies

Yihan Li1,2,*,† · Yating Feng1,* · Shengjiu Sun2 · Jianing Chen2 · Hao Ren2
Bowen Yang1 · Weisheng Xu1 · Qiwei Wu1 · Hui Cheng2 · Renjing Xu1,†

1 HKUST (Guangzhou) 2 Sun Yat-sen University HCL Lab — Humanoid Computing Lab

* Equal contribution (co-first authors) · † Corresponding authors

arXiv Source Code Cite Dataset
Agentic RSR cycle: real observations, robot alignment, scene modeling, physics verification, simulation policies, and experience transfer.
From a workspace video to a task-linked simulation, executable policy, and feedback-driven real-robot adaptation.

From real to simulation

White drawer · Astra / DSH
Real SIM

One task, from reconstruction to execution.

Agentic RSR connects scene reconstruction and policy development through the same manipulation task. Known robot geometry anchors metric scale; visual feedback refines the scene; MuJoCo checks task-relevant interactions. A coding agent develops policies through privileged, visual, and randomized simulation. Observations, actions, and experience then carry to the real robot through a shared task-level interface.

Visualizations

Real scenes, reconstructed simulations, and robot execution.

Matching real and simulated workspace views for Franka FR3 and AgileX Piper manipulation scenes.
Representative real and simulated views from the paper.

Method

Recover scale. Refine scenes. Learn from execution.

01

Robot alignment

VGGT-Ω depth and cameras, SAM3 masks, and URDF surface alignment establish robot-base coordinates and metric scale.

02

Scene modeling

A coding agent builds objects, materials, and lighting in Blender while robot geometry and reference cameras stay fixed.

03

Physics & policy

MuJoCo probes interactions. A policy progresses from privileged states to visual observations and randomized simulation.

Metric scale recovery from 32 video frames, VGGT-Omega reconstruction, SAM3 masks, known URDF, and ICP registration.
Known robot geometry anchors the video-estimated scene scale.
The paper's execution-grounded policy iteration loop.
Execution feedback supports continuation, retry, revision, and experience transfer.

Repository

Start from a source checkout.

The initial full-inference target is Linux, NVIDIA CUDA, and Python 3.11. Model source revisions and dependencies are pinned; gated checkpoints and task videos are supplied separately.

git clone --recurse-submodules https://github.com/HY-LiYihan/agentic_rsr.git
cd agentic_rsr
uv sync --locked --python 3.11 --extra local --extra sim --extra agent --extra dev
uv run --no-sync real2sim-models check

Current reproduction status

Offline tests and CPU registration / Blender delivery are verified. Fresh local CUDA checkpoint inference remains unverified. The public agent implementation is Codex-based; the latest DSH reconstruction batch in the paper is not reproduced by this checkout. Project code uses Apache-2.0. Historical experiment artifacts are archived in a private Hugging Face dataset requiring authorized access; asset migration and some permissions remain pending. Hardware execution requires independent safety review.

Installation & weights ↗Stage-by-stage workflow ↗Asset distribution ↗Third-party terms ↗Release preparation ↗

Citation

@misc{li2026agenticrsr,
  title={Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies},
  author={Li, Yihan and Feng, Yating and Sun, Shengjiu and Chen, Jianing and Ren, Hao and Yang, Bowen and Xu, Weisheng and Wu, Qiwei and Cheng, Hui and Xu, Renjing},
  year={2026},
  eprint={2610.10479},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  doi={10.48550/arXiv.2610.10479},
  url={https://arxiv.org/abs/2610.10479}
}