OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport

Abstract

Transferring human motion to humanoid robots requires adapting the demonstrated motion to the robot morphology while preserving interactions with the environment. This is particularly challenging for loco-manipulation tasks, where contacts with the ground and manipulated objects must remain consistent despite differences in body proportions. Yet, skeletal motion alone does not fully describe these interactions, and fixing object trajectories limits the adaptation to a new embodiment. In this paper, we introduce OTRetarget, a unified approach to jointly retarget robot and multi-object motion from human demonstrations. Our approach represents surface interactions through signed distances, closest surface points, and relative directions, and uses entropic optimal transport to transfer these quantities across human, robot, and object geometries. We incorporate the resulting interaction targets into a constrained inverse kinematics formulation that balances contact preservation with motion style and jointly optimizes robot and object poses at each frame. This formulation accommodates robot–object and object–object interactions without rescaling the scene or the demonstration. We validate the proposed approach on OMOMO, where it achieves a robot–object interaction Jaccard score of 87% and a depth error of 8.7 mm, compared with 28% and 29.3 mm for OmniRetarget. Finally, we demonstrate transfer to a physical G1 humanoid using whole-body policies trained with reinforcement learning on the retargeted references, across motions including two-handed box pick-and-place onto a table.

How it works

Six stages, left to right: human motions, proximity triples, optimal transport,
optional data augmentation, per-frame QP solve, RL training and sim-to-real.

One demonstration, six stages. A capture enters as meshes. Every probe on one surface reads its distance to the other, the point it reads it at, and the direction between them. A single transport plan, baked once in a T-pose, carries those readings onto the robot. A per-frame constrained program then arbitrates them against the demonstrated style and solves the robot pose and every object pose in one go. The dashed stage is optional: it swaps the object for another one.

Capture. Human motion and object poses are recovered from a single demonstration. Surfaces, not skeletons, carry the interaction.

Proximity triples. Every surface sample reads a signed distance, its closest witness point on the other surface, and the direction to it.

Optimal transport. One entropic plan per human part and robot link, solved once in a T-pose, carries those targets across morphologies. Every human sample receives mass, so fingertips and heels are not left behind.

The solve, from the inside. The same pick-and-place with the residuals drawn on it: yellow the target field carried over by transport, blue that field re-evaluated at the pose the solver returns. Feet on the floor, hands onto the box, the box onto the tabletop.

Results

The baseline never closes a hand on the pole. Same OMOMO sequence, same world. Ours grips the shaft with both hands and holds it through the reach. OmniRetarget leaves the hand open in the air beside the shaft. It recovers none of the demonstrated contact.

Hands and knees on the floor. A CMU sequence, one fixed camera for all four panels. OTRetarget lays both hands flat on the ground, as the demonstration does. OmniRetarget reaches the ground with the wrist bent. GMR represents no terrain and leaves the lead hand hovering. PHC drives the forearm and the supporting foot through the floor.

The object is solved, not imposed. The robot pose and every object pose are solved together, so the whole contact structure survives the change of body: the hands on the object, the object on the ground and on the other objects. Fixed to the demonstrated trajectory, the box ends up above the robot's own head and the robot chases it off the floor. Solved jointly, it stays within reach and the feet stay down.

One demonstration, a distribution of objects

The same demonstration is replayed onto unseen shapes, from a ball to a beam, by transporting the demonstrated witness points onto the substitute surface. The grip holds on shapes the capture never touched. A substitute shape does not sit where the demonstrated one did, so it is free to leave the demonstrated path and the rest of the scene solves with it: the new object still clears the floor and comes to rest on the tabletop, and the hands stay on its surface. Replaying the demonstrated poses would hold the new shape on a path meant for a different geometry.

Sim-to-real

From the solve to the policy. Left, the kinematic reference our solver returns. Right, a policy learning to track it in simulation, across many instances of the scene at once.

Pick-and-place on the robot. The same policy, unchanged and still blind to the object, completes the task on the physical G1. Inset, the human demonstration it all started from.

Locomotion. Tracking policies trained on OTRetarget references and deployed on the same robot: spin, crawl and cha-cha. Each is shown next to the reference it tracks.

BibTeX

@misc{besset2026otretarget, title = {{OTRetarget}: Joint Robot and Object Motion Retargeting via Optimal Transport}, author = {Besset, Guillaume and Carn, Erwann and Carecchio, Timoth\'ee and Tordjman--Levavasseur, Valentin and Schramm, Fabian and de Mont-Marin, Yann and Carpentier, Justin and Sathya, Ajay Suresha}, year = {2026}, eprint = {XXXX.XXXXX}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/XXXX.XXXXX} }