Human-to-Robot Imitation with Symbolic Planning in a Unified Latent Space

Direct imitation of humans by robots offers a promising direction for remote teleoperation and intuitive task instruction, where a human can perform a task naturally and the robot autonomously interprets and executes it using its own embodiment. Existing methods often rely on close alignment between human and robot scenes. This prevents robots from inferring the intent of the task or executing demonstrated behaviors when the initial states mismatch. Hence, it poses difficulties for non-expert users, who may need domain knowledge to adjust the setup. To address this challenge, we propose a neuro-symbolic framework that unifies visual observations, robot proprioceptive states, and symbolic abstractions within a shared latent space. Human demonstrations are encoded into this representation as predicate states. A symbolic planner can thus generate high-level plans that account for the different robot initial states. A flow matching module then synthesizes continuous joint trajectories consistent with the symbolic plan. We validate our approach on multi-object manipulation tasks. Preliminary results show that the framework can infer human intent and generate feasible symbolic plans and robot motions under mismatched initial states. These findings highlight the potential of neuro-symbolic models for more natural human-robot instruction. and they can enhance the explainability and trustworthiness of robot actions.