Coordinated Interplay Between Seeing and Acting

This research theme integrates perception with robot motion. Robots use perceptual inputs to refine their actions and vice versa. Research highlights frameworks that allow robots to adapt behaviours based on changes in their perceived environment, ensuring stability and theoretical guarantees. Innovations include using robot motion to simultaneously represent the environment and calibrate the camera, enhancing spatial accuracy without manual intervention. Other advancements involve enabling robots to visually understand held objects through motion and incorporating them into motion planning as tools. These integrated systems close the loop between perception and action, enabling robots to interact dynamically and safely with their surroundings, as well as to better understand their environment from motion. Ultimately, this synergy empowers robots to operate seamlessly in unstructured, real-world settings.

Selected Papers: 

Diffeomorphic Transforms for Generalised Imitation Learning. W. Zhi, T. Lai, L. Ott, F. Ramos. L4DC 2022, Best Paper Award.

3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects. W. Zhi, H. Tang, T. Zhang, M. Johnson-Roberson. IEEE RA-L, 2024 and ICRA 2025.

Unifying Representation and Calibration with 3D Foundation Models. W. Zhi, H. Tang,  T. Zhang, M. Johnson-Roberson. IEEE RA-L 2024 and ICRA 2025.