CS Events

Qualifying Exam

Self-Supervised Object Understanding for Robot Perception and Manipulation


Download as iCal file

Monday, November 28, 2022, 10:30am - 12:00pm


Speaker: Shiyang Lu

Location : 1 Spring Street, Room 204


Prof. Kostas Bekris (advisor)

Prof. Abdeslam Boularias

Prof. Kristin Dana

Prof. Sepehr Assadi

Event Type: Qualifying Exam

Abstract: : To be deployed in indoor human environments, autonomous robots should be able to reason about unknown objects so as to manipulate them safely and efficiently. It is, however, challenging because household objects can be arbitrary with various textures and geometries. And it is not feasible to manually annotate or train a model for every single object or even at the category level. We aim to let robots understand objects from their own experiences, either through passively observing objects from different viewpoints and locations, or actively manipulating them. In particular, we designed algorithms to automatically find patterns of rigid objects across different scenes even in clutter, and use the pseudo labels to train a segmentation model via contrastive learning. We also built a robotic system to manipulate objects without access to mesh models. The target object is tracked and reconstructed during the manipulation process. Its texture information and reconstructed model are saved in a memory bank to speed up future manipulation of the same object. Through a lifelong process, the robot becomes more capable over time as it observes and manipulates more objects. In the future, we aim to extend the current research to allow robots to robustly handle objects in more complex scenes.


can also join online using the following zoom link: https://rutgers.zoom.us/j/94029006813?pwd=OEtzUzJxMHI1ak83dVFHL2JwNEhqdz09&from=addon