CS Events

Computer Science Department Colloquium

Towards Closing the Perception-Planning and Sim2Real Gaps in Robotics


Download as iCal file

Monday, September 19, 2022, 10:30am


Speaker: Kostas Bekris


Kostas Bekris is an Associate Professor of Computer Science at Rutgers University in New Jersey. He is working in algorithmic robotics, where his group is developing algorithms for robot planning, learning and perception especially for robot manipulation and multi-robot problems. Applications include logistics and manufacturing with a focus on taking advantage of novel soft, adaptive mechanisms. His research has been supported by NSF, DHS, DOD and NASA, including a NASA Early Career Faculty award. He received his Ph.D in Computer Science from Rice University under the guidance of Prof. Lydia Kavraki. Kostas is Program Chair for the “Robotics: Science and Systems” (RSS) 2023 conference.

Location : CoRE 301 + Virtual

Event Type: Computer Science Department Colloquium

Abstract: Robotics is at the point where we can deploy complete systems across applications, such as logistics, service and field robotics. There are still critical gaps, however, that limit the adaptability, robustness and safety of robots, which lie at: (a) the interface of domains, such as perception, planning and learning, that must be viewed holistically in robotics, and (b) the sim2real gap, i.e., the deviation between internal models of robots’ AI and the real world. This talk will first describe efforts in tighter integration of perception and planning for vision-driven robot manipulation. We have developed high-fidelity, high-frequency tracking of rigid bodies’ 6D poses - without using CAD models or cumbersome human annotations - by utilizing progress both in deep learning and pose graph optimization. These solutions together with appropriate shared representations, tighter closed-loop operation and compliant mechanisms are unblocking the deployment of full-stack robot manipulation systems. This talk will provide examples of robust robotic packing, assembly under tight tolerances as well as constrained placement given a single demonstration that generalizes across an object category. The talk’s second part is motivated by tensegrity robots, which combine rigid and soft elements, to achieve safety and adaptability. They also complicate, however, modeling and control given their high-dimensionality and complex dynamics. This sim2real gap of analytical models has motivated us to look into reinforcement learning (RL) for controlling robot tensegrities, which allowed the development of new skills for them. RL applicability is limited, however, due to its high data requirements. Training RL in simulation is promising but is blocked again by the sim2real gap. For this reason, we are developing differential engines for tensegrity robots that reason about first-principles so as to be trained with few example trajectories from the real robot. They provide accurate-enough simulations to train a controller that is directly transferrable back to the real system. We report our first success in such a real2sim2real transfer for a 3-bar tensegrity robot. The talk will conclude with a brief discussion on how closing these gaps empowers the next step of developing robots that are socially cognizant and can be safely integrated into our society.


Department of Computer Science

School of Arts & Sciences

Rutgers University

Contact  Matthew Stone

Zoom Link: https://rutgers.zoom.us/j/92479630192?pwd=Q29nQklLNXJzaHlpOUVlaDNOd0hjdz09&from=addon