Skip to content Skip to navigation

Learning to Manipulate Unknown Objects


Summary of the project:

Outdoor robots, such as search-and-rescue robots and planetary rovers, often need to grasp and push objects such as debris and rocks that have irregular shapes. While most object manipulation techniques require mechanical or geometric models of the objects, objects that are encountered in outdoor environments do not typically have any known models.

To grasp unknown novel objects, data-driven approaches are becoming increasingly popular. These approaches learn from examples statistical models that predict the success of grasping or pushing actions. The biggest drawback of statistical models is perhaps their inherent inaccuracies, the predicted values are always subject to an error unless the objects used during testing are identical to some of the objects used for training.

To solve these issues, we explore in this project two self-supervisory learning techniques that allow the robot to adapt to new objects by correcting online the predicted values. Using these techniques, we show how a robot can efficiently learn on the fly and in real-time to push and grasp novel objects. We test both techniques on the task of autonomously clearing piles of natural and man-made objects. 


  • The first technique is used for clearing piles wherein grasping actions alone are enough for clearing the pile. The outcomes of the grasping actions are modeled as a Gaussian Process, and an entropy-guided method is used in order to learn where the best grasp is most likely to be found.
  • The second technique is used for tight piles wherein grasping actions alone are not effective, the robot first needs to push obstacles away in a particular way that helps grasping by creating empty space around them to insert the robot’s fingers. We use a reinforcement learning approach that we use for selecting the best sequence of pushing and grasping actions to execute in order to clear a given pile.

In this project, we also develop a perception technique that is used, along with the online adaptation techniques, to build a fully autonomous system. 

Relevent publications: