Whether they are exploring the deepest depths of the ocean or responding to a disaster, robots have proven tremendously effective as our surrogates, performing tasks that are either too difficult, dangerous, or dull for humans. The next generation of intelligent systems will cooperate with people in our homes and workplaces, providing personalized care, assisting the disabled, and carrying out advanced manufacturing. To be effective partners, robots must reason about their environment and their actions in the same way that humans do. However, today's robots use representations that are either hard-coded or require significant supervision by a domain expert. I seek to enable robots to efficiently learn shared cognitive models of their surroundings and available actions from their interaction with humans.

This talk highlights my recent advances in semantic perception that enable robots to acquire shared cognitive models of objects and of their environment from limited supervision provided by human partners. First, I will describe a visual appearance-based algorithm that efficiently learns a robust representation of objects from a single, user-provided segmentation cue. Second, I will provide a probabilistic framework that allows robots to formulate human-centric models of their environment from natural language descriptions. I will then demonstrate how these learned representations allow people to command and interact with robots using free-form speech. Finally, I will end with my vision for how robots will formulate hierarchical cognitive models of their environments, the objects they contain, and the rich space of actions available to the robot.

Representative publication: http://people.csail.mit.edu/mwalter/papers/walter13.pdf

Related Video: http://vimeo.com/67438012