Events

Qualifying Exam Autoregressive Action Sequence Learning for Robotic Manipulation
Export Event Save iCal Send to Google Calendar Send to Yahoo Calendar Send to Outlook Live Send to Microsoft Outlook Save iCal Preserve formatting in description (only supported in some calendar applications) Close Wednesday, March 26, 2025, 02:00pm - 03:00pm

Speaker:  Xinyu Zhang

Bio

Location : Room 402, 4th floor, 1 Spring Street, Downtown New Brunswick

Committee: 

Professor Abdeslam Boularias

Professor Kostas Bekris

Professor He Zhu

Professor Karthik C. S.

Event Type: Qualifying Exam

Abstract: A key challenge remains to design a universal policy architecture that performs well across diverse robots and task configurations. In this work, we address this by representing robot actions as sequential data and generating actions through autoregressive sequence modeling. Existing autoregressive architectures generate end-effector waypoints sequentially as word tokens in language modeling, which are limited to low-frequency control tasks. Unlike language, robot actions are heterogeneous and often include high-frequency continuous values---such as joint positions, 2D pixel coordinates, and end-effector poses—which are not easily suited for language-based modeling. Based on this insight, we extend causal transformers' single-token prediction to support predicting a variable number of tokens in a single step through our Chunking Causal Transformer (CCT). This enhancement enables robust performance across diverse tasks of various control frequencies, greater efficiency by having fewer autoregression steps, and lead to a hybrid action sequence design by mixing different types of actions and using a different chunk size for each action type. Based on CCT, we propose the Autoregressive Policy (ARP) architecture, which solves manipulation tasks by generating hybrid action sequences. We evaluate ARP across diverse robotic manipulation environments, including Push-T, ALOHA, and RLBench, and show that ARP, as a universal architecture, matches or outperforms the environment-specific state-of-the-art in all tested benchmarks, while being more efficient in computation and parameter sizes.List of Publications:Autoregressive action sequence learning for robotic manipulation. (RA-L 2025). Xinyu Zhang, Yuhan Liu, Haonan Chang, Liam Schramm, Abdeslam Boularias

Organization: 

Contact  Professor Abdeslam Boularias

https://rutgers.zoom.us/my/xz653?pwd=ckZKVDVOWkY3Tm9DZzBCNTA1b3hhQT09