Aligning Robot Representations with Humans

Workshop CoRL 2022 - December 15th (Hybrid)

In person location: ENG Building 405 Room 470 (405.470)

In person poster session: 4th floor of ENG 405; Gather.Town session: Link

Submit your questions in the Pheedloop chat

Robots deployed in the real world will interact with many different humans to perform many different tasks in their lifetime, which makes it difficult (perhaps even impossible) for designers to specify all the aspects that might matter ahead of time. Instead, robots can extract these aspects implicitly when they learn to perform new tasks from their users' input. The challenge is that this often results in representations which pick up on spurious correlations in the data and fail to capture the human’s representation of what matters for the task, resulting in behaviors that do not generalize to new scenarios. In this workshop, we are interested in exploring ways in which robots can align their representations with those of the humans they interact with so that they can more effectively learn from their input. By bringing together experts from representation learning, human-robot interaction, and cognitive science, we believe we can foster an environment where we can exchange ideas for how the robot learning community can best benefit from learning representations from human input and vice-versa, and how the HRI community can best direct their efforts towards discovering more effective human-robot teaching strategies. We encourage participation from researchers working in robot learning, human-robot interaction, cognitive science, and representation learning. The workshop will adopt a hybrid format, including in-person presenations, live streams, and a hybrid poster session. NVIDIA is sponsoring two Best Paper Awards!

Speakers and Panelists

Jacob Andreas

Massachusetts Institute of Technology

Daniel S. Brown

University of Utah

Matthew Gombolay

Georgia Institute of Technology

Mark Ho

Princeton University

George Konidaris

Brown University

Lerrel Pinto

New York University

Dorsa Sadigh

Stanford University


Time (NZDT)
08:30 am - 08:45 am Organizers
Introductory Remarks
08:45 am - 09:15 am Mark Ho
Artificial intelligence, natural stupidity, and resource rational cognition
09:15 am - 09:55 am Jacob Andreas
Toward natural language supervision
09:45 am - 10:00 am Coffee Break
10:00 am - 10:30 am Lerrel Pinto
Teaching Robots to Manipulate in an Hour
10:30 am - 11:00 am Matthew Gombolay
Confronting the Correspondence Problem with Self-supervised and Interactive Machine Learning
11:00 am - 12:00 pm Contributed Talks
12:00 pm - 12:30 pm Daniel Brown
Latent Spaces and Learned Representation for Better Human Preference Learning
12:30 pm - 01:30 pm Lunch Break
01:30 pm - 02:00 pm Coffee Break
02:00 pm - 02:30 pm Conference Opening Session
02:30 pm - 03:00 pm Amy Zhang
Attending to What Matters with Representation Learning
03:00 pm - 03:30 pm Dorsa Sadigh
Aligning Humans and Robots : Active Elicitation of Informative and Compatible Queries
03:30 pm - 04:00 pm George Konidaris
Reintegrating AI: Skills, Symbols, and the Sensorimotor Dilemma
04:00 pm - 05:00 pm Panel Session
05:00 pm - 05:10 pm Organizers
Concluding Remarks
05:10 pm - 06:00 pm In person: 4th floor of ENG 405; Virtual: On Gather.Town
Poster Session


Congratulations to Abhijat Biswas (Mitigating causal confusion in driving agents via gaze supervision) and Ruohan Zhang (A Dual Representation Framework for Robot Learning with Human Guidance) for each winning a Best Paper Award!

  • Mitigating causal confusion in driving agents via gaze supervision [link] (spotlight)
    Abhijat Biswas; Badal Arun Pardhi; Caleb Chuck; Jarrett Holtz; Scott Niekum; Henny Admoni; Alessandro Allievi
  • Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased [link]
    Chao Yu; Jiaxuan Gao; Weilin Liu; Botian Xu; Hao Tang; Jiaqi Yang; Yu Wang; Yi Wu
  • Spatial Generalization of Visual Imitation Learning with Position-Invariant Regularization [link]
    Zhao-Heng Yin; Yang Gao; Qifeng Chen
  • Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training [link]
    Yecheng Ma; Shagun Sodhani; Dinesh Jayaraman; Osbert Bastani; Vikash Kumar; Amy Zhang
  • Do you see what I see? Using questions and answers to align representations of robotic actions [link]
    Chad DeChant; Iretiayo Akinola; Daniel Bauer
  • A Sequential Group VAE for Robot Learning of Haptic Representations [link]
    Ben Richardson; Katherine J. Kuchenbecker; Georg Martius
  • A Dual Representation Framework for Robot Learning with Human Guidance [link] (spotlight)
    Ruohan Zhang; Dhruva Bansal; Yilun Hao; Ayano Hiranaka; Jialu Gao; Chen Wang; Roberto Martín-Martín; Li Fei-Fei; Jiajun Wu
  • Learning Abstract Representations of Agent-Environment Interactions [link]
    Tanmay Shankar; Jean Oh
  • Learning Visualization Policies of Augmented Reality for Human-Robot Collaboration [link]
    Kishan Chandan; Jack Albertson; Shiqi Zhang
  • A Graph Neural Network Approach for Choosing Robot Addressees in Group Human-Robot Interactions [link]
    Sarah Gillet; Iolanda Leite; Marynel Vázquez
  • Graph Inverse Reinforcement Learning from Diverse Videos [link]
    Sateesh Kumar; Jonathan Zamora; Nicklas A Hansen; Rishabh Jangir; Xiaolong Wang
  • Watch and Match: Supercharging Imitation with Regularized Optimal Transport [link]
    Siddhant Haldar; Vaibhav Mathur; Denis Yarats; Lerrel Pinto

We thank the following people for their assistance in reviewing submitted papers.

  • Andrea Bajcsy
  • Arjun Sripathy
  • Daniel Brown
  • Eoin Kenny
  • Erdem Biyik
  • Felix Wang
  • Jerry He
  • Megha Srivastava
  • Micah Carroll
  • Minae Kwon
  • Nick Walker
  • Rohin Shah
  • Serena Booth
  • Xavier Puig
  • Xuning Yang
  • Yuchen Cui


Andreea Bobu

University of California Berkeley

Andi Peng

Massachusetts Institute of Technology

Pulkit Agrawal

Massachusetts Institute of Technology

Julie Shah

Massachusetts Institute of Technology

Anca Dragan

University of California Berkeley

Reach out to for any questions.