Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Abstract

In this work, we present methods for parsing natural language to underlying meanings, and using robotic sensors to create multi-modal models of perceptual concepts. We combine these steps towards language understanding into a holistic agent for jointly improving parsing and perception on a robotic platform through human-robot dialog. We train and evaluate this agent on Amazon Mechanical Turk, then demonstrate it on a robotic platform initialized from that conversational data. Our experiments show that improving both parsing and perception components from conversations improves communication quality and human ratings of the agent.

BibTeX Entry

@inproceedings{thomason:robodial18,
  title = {Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog},
  author = {Thomason, Jesse and Padmakumar, Aishwarya and Sinapov, Jivko and Walker, Nick and Jiang, Yuqian and Yedidsion, Harel and Hart, Justin and Stone, Peter and Mooney, Raymond J.},
  booktitle = {Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDIAL-18)},
  month = jul,
  year = {2018},
  wwwtype = {workshop}
}