Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Abstract

In this work, we present methods for parsing natural language to underlying meanings, and using robotic sensors to create multi-modal models of perceptual concepts. We combine these steps towards language understanding into a holistic agent for jointly improving parsing and perception on a robotic platform through human-robot dialog. We train and evaluate this agent on Amazon Mechanical Turk, then demonstrate it on a robotic platform initialized from that conversational data. Our experiments show that improving both parsing and perception components from conversations improves communication quality and human ratings of the agent.

BibTeX Entry

@inproceedings{thomason2018robodial,
  title = {Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog},
  author = {Thomason, Jesse and Padmakumar, Aishwarya and Sinapov, Jivko and Walker, Nick and Jiang, Yuqian and Yedidsion, Harel and Hart, Justin and Stone, Peter and Mooney, Raymond J.},
  booktitle = {Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDIAL-18)},
  month = jul,
  year = {2018},
  wwwtype = {lbr},
  wwwhidden = {true},
  wwwvideo = {https://www.superlectures.com/sigdial2018/jointly-improving-parsing-and-perception-for-natural-language-commands-through-human-robot-dialog},
  wwwpdf = {https://jessethomason.com/publication_supplements/SIGDIAL18_late_breaking.pdf}
}