Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Newer version: thomason2020improving.

J. Thomason et al., “Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog,” in Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue, July 2018.

Abstract

In this work, we present methods for parsing natural language to underlying meanings, and using robotic sensors to create multi-modal models of perceptual concepts. We combine these steps towards language understanding into a holistic agent for jointly improving parsing and perception on a robotic platform through human-robot dialog. We train and evaluate this agent on Amazon Mechanical Turk, then demonstrate it on a robotic platform initialized from that conversational data. Our experiments show that improving both parsing and perception components from conversations improves communication quality and human ratings of the agent.

BibTeX Entry

@inproceedings{thomason2018robodial,
  title = {Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog},
  author = {Thomason, Jesse and Padmakumar, Aishwarya and Sinapov, Jivko and Walker, Nick and Jiang, Yuqian and Yedidsion, Harel and Hart, Justin and Stone, Peter and Mooney, Raymond J.},
  booktitle = {Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue},
  month = jul,
  year = {2018},
  wwwnewer = {thomason2020improving},
  wwwhidden = {true},
  wwwvideo = {https://www.superlectures.com/sigdial2018/jointly-improving-parsing-and-perception-for-natural-language-commands-through-human-robot-dialog},
  wwwpdf = {https://jessethomason.com/publication_supplements/SIGDIAL18_late_breaking.pdf}
}