Comparison of Multimodal Heading and Pointing Gestures for Co-Located Mixed Reality Human-Robot Interaction

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018) (accepted) - oct 2018
Download the publication : iros-holoros-application.pdf [341Ko]  
Mixed reality (MR) opens up new vistas for human-robot interaction (HRI) scenarios in which a human operator can control and collaborate with co-located robots. For instance, when using a see-through head-mounteddisplay (HMD) such as the Microsoft HoloLens, the operator can see the real robots and additional virtual information can be superimposed over the real-world view to improve security, acceptability and predictability in HRI situations. Inparticular, previewing potential robot actions in-situ before they are executed has enormous potential to reduce the risks of damaging the system or injuring the human operator. In this paper, we introduce the concept and implementation of such an MR human-robot collaboration system in which a human can intuitively and naturally control a co-located industrial robot arm for pick-and-place tasks. In addition, we compared two different, multimodal HRI techniques to select the pick location on a target object using (i) head orientation (aka heading) or (ii) pointing, both in combination with speech. The results show that heading-based interaction techniques are more precise, require less time and are perceived as less physically, temporally and mentally demanding for MR-based pick-and-place scenarios. We confirmed these results in an additional usability study in a delivery-service task with a multi-robot system. The developed MR interface shows a preview of the current robot programming to the operator, e. g., pick selection or trajectory. The findings provide important implications for the design of future MR setups.

Images and movies

 

BibTex references

@InProceedings\{KSLJGZ18,
  author       = "Krupke, Dennis and Steinicke, Frank and Lubos, Paul and Jonetzko, Yannick and Görner, Michael and Zhang, Jianwei",
  title        = "Comparison of Multimodal Heading and Pointing Gestures for Co-Located Mixed Reality Human-Robot Interaction",
  booktitle    = "Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018) (accepted)",
  month        = "oct",
  year         = "2018",
  url          = "http://basilic.informatik.uni-hamburg.de/Publications/2018/KSLJGZ18"
}

Other publications in the database

» Dennis Krupke
» Frank Steinicke
» Paul Lubos
» Jianwei Zhang