Probabilistic
Action Cores

|

We are happy to announce Version 1.0 of PRAC!

Try out online!

About

A robot that can be simply told in natural language what to do – this has been one of the ultimate long-standing goals in both Artificial Intelligence and Robotics research. In near-future applications, robotic assistants and companions will have to understand and perform commands such as “set the table for dinner”, “make pancakes for breakfast”, or “cut the pizza into 8 pieces.” Although such instructions are only vaguely formulated, complex sequences of sophisticated and accurate manipulation activities need to be carried out in order to accomplish the respective tasks. The acquisition of knowledge about how to perform these activities from huge collections of natural-language instructions from the Internet has garnered a lot of attention within the last decade. However, natural language is typically massively unspecific, incomplete, ambiguous and vague and thus requires powerful means for interpretation.

This work presents PRAC – Probabilistic Action Cores – an interpreter for natural-language instructions which is able to resolve vagueness and ambiguity in natural language and infer missing information pieces that are required to render an instruction executable by a robot. To this end, PRAC formulates the problem of instruction interpretation as a reasoning problem in first-order probabilistic knowledge bases. In particular, the system uses Markov logic networks as a carrier formalism for encoding uncertain knowledge. A novel framework for reasoning about unmodeled symbolic concepts is introduced, which incorporates ontological knowledge from taxonomies and exploits semantically similar relational structures in a domain of discourse. The resulting reasoning framework thus enables more compact representations of knowledge and exhibits strong generalization performance when being learnt from very sparse data. Furthermore, a novel approach for completing directives is presented, which applies semantic analogical reasoning to transfer knowledge collected from thousands of natural-language instruction sheets to new situations. In addition, a cohesive processing pipeline is described that transforms vague and incomplete task formulations into sequences of formally specified robot plans. The system is connected to a plan executive that is able to execute the computed plans in a simulator. Experiments conducted in a publicly accessible, browser-based web interface showcase that PRAC is capable of closing the loop from natural-language instructions to their execution by a robot.

This package consists of an implementation of probabilistic knowledge services for natural-language instruction interpretation as a Python module (prac) that you can use to work with these services in your own Python scripts. For an introduction into using PRAC in your own scripts, see API-Specification.

Release notes

  • Release 1.0.0 (XX.06.2016)
    • Initial Release

Credits

Lead Developer

Daniel Nyga (nyga@cs.uni-bremen.de)

Contributors

  • Mareike Picklum
  • Sebastian Koralewski

Acknowledgments

This work is supported in part by the EU FP7 projects RoboHow (grant number 288533), ACAT (grant number 600578) and CoTeSys cluster of excellence (Cognition for Technical Systems), part of the Excellence Initiative of the German Research Foundation (DFG):

_images/robohow-logo.png _images/acat-logo.png _images/fp7-logo.png _images/cotesys-logo.png _images/dfg.jpg

Publications

  • Gheorghe Lisca, Daniel Nyga, Ferenc Bálint-Benczédi, Hagen Langer, and Michael Beetz. Towards robots conducting chemical experiments. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany, 2015.
  • Daniel Nyga. Interpretation of Natural-language Robot Instructions: Probabilistic Knowledge Representation, Learning, and Reasoning. PhD thesis, University of Bremen, 2017. URL: http://nbn-resolving.de/urn:nbn:de:gbv:46-00105882-13.
  • Daniel Nyga and Michael Beetz. Everything robots always wanted to know about housework (but were afraid to ask). In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vilamoura, Portugal, October, 7–12 2012.
  • Daniel Nyga and Michael Beetz. Cloud-based Probabilistic Knowledge Services for Instruction Interpretation. In International Symposium of Robotics Research (ISRR). Sestri Levante (Genoa), Italy, 2015.
  • Daniel Nyga and Michael Beetz. Reasoning about Unmodelled Concepts – Incorporating Class Taxonomies in Probabilistic Relational Models. In Arxiv.org. 2015. Preprint: http://arxiv.org/abs/1504.05411.
  • Daniel Nyga, Mareike Picklum, and Michael Beetz. What No Robot Has Seen Before – Probabilistic Interpretation of Natural-language Object Descriptions. In International Conference on Robotics and Automation (ICRA). Singapore, 2017. Accepted for publication.
  • Daniel Nyga, Mareike Picklum, Sebastian Koralewski, and Michael Beetz. Instruction Completion through Instance-based Learning and Semantic Analogical Reasoning. In International Conference on Robotics and Automation (ICRA). Singapore, 2017. Accepted for publication.
  • Mihai Pomarlan, Daniel Nyga, Mareike Picklum, Sebastian Koralewski, and Michael Beetz. Deeper Understanding of Vague Instructions through Simulated Execution (Extended Abstract). In Proceedings of the 2017 International Conference on Autonomous Agents & Multiagent Systems, AAMAS ‘17. International Foundation for Autonomous Agents and Multiagent Systems, 2017.

Indices and tables