Translating Symbolic Environments to GameEngines

Darstellung symbolischer Umgebungen in Game Engines

Bachelorarbeit

Björn Veit

Prüfer der Bachelorarbeit: 1. Prof. Michael Beetz PhD

2. Dr. Robert Porzel

Supervisor Andrei Haidu

Eidesstattliche Erklärung

Hiermit erkläre ich, dass ich die vorliegende Arbeit selbstständig angefertigt, nicht anderweitigzu Prüfungszwecken vorgelegt und keine anderen als die angegebenen Hilfsmittel verwendethabe. Sämtliche wissentlich verwendete Textausschnitte, Zitate oder Inhalte anderer Verfasserwurden ausdrücklich als solche gekennzeichnet.

Bremen, den 20th March 2019

Björn Veit

I

Abstract

The goal of this thesis is to provide robots with a dynamically spawn-able simulation space ina game engine. This is achieved by parsing a symbolic representation of the environment intothe robot’s knowledge base. All the needed information is then collected from this knowledgebase and used to spawn the parsed environment in a modern physics-based game engine,namely Unreal Engine 4. The spawned environment is an exact representation of the symbolicenvironment, that was parsed to begin with. This is tested and proved to be true. Additionallythe environment is fully connected to the knowledge base of the robot, this is showcased bya few exemplary calls a robot would need to make in an simulation context. This means thatthis thesis successfully provides a dynamically spawn-able simulation space, by translating asymbolic environment to a game engine.

This thesis is of most interest to anybody who wants to build on the fundamental work donein this thesis and use the provided simulation space, or anybody that is interested in the fieldof robot simulation capabilities.

III

Contents

1. Introduction 31.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2. Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3. Structure of this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2. Foundations 72.1. Symbolic Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2. ROS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3. Knowrob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4. Unreal Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.5. Prolog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.6. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3. Implementation 213.1. Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2. Architecture & Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3. Spawning the Semantic Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.4. Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4. Evaluation 334.1. Visualizing Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5. Conclusion 395.1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395.2. Known issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395.3. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.4. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

A. Appendix 43A.1. Prolog Code – Highlighting Example Code . . . . . . . . . . . . . . . . . . . . . . . 43

B. Glossary 45

C. Bibliography 47

1

Chapter 1

Introduction

1.1. Motivation

If you tell a person to get you a glass of milk in an unfamiliar kitchen, they will certainlysucceed at that task, without putting much thought into it. Now think about what a robotneeds to know to achieve this task. It becomes clear that even for such a simple task a wealthof knowledge is needed: “Where do I get milk?”, “Where are the glasses stored?”, “How tocarry a glass of milk, without spilling?” and “How full is the glass supposed to be?” – Theseare just a few questions that are essential to the successful completion of the task.

As humans we have learned the answers to all these questions, because we have completedthis task so many times before and have learned from our past experiences. Robots do nothave the luxury of growing up and learning how the world works, yet it is expected that theyhave learned everything they need to know before we put them into action. While modernrobots are very capable of learning from their past experiences, it is not very suitable to have arobot fail the task multiple times before successfully completing it, especially not if it is actingin the real world and not a lab.

Let us put aside all questions, like “Where are the glasses stored?” and assume the robot hascomplete knowledge about itself and the environment it is acting in. Now if the robot is able totranslate all this knowledge to a modern game engine, it would have a photo-realistic, physics-based simulation space at its disposal. The robot could now execute mental simulations [1]and therefore “grow up” without having to act in the real world.

This thesis solely focuses on implementing the translation of knowledge from a symbolic rep-resentation to a game engine and not the functionalities needed for mental simulation.

3

1. Introduction

1.2. Contribution

The process of "translating a symbolic environment to a game engine" goes through the followinglayers:

• KnowRob: The knowledge base.

• ROS: The robots operating system.

• Unreal Engine 4: The game engine.

Each layer has to pass its information down to the next one. The contribution of this the-sis contains implementations on all three levels, that work together to form the translationprocess. The initial symbolic representation is transformed to a fully functional level in theUnreal game engine. In this case fully functional means, that the environment can be inter-acted within the game engine and behaves in a natural way, since physical forces like gravityare applied during the interactions. On top of that, the level is fully connected to the knowl-edge base, this will be showcased with the implementation of some visualization capabilities.The visualization will be used to show the results of queries on the knowledge base, directlyin the translated level.

In other words, the contribution of this thesis is not only the implementation of the translationprocess, but also the validation of the said process with an exemplary but also realistic usecase.

1.3. Structure of this Thesis

In the following paragraphs, the general structure of this thesis and its chapters will be intro-duced.

Ch.2 Foundations will shine light on all the individual parts of the translation process andthe used technologies. Everything will be explained to the extent that is needed to understandthe work done in this thesis. Lastly some related work will be presented.

Ch.3 Implementation will explain the general approach and how the aforementioned tech-nologies work together with the implemented contribution to form the translation process.The correctness of said process will be tested by comparing the values in the translated levelwith the ones of the initial symbolic representation. This ensures that no information waslost during the translation process. Lastly this chapter will also provide a section on how theprocess is to be executed.

4

1.3. Structure of this Thesis

Ch.4 Evaluation is going to validate the implemented process by showing the connectivityof the knowledge base and the translated environment. For this purpose some queries on theknowledge base have been implemented, the queries are oriented on situations that wouldoccur in actual robot simulation.

Ch.5 Conclusion will give a summary of the work done. It will talk about some known issueswith the approach that was chosen and how they can be dealt with in future work. Also theoverall success of the thesis will be discussed and lastly some future work that could use thiswork as a foundation will be proposed.

5

Chapter 2

Foundations

In this section the individual parts of the process will be introduced and explained to theextent that is needed for this thesis. The following figure outlines the communication processof the translation.

Figure 2.1.: Communication Process

At this point in time the most relevant part of Figure 2.1 is that there are two systems involvedin the process: the system of the robot (ROS) and the system of the game engine (UnrealEngine 4).

In the remainders of this chapter, the individual parts will be explained. But first, we needto understand what this symbolic environment, which we are trying to translate to the gameengine, is.

2.1. Symbolic Environment

A symbolic environment is the attempt to symbolically represent a real life environment asaccurately as possible. It is therefore essentially a text representation of an environment. Ina real life environment objects also have invisible properties, for example the weight of anobject. All these properties need to be represented in the symbolic environment, if they arerelevant to the use case at hand. So it is not only listing all the objects, but also all the neededproperties. They are easy to read/process for machines, but also understandable by humans.

7

2. Foundations

This is important because humans are the ones that need make the decision which parametersare important for the representation and which are not.

The Web Ontology Language (short OWL) is an XML language that was designed for a moremachine friendly version of the internet, called the semantic web. Knowledge representedby OWL is mostly referred to as an OWL ontology, in this work OWL ontologies will alwaysrepresent knowledge in the RDF format. This combination of RDF and OWL will be referredto as a semantic map.

The Resource Description Framework (short RDF) is a data model, where everything is repre-sented in triplets. Every triplet has the following parts: a resource, a property and a propertyvalue – together they form a RDF statement. The resource is the subject, the property thepredicate and the property value the object of the statement.

Let us look at an example statement: "The author of this thesis is Björn Veit." In RDF thisstatement would be split into the following parts:

• the resource (subject): this thesis

• the property (predicate): author

• the property value (object): Björn Veit

Since the parsing of the semantic map is not part of this thesis we do not need to worry abouthow the resulting XML-structure actually looks, only the concept is crucial to this work.

2.2. ROS

The "Robot Operating System" (short ROS) is an open source framework for developing allkinds of robot software. The world of ROS is a very modular one, as everything is published inpackages, so it is easy to only use what you need and avoid overhead. A package is essentiallya software project within the ROS world. While there are library-only packages, most willprovide some form of executable code, called node. A running ROS system is therefore acollection of running nodes, communicating with each other. For this communication ROSprovides a messaging system, where all messages need to be clearly defined and typed. Thecommunication between nodes can be both synchronous and asynchronous. The synchronouscommunications are called services. For the translation process of this thesis, services areused exclusively, as the robot always wants feedback for the actions it triggered in the Unrealenvironment.

The ROS world can be imagined as a puzzle, where the puzzle pieces are the nodes and their"shape" is defined by their communication channels. By developing for ROS you create more

8

2.3. Knowrob

and more puzzle pieces, that if published can technically be used in any system, other than ina real life picture puzzle, where every piece can only be used within its set.

This high level of modular programming has a lot of advantages. For example, if you have asystem that is working great on one robot and you want to transfer that same system ontoa robot with different sensors, all you need to do is replace the sensor puzzle pieces (pack-ages/nodes). As long as the puzzle pieces for the new sensors still have the same "shape"(communication channels), you are good to go. ROS also supports a variety of programminglanguages, as each language needs to follow the same ROS communication protocol, packagescan have different languages and still run in the same system.

Rosbridge Suite1 is a package that extends the modularity even further, by providing a JSONAPI for the ROS messaging system. With this package it is possible to include non-ROS pro-grams into the ROS world, via a web-server. As this thesis is translating knowledge out ofthe ROS system and into a game engine, this is a crucial package, otherwise it would not bepossible to respond to service calls in the Unreal environment.

In this thesis no physical robot was needed, so all references to an actual robot are theoreticaland point to the ROS system. However, since all the actions are triggered from within the ROSsystem, an actual robot would have the exact same capabilities as a user operating a consoleby hand.

2.3. Knowrob

For this work, the most relevant part about Knowrob is that it gives us access to the knowledgeof the robot. However understanding the general capabilities will be very helpful in under-standing the big picture of this work. It will also give insights to how the process should beconstructed in order to be the most useful for future work.

The projects website2 states that Knowrob is: "a knowledge processing system that combinesknowledge representation and reasoning methods with techniques for acquiring knowledge andfor grounding the knowledge in a physical system and can serve as a common semantic frameworkfor integrating information from different sources." In the following paragraphs this claim willbe broken down into pieces and explained in an understandable way.

We start by answering the question "What is a knowledge processing system?" A knowledgeprocessing system is a system which is able to process all kinds of knowledge a robot hasabout its environment and its goals. Naturally the next question to answer is "Why do wewant this and what is so special about it?". In the Motivation of this thesis we briefly talkedabout some of the knowledge a robot needs to fulfill an everyday task like getting a glass ofmilk in an unfamiliar kitchen. As humans we have the knowledge that milk is a perishable

1http://wiki.ros.org/rosbridge_suite2http://www.knowrob.org/knowrob

9

2. Foundations

item, and will therefore most likely be in the fridge. A programmer could now make this arule and program the robot to always look in the fridge if it is looking for any perishableitem. However, for this to be useful, the robot needs to have knowledge about the items it islooking for. In other words, it needs to know that milk is perishable and a glass is not. Thissimple example shows why and how knowledge is important to robots. Having this knowledgesaved somewhere is one thing, but knowing how to manage, update and process it, is wherea knowledge processing system is needed.

If we now look at the next part of the website’s claim: "... that combines knowledge represen-tation and reasoning methods with techniques for acquiring knowledge and for grounding theknowledge in a physical system", it becomes more clear what is special about Knowrob. It is notonly capable of managing knowledge, but also able to use it as reasoning for the acquisitionof new knowledge. To understand how the knowledge is accessed and used, we first look athow the knowledge is stored.

The Knowrob knowledge base is saved in an OWL ontology as RDF-triplets. Most statementsare as simple as: <something more specific> is-a <something more generic>. For example: A<DairyProduct> is a <Perishable> – most of the ontology is therefore just describing who isthe subclass of whom and in the end all classes are connected, by eventually being a subclassto Thing – not a direct subclass, but Thing is at the root of the ontology and therefore every-thing is a descendant of it. According to Beetz[2] the most important branches of the upperKnowrob ontology are SpatialThing and TemporalThing. The latter is describing all kinds ofevents and also has the very important subclass Action, while SpatialThing describes abstractspatial concepts, such as containers, places and all general object classes. With every step yougo deeper in the ontology, things get more and more specific. For example the class Refrigeratoris a direct child of ElectricalHousholdAppliance, Box-Container and RefrigeratedStorageDevice.Information about class properties is passed from parents to all their children and can there-fore be declared at the most generic point possible. For example, the class ElectricalDevicemight have a property that declares it needs electricity to run, now since the Refrigerator isa subclass of ElectricalDevice, the knowledge base is able to correctly infer that it also needselectricity to run. Figure 2.2 shows a small snippet of the KnowRob ontology.

Now that we have a basic understanding of how the knowledge is stored, let us look backat our earlier example of getting a glass of milk, in an unfamiliar kitchen. If we assume thecontrol system of the robot is technically capable of getting a glass of milk, we still are missinginformation to complete the task, since the robot knows nothing about the actual whereaboutsof the glass or the milk. With KnowRob the robot can now reason about the most likely place itwill find milk, seeing that the knowledge base has quite some information about the propertiesof milk.

Figure 2.2 contains all the knowledge needed for the task at hand. Since the robot knows it islooking for milk, it can query the knowledge base to find a class that is subclassed by milk andhas a specific storage place. In this case it would start by checking if milk has a specific storageplace, since it doesn’t it would check the next thing in line and eventually come up with therefrigerator as a storage place for perishable items. Even though the milk will not always be inthe fridge, it is still the best place to look. The most important part however is that knowledge

10

2.3. Knowrob

Figure 2.2.: KnowRob Ontology Snippet

was used as a reason to come up with a likely storage place. This knowledge based approachis way more generic than a more conventional approach, where a programmer writes logicin the control program. For example, if a control program is setup in a way it that alwayslooks in the fridge if it is looking for something perishable, then it would also check the fridgeif it is looking for bread, since bread is a perishable item. Now with the knowledge basedapproach this should not happen, provided we know a more specific storage place for bread,a breadbasket for example. Since we would have knowledge that a breadbasket is used tostored individuals of the class Bread and this class is more specific than the class perishable,the same query that was used before would come up with the breadbasket first. Consequentlythe reasoning gets better and smarter, by additional knowledge in the knowledge base, ratherthan additional code in the control program of the robot.

There is another advantage of the knowledge base approach of KnowRob, a robot can tell whatinformation it is missing to successfully complete a task. We assume the robot does not knowthat bread is stored in a breadbasket, but it is trying to find bread. In this case it would cometo the conclusion that bread is perishable and check the fridge. After this attempt fails, therobot can conclude that it is missing information about the storage place of bread and it coulduse a web based knowledge base to query for that exact piece of information. A conventionalcontrol program can not tell what information it is missing and could therefore never acquirethis information on its own [2].

So far this section has mostly been about the usefulness and usage of knowledge, rather thanits acquisition. The real power of the KnowRob system comes from integrating many differentknowledge acquisition systems into one knowledge base, which is also the last part of theclaim: "... and can serve as a common semantic framework for integrating information from

11

2. Foundations

different sources." While this is surely the most impressive part about KnowRob, it will not beexplained further, as it is not directly relevant for this thesis, but if the work of this thesis isused to give a robot simulating capabilities, this is where it would fit in the system. It wouldbe added as just another knowledge acquisition system.

In conclusion Knowrob is a knowledge processing system, capable of using its existing knowl-edge to reason about non-existing knowledge, as well as being able to manage and update itsknowledge base from different sources.

2.4. Unreal Engine

Before we talk about the specifics of Unreal Engine, we need to understand why a game engineis used in the first place. Unreal Engine is a physics-based game engine. Being physics-basedmeans that realistic physics can be applied to any game object, which makes game enginesperfect for a simulation context. Realistically acting environments can easily be created usingthe physics of the game engine. While gravity is the first thing that comes to mind, manyphysics related things can be simulated. A cup that breaks if it is placed on a table with tomuch force, or a joint of any kind that can only support so much weight. Those are just somethings that can be achieved by the properties of game objects alone, without any furtherimplementation.

Unreal Engine is not only physics-based, but also has state of the art rendering techniques,which make it possible to create environments that not only act naturally, but also look veryrealistic. Unreal Engine was already successfully used for rendering 100-square-mile photo-realistic environments [3], so it is more than equipped to handle simulation environmentsfor a robot, which will most likely be limited to a few rooms. Figure 2.3 shows a picture ofthe virtual IAIKitchen environment [4] in Unreal Engine, which is the environment that isrepresented by the semantic map this work was tested on.

Amongst a variety of other editors, the Game Engine comes with a level editor. This editoris usually used to create and change levels, as well as other game content and will not bepart of the game binary that is shipped out. For this thesis however, this is not the case. Theeditor will be a part of the project, as the environment that is being translated will end upbeing represented within the editor. It is worth noting that the code for this thesis is writtento work without the editor as well, the editor is simply used to see knowledge that is easilyaccessed through code but not visualized, e.g. meta data like IDs or symbolic representationsof parameters. We will now briefly take a look at the most important parts of the editor.

12

2.4. Unreal Engine

Figure 2.3.: Unreal Engine – IAIKitchen Environment

The Viewport

Figure 2.4.: The Viewport

The Unreal editor has a viewport that showsthe currently loaded level. Within it, the usercan freely move around the level. Any gameobject in the level can be selected, and a vari-ety of manipulation tasks can be executed onthat object from within the viewport. In fig-ure 2.4 the selected object is the oven door,by dragging on any of the colored arrows thedoor could be moved in that direction. In theviewport the selected object can be scaledand rotated in a similar fashion. The type ofthe manipulation task can be changed withshortcuts, or by selecting another type withone of the buttons at the top. Besides theview we see here, a few other perspectivesand view modes can be selected, for exam-ple a wire frame view.

13

2. Foundations

The Details Tab

Figure 2.5.: The Details Tab

If a game object has been selected in the ed-itor, its properties will show up in the detailstab. Game objects can have many differenttypes, depending on that type its parametersare different. In this case the selected objectis AStaticMeshActor, which will be the mostcommon class used in this thesis. An UnrealActor is simply something that can be placedin a world, the naming conventions of Un-real dictate that any Actor type starts witha capital A. Here we have AStaticMeshActor,which can not only be placed in a world, butalso has visual representation depending onthe static mesh assigned to it. A static mesh isbasically a 3D structure of something, in thiscase the oven door. The Details tab can notonly be used to see the current values of theparameters, but can also be used to changethem. It is also possible to add new compo-nents to the game object at hand, basically everything about a selected game object can beseen and changed in the details tab. As the scrollbar in figure 2.5 suggests, the list of detailscan become quite long, but to keep things as simple as possible the image only shows veryfew of the properties.

The World Outliner

Figure 2.6.: The World Outliner

The World Outliner is listing all the game ob-jects that are placed in the current level, andshows how they are related to each other.Objects can also be selected, which makesthe aforementioned Details tab show all theirproperties. Double clicking any of the objectswill make the view port jump to that very ob-ject. The relation between objects cannot beseen directly in any other way, as relationsdo not change the way the objects look, butonly the way they interact with each other.We can see that the oven door is a child ofthe oven’s case, which means if the case wereto be moved, the door and all its other children would move with it. This does not apply theother way around. Now it is true that we want the case not to be moved when we pull on

14

2.4. Unreal Engine

the door, however when the door-handle is moved we want the door to move with it, eventhough it is a child of the oven door. This can be achieved by disabling the Simulate Physicsproperty in the details tab for the handle, but not the door. A close look at figure 2.6 revealsthat the door is not directly attached to the ovens case, but the two are attached by an APhysic-sConstraintActor call OvenDoor, which is a game object that represents a joint. In this case theproperties have been set in a way that the door opens just like one would expect it to open.This constraint also keeps the door from falling to the ground when it is opened.

In the following sections, some necessary Unreal plugins will be introduced, all of which weredeveloped as part of the RobCoG project. As this thesis is in some ways the counterpart to whatthe RobCoG project is doing, RobCoG will also be further introduced in the section 2.6.1. InUnreal Engine it is the standard naming convention to start all plugin names with a capitalU.

2.4.1. UROSBridge

Figure 2.7.: UROSBridge Editor Tab

UROSBridge3 is the counterpart to the afore-mentioned Rosbridge Suite package. UROS-Bridge takes care of the connection to theRosbridge web-socket running on the ROSsystem. Once connected, this makes it pos-sible to respond to calls from the ROS sys-tem. While this package provides the neededmeans to connect the two worlds, the han-dling of incoming messages or service callsis completely up to the user.

Every service that wants to be providedneeds to have its own server class, which hasto be registered with the UROSBridgeHan-dler, that is then calling the server’s callbackfunction whenever a request for that serviceis coming in. Since Unreal Engine is not partof the ROS system, all the used message and service file definitions need to be generated asUnreal classes in code that is understandable by Unreal. The handler then transforms the in-coming JSON messages to these classes, which makes the information accessible in the codewritten for Unreal. Thanks to the simplicity of these files on both sides, this can easily beachieved via script.

While UROSBridge can be used completely without the editor, it also provides a tab in the edi-tor (figure 2.7) that can be used to conveniently connect it to the web-socket of the RosbridgeSuite.

3https://github.com/robcog-iai/UROSBridge

15

2. Foundations

2.4.2. UUtils

UUtils4 provides various utility modules for the RobCoG project that can be used just as well inthis thesis. It takes care of value transformations between the two worlds which are necessarybecause the worlds have different standard representations. For example in ROS everythingis using right-handed orientation, which means the x-axis is pointing forward, the y-axis ispointing left and the z-axis is pointing up. Game engines however are generally using left-handed orientation, therefore the x-axis and the y-axis are flipped between the two worlds.

Besides transforming values from one standard to the other, it also provides the UTags sub-package, which allows game objects to be tagged with key-value pairs. UTags provides func-tionality to get objects based on the key-value pairs they are tagged with. This is used asthe main way to access specific game objects after they have been spawned. When the gameobjects are spawned, they are tagged with their IDs from the semantic map, which makes itpossible to access them using this ID later on.

2.4.3. USemLog

The USemLog5 plugin is technically not needed for the translation process of this thesis, how-ever it is used for testing, as it is the counterpart to this work. The plugin provides a semanticlogger that is able to convert an Unreal world state to a semantic map. In other words, it istranslating a game engine world state to a symbolic environment, opposed to translating asymbolic environment to a game engine.

2.5. Prolog

Prolog is a logic based programming language that is used to access the KnowRob knowledgebase and therefore plays a center role in this thesis. With that in mind, it is important to atleast understand the fundamental concepts of Prolog, to fully understand the work done inthis thesis. In logic based programming languages, everything boils down to facts and rules.The rules are used to evaluate the facts at hand. In the following, the basic concepts of Prologwill be shown, on some simple Prolog facts about this thesis.

1 author ( " This Thes i s " , " Bjoern Ve i t " ) .2 supervisor ( " This Thes i s " , " Andrei Haidu " ) .3 examiner ( " This Thes i s " , 1 , " Pro f . Michael Beetz PhD" ) .4 examiner ( " This Thes i s " , 2 , " Dr . Robert Porze l " ) .

These are the definitions of the facts we will be working with. Bjoern Veit is the author ofthis thesis, Andrei Haidu is the supervisor, the first examiner is Prof. Michael Beetz PhD and

4https://github.com/robcog-iai/UUtils5https://github.com/robcog-iai/USemLog

16

2.5. Prolog

the second one is Dr. Robert Porzel. These facts are saved in a file and loaded by the Prologenvironment. All information that is loaded in the Prolog environment can be queried for inthe Prolog console. All the facts have in common, they all are about this thesis. This circum-stance connects them, which allows the following simple Prolog query to questions about thisthesis.

1 ?− supervisor ( " This Thes i s " , " Andrei Haidu " ) .2 true .34 ?− supervisor ( " This Thes i s " , " Bjoern Ve i t " ) .5 f a l se .67 ?− author ( Pub l i ca t ion , " Bjoern Ve i t " ) .8 P u b l i c a t i o n = " This Thes i s " .9

10 ?− author ( Pub l i ca t ion , " Bjoern Ve i t " ) ,11 supervisor ( Pub l i ca t ion , " Andrei Haidu " ) .12 P u b l i c a t i o n = " This Thes i s " .1314 ?− author ( Pub l i ca t ion , " Bjoern Ve i t " ) ,15 supervisor ( Pub l i ca t ion , Superv i sor ) .16 P u b l i c a t i o n = " This Thes i s " ,17 Superv i sor = " Andrei Haidu " .

We start simple by asking if Andrei Haidu is the supervisor of this thesis. Since this is one of thefacts, true is returned. Next we check if Bjoern Veit is the supervisor of this thesis, consideringthere is no fact supporting this, false is returned. Prolog has a closed-world assumption, whichmeans everything that is not known to be true is assumed to be false [2].

The first two queries are simple facts, only their existence is checked. Now on the third querywe actually get a value returned. We still check facts, but in this case we use the variablePublication within the query. In Prolog everything starting with a capital letter is considereda variable. By using this variable, we are asking for all the publications of Bjoern Veit – sincein this case Prolog is only aware of one, it returns "This Thesis" as the publication. If the factswould support more than one correct answer to the query, they would be returned one byone.

On the next query we want to know if there is a publication written by Bjoern Veit and su-pervised by Andrei Haidu. This requires the combination of two facts. In Prolog a comma isrepresenting a logic AND, which means the query reads like this: Is there a publication whereBjoern Veit is the author and Andrei Haidu is the supervisor? It is worth noting that Prologwould not evaluate the second part of the query if the first one returned false already.

The last query is very similar, but it showcases that it is possible to use more than one variablein a query.

So far, all we did was check facts, now we also want to look at two simple Prolog rules.Just like facts, rules are declared beforehand. Rules are essentially facts that first have to beevaluated, but they can be queried, just like facts. However, their evaluation depends on theirimplementation, which is a logical formula of facts and other rules. A rule consists of two

17

2. Foundations

parts: the first part looks just like a fact with variables, and the second part describes how thevalues of these variables need to be evaluated. The ":-" separating the two parts can be readas "if".

1 examiner ( Pub l i ca t ion , Examiner ) :−2 examiner ( Pub l i ca t ion , _ , Examiner ) .34 work_together (A , B) :−5 author ( Pub l i ca t ion , A) , supervisor ( Pub l i ca t ion , B ) ;6 author ( Pub l i ca t ion , B) , supervisor ( Pub l i ca t ion , A) ;7 examiner ( Pub l i ca t ion , A) , examiner ( Pub l i ca t ion , B) , A \== B .

In the facts, we differentiated between the first and second examiner, but now we can alsochoose to ignore that part of the fact. In Prolog the underscore is a wild-card, which is essen-tially a variable that is never used and therefore ignored. This allows the new examiner ruleto look at all the facts and ignore the part of the wild-card.

The second rule is a little more complex, it evaluates if two people are working together. Forsimplicity it is assumed that the author and supervisor work together, as well as the examiners,but no one else. We need to use the logical OR, which in Prolog is represented by a semicolon.First we check if person A and B are author and supervisor on the same publication, butconsidering we don’t know who is who, we need to check it both ways. Since examiners arealso working together, it has to be checked if person A and person B are examiners of thesame publication. We can use the first rule, since in this case it is not relevant who the first orsecond examiner is. Lastly we make sure that person A and person B are different.

Here are some queries in the Prolog console to showcase the behaviour of the defined rules.1 ?− examiner ( " This Thes i s " , 1 , " Dr . Robert Porze l " ) .2 f a l se .34 ?− examiner ( " This Thes i s " , " Dr . Robert Porze l " ) .5 true .67 ?− examiner ( " This Thes i s " , Examiner ) .8 Examiner = " Pro f . Michael Beetz PhD" ;9 Examiner = " Dr . Robert Porze l " .

1011 ?− examiner (_ , Examiner ) .12 Examiner = " Pro f . Michael Beetz PhD" ;13 Examiner = " Dr . Robert Porze l " .1415 ?− work_together (A , B ) .16 A = " Bjoern Ve i t " ,17 B = " Andrei Haidu " ;18 A = " Andrei Haidu " ,19 B = " Bjoern Ve i t " ;20 A = " Pro f . Michael Beetz PhD" ,21 B = " Dr . Robert Porze l " ;22 A = " Dr . Robert Porze l " ,23 B = " Pro f . Michael Beetz PhD" .

On the first call we are directly checking a fact, since there is no rule to apply. The fact doesnot exist and is therefore assumed to be false. The second call only has two parameters and

18

2.5. Prolog

therefore matches the first rule we created. In this case the rule is supported by the facts andthe query therefore returns true.

Queries with variables always show all possible assignments for which the query would evalu-ate to true. This is showcased by the remaining queries, especially by the very last one, whichcorrectly returns each pair in both possible orders. Even though the third and fourth queryreturn the same thing, there is a fundamental difference between the two. The second is re-turning all examiners of all publications, while the other is only returning the ones for "thisthesis", however since there are only facts about "this thesis", the results are identical in thiscase.

While the rules implemented for this thesis are more complicated, in the end they boil downto the simple concepts shown above. After the semantic map is parsed, it is turned into acollection of facts and the implemented rules are designed to iterate through these facts andfind the information that is needed. After the information is collected it is used to call the ROSservices.

2.5.1. SWI-Prolog

SWI-Prolog is an open source implementation of Prolog [5] that is used by KnowRob. Thereason KnowRob is using this implementation of Prolog is that it brings a built-in supportfor OWL and RDF [2]. One of the most prominent Prolog rules in this work is rdf_has, whichchecks if a given triplet exist in the loaded knowledge. This rule and many more RDF reasoningmethods are all part of the build-in SWI-Prolog Semantic Web Library [5] and the KnowRobimplementation is based on this very library [2]. The library is using several indexing schemesfor the internal storage of the RDF triplets, which makes it scale well up to large knowledgebases of approximately 300 million triples on a computer with 64 GB of memory [5].

Even though ROS does support a variety of programming languages, Prolog is not one ofthem. The actual ROS service calls are implemented in C++ and fed with information byProlog queries. Luckily SWI-Prolog also comes with a C++ Interface. It makes it possible toexpose C++ functions to Prolog. In other words, it is possible to write Prolog rules in C++.

Integrating an imperative programming language with a logic based one does not come with-out certain problems, these will be discussed in the Implementation section of this thesis.

19

2. Foundations

2.6. Related Work

2.6.1. RobCoG – Robot Commonsense Games

In modern robotics, there is an approach that is called Robot Learning by Demonstration – ittries to teach robots new tasks by showing the robot how its done, rather than programminghow to do it [6]. This is mostly realized by physically moving the robot to execute the task athand.

The RobCoG project is taking the approach one step further by having robots learn tasks from"seeing" humans do the task. For the robot to be able to imitate what it saw, it needs to knowdetailed information about how the human moved, for example the angle of the hand whengrabbing something and the joint positions of the wrists and arms. Accurate information likethis is very easy to obtain if the human is executing the task in a computer game using virtualreality instead of real life. Having the robot "watch" in a virtual environment has the advantagethat it has access to the ground truth at all times [7].

In the game a player is confronted with a simple task, for example: "Set the table for a twoperson breakfast." As the person is achieving this goal, the steps are logged and saved as an"episode". Since the robot can not just copy what the humans do – as there is no guaranteethat they did a good job, or even successfully completed the task – it is necessary to evaluateeach and every episode of a human doing a task. The logged episodes are saved and can beaccessed as part of an episodic memory. If a robot is now confronted with a task it can searchthe episodic memory for similar situations and imitate the actions [7].

The relevance of the RobCoG project for this thesis is, that if this work is used to give a robotsimulation capabilities in the game engine, the evaluation techniques that were developedfor watching the human simulations can also be applied to simulations of the robot itself.The combination of the two could also be very powerful, as the robot could validate itselfand what it has learned from the human episodes. With this approach the robot could startlearning from itself after it gets the basic concepts from watching humans. Even though atthis point this is only a theoretical concept, it could be a reliable approach to robot learning,because it eliminates a few weaknesses of each individual approach. If imitating actions ofthe humans is not working out correctly, it will be noticed by the robot itself, in simulationrather than the real world. This would lead to the robot only executing plans that have provento work in simulation. If a plan was not executed successfully in simulation, the robot couldpotentially learn from the reasons of failure and change the plan until it finds something thatworks.

The logged information has to be in a format that is readable by KnowRob (and thus therobot). Therefore a logger has to translate the recorded knowledge from Unreal to KnowRob.For this task the aforementioned USemLog plugin was developed.

20

Chapter 3

Implementation

Even to a novice reader the title "Translating Symbolic Environments to Game Engines" shouldstart to make sense, as they should now have an idea of what symbolic environments and gameengines are. However, translation from one to the other still needs to be elaborated further.This should be cleared up as the approach of this thesis is explained. Afterwards a close lookat the resulting architecture and implementation will be given.

3.1. Approach

Let us start by defining what we really are trying to achieve when we are talking about "trans-lating" an environment. Let us think about translating an actual sentence into another lan-guage for a second. One could just use a translator for every word of the sentence, one byone, and could argue that the sentence is now translated to the other language. Even thoughthe words are translated, most of the time the meaning of the sentences will not stay the same.So in a successful translation the words do not necessarily stay the same, but the meaning al-ways does. In other words, the basic concept is that we want to switch the representation ofknowledge from one world to the other (KnowRob to Unreal Engine) without losing informa-tion.

In this case not only is the language changing, but also the medium representing the knowl-edge. The strictly symbolic representation is transformed into "physical" game objects. Whilesome information can be represented very well by game objects (e.g. size and weight) otherinformation, for example meta data like IDs, cannot. Since the goal is not to lose information,a way of storing meta data in game objects is needed. In the semantic map all knowledgeis represented by triplets, with the help of the UTags plugin we are able to tag game objectswith key-value pairs, therefore rebuilding our triplets on the Unreal side. If we apply this tothe example in section 2.1, we would get a game object of "this thesis" tagged with the key"author" and the value "Björn Veit".

With this approach it is possible to translate all knowledge to the game engine, and we couldsimply rebuild all triplets on empty game objects. This would of course not be of any use, as

21

3. Implementation

the point to translating the knowledge is to have a realistic looking and acting environment,but it shows that this approach is capable of representing all the knowledge in the semanticmap. In the actual work however, almost all the knowledge is stored in the game objects andtheir properties rather than in UTags.

The semantic map that is being translated contains many objects, each object represents aconcrete instance of a class. The instances will be called individuals form now on. The indi-viduals in the semantic map, have a variety of different classes. Most of these classes describespecific physical objects like IAIRefrigerator, but there is also the class Constraint, which needsto be handled completely differently than the object classes. After the translation is done, eachphysical object is represented by a AStaticMeshActor, which is essentially a game object with aphysical appearance. A static mesh can be visualized as a 3D picture of an object. So in Unrealthe different object classes are represented by the same game object type, but depending onthe object they represent, they have different meshes (visuals) assigned to them. This meansthey have the same parameter structure in the map and all need to be translated the sameway. The Constraint class is the only one that needs to be handled separately, because it hasno visual representation and will be represented by a different game object, namely APhysic-sConstraintActor – as the name suggests, this object is constraining other objects in the wayphysics can be applied to them. For example the physical constraints of a door need to besetup in a certain way, so that if forces are applied to it, it opens in a realistic way. Strictlyspeaking, any joint in the environment is represented in the form of a physical constraint.

One of the properties an individual can have in the semantic map is their set of relationsto other individuals. It will potentially have a property for its parent, as well as for all hischildren. This also needs to be handled separately from the rest of the translation, becausefor Unreal to be able to relate two game objects, they both need to be spawned already. Thismeans the translation process will collect all of these relations and only relate the objects aftereverything was spawned.

3.2. Architecture & Implementation

While Figure 2.1 already showed the very basic communication process, we will now take acloser look at how the UROSWorldControl plugin, which was developed as part of this thesis,is structured.

3.2.1. UROSWorldControl

The UROSWorldControl1 plugin is made out of the sub plugins UWorldControl and the mainplugin UROSControl. UWorldControl is a library plugin and provides all the needed function-ality to control the Unreal environment. The functionalities are split into four separate parts:

1https://github.com/robcog-iai/UROSWorldControl

22

3.2. Architecture & Implementation

the AssetSpawner, AssetModifier, AssetHighlighter and the ConstraintSpawner. The specificfunctions they provide can be seen below in Figure 3.1.

Figure 3.1.: Functions Provided by UWorldControl

The UROSControl part provides all the needed files for UROSBridge to work properly. Forevery ROS service UROSWorldControl wants to provide, it needs to have a service file describ-ing the data structure of the request. Most of the time these service files contain self declaredmessage types, all of which also need to be provided. These message and service files are verysimple data objects, that can be constructed from a JSON-Object of the incoming service calland have simple getter-functions for all their values. All these files are generated via scriptfrom the corresponding ROS service and message files.

While they make sure that Unreal knows the structure and content of incoming requests, theydo not provide any kind of functionality. Every service needs to have its own UROSBridgeserver class. These classes are the ones with the actual callback function, and need to beregistered with the UROSBridge handler, so it can call the function whenever a request for thespecific service is coming in. These callback functions contain the actual calls to the Unrealworld, but since most of the logic is split into the UWorldControl library plugin, all thesecallbacks have to do is call the right functions to execute the service. This will be showcasedon the concrete example of the SpawnSemanticMap service.

To fully understand what is going on in Figure 3.2 we first take a look at the structure of theincoming request:

1 world_control_msgs /ModelDescr ipt ion [ ] models2 world_control_msgs /C o n s t r a i n t D e s c r i p t i o n [ ] c o n s t r a i n t s3 world_control_msgs /Re la t i on De s c r i p t i on [ ] r e l a t i o n s

23

3. Implementation

Figure 3.2.: SpawnSemanticMap Service

The request is made out of three lists, one for all the models, one for all the constraintsand lastly one for all the relations between the models. The callback of this service is simplyiterating through all the lists and calling the associated library functions. In the end it sendsout a service response of the following structure:

1 bool succes s2 s t r i n g [ ] f a i l ed_mode l_ ids3 s t r i n g [ ] f a i l e d _ c o n s t r a i n t _ i d s4 world_control_msgs /Re la t i on De s c r i p t i on [ ] f a i l e d _ r e l a t i o n s

The boolean success is only true if nothing failed, the following lists are therefore empty. Whenone or more actions fail, the corresponding IDs will be listed in the appropriate list. Since weare trying not to lose any information when translating the semantic map to Unreal Engine,only a success value of true is acceptable, considering we have lost information, whenevereven a single action fails. Therefore in this work, the lists are for debugging only, but theirgeneral use can be decided by the user calling the service.

24

3.2. Architecture & Implementation

3.2.2. ROS Services

In ROS all message and service types have to be defined in srv (service) and msg (message)files. A ROS package (world_control_msgs2) was provided doing just that. The package is onlyproviding the needed definitions and contains no logic at all. The actual calling of the servicesis done from Knowrob. Below is a list of all the service files that the package provides, all ofwhich have also been implemented in corresponding UROSWorldControl Unreal plugin andhave a C++ Prolog rule to execute them.

• SpawnSemanticMap

• HighlightModel

• AttachModelToParent

• DeleteModel

• SetMaterial

• SetModelPose

• SetPhysicsProperties

• SpawnModel

• SpawnPhysiscConstraint

For most of these services specific message files need to be provided as well, all of which arepart of the aforementioned world_control_msgs ROS package. In the remainder of this workonly the SpawnSemanticMap and HighlightModel service are used, however all of the serviceshave been implemented and could be used to keep the environment up to date once it isspawned.

3.2.3. KnowRob & Prolog

For each service a Prolog rule is implemented in C++, because only the KnowRob knowledgebase can be accessed by Prolog, but the ROS service calls cannot. This however comes withcertain difficulties that will be explained further in the following paragraphs.

In logic based programming languages like Prolog there is no uncertainty: if the facts stay thesame, then the rules will evaluate to the same result, no matter how often you call a rule. Inimperative programming languages, like C++, this is not the case. The results of functionsare often dependant on internal or external states. By implementing the service rules in astatic way, dependencies on internal states can be avoided, however external ones cannot. If aC++ rule returns false to Prolog, there is no direct way of telling if it failed because the given

2https://github.com/robcog-iai/unreal_ros_pkgs/tree/master/world_control_msgs

25

3. Implementation

parameters are not supported by the rule, or if there is some external problem. For example,if there is no ROS server accepting the service call, the rule would fail, while if there is one, acall with the same parameters might succeed.

Pure Prolog rules can usually be called with variables for all parameters and return all possibleassignments that would return true. This is not the case for the rules implemented in C++,because they are executing actions, rather than just checking facts. As a consequence, theserules have to be treated like functions, for this reason higher order Prolog rules should callthem last in most cases. When implementing the rules that execute these C++ functions, thegoal should be to collect all the needed facts for the service in pure Prolog and then call theC++ function with this information. Strictly speaking, the C++ function should have nothingto do with the selection of the facts that are needed and used.

To keep the example as simple as possible, we will take a look at one of the simpler ROSservices, the attach_models-service. Before we can look at the Prolog code however, we needto look at an exemplary part of the semantic map to understand the structure of the resultingfacts we are trying to access.

1 <owl:NamedIndividual rd f : abou t="&UE−IAI−Kitchen ; K i t chenI s land1 ">2 <r d f : t y p e r d f : r e s o u r c e="&knowrob ; K i t chen I s l and "/>3 <knowrob:describedInMap r d f : r e s o u r c e="&IAI−Kitchen ;Map1"/>4 <knowrob:chi ld r d f : r e s o u r c e="&UE−IAI−Kitchen ; CookingPlate1 "/>5 </owl:NamedIndividual>

Although this part of the semantic map is trimmed to only show the relevant parts for thisservice and the otherwise generated IDs have been replaced by readable names, it is stillsufficient to showcase the general approach of all services. We start by translating it to areadable language.

Here we have an object of a kitchen island, with the ID "KitchenIsland1", that is part of theIAI-Kitchen Map, with the ID "Map1" and having a child object with the ID "CookingPlate1".All the values start with a namespace, this namespace will be ignored in the following to keepthings as simple as possible.

All this information is represented by Prolog facts in the form of RDF triplets. The relevantones are listed below.

1 rdf ( " K i t chenI s land1 " , rdf : type , " NamedIndividual " ) .2 rdf ( " K i t chenI s land1 " , rdf : type , " K i t chen I s l and " ) .3 rdf ( " K i t chenI s land1 " , knowrob : describedInMap , " xyz123 " ) .4 rdf ( " K i t chenI s land1 " , knowrob : ch i ld , " CookingPlate1 " ) .

When constructing the Prolog rule we have to keep in mind that we not only want to acceptall objects that show this structure, but also exclude all others.

26

3.2. Architecture & Implementation

1 spawn_relations (Map, Parent , Chi ld ) :−2 map_root_object (Map, ParentWithNS ) ,3 rdf_has ( ParentWithNS , knowrob : ch i ld , ChildWithNS ) ,4 rdf_has ( ParentWithNS , rdf : type , TypeName) ,5 not ( rdf_equal (TypeName , owl : ’ NamedIndividual ’ ) ) ,6 remove_namespace( ParentWithNS , Parent ) ,7 remove_namespace( ChildWithNS , Chi ld ) ,8 attach_model ( Parent , Chi ld ) .

It is important to understand that Prolog looks at the rules in order, that means if any partof the query turns out to be false, Prolog iterates to the next result under consideration ofall rules, before the one that failed. This allows us to optimize queries by carefully orderingthe rules, to exclude as many facts as fast as possible. This kind of optimization is most likelynot noticeable, but should still be in the back of our mind when constructing more complexqueries, to keep everything as scalable as possible. To understand the rule above we will do thesame thing as Prolog, and look at the rule step by step. We will also assume the query is calledwith variables for Parent and Child, but with a specific Map (e.g. spawn_relations("Map1",Parent, Child)) in other words we are looking to spawn all relations within that map.

In line 2 we start by applying a rule which will return everything that has the specified mapas its root object, so basically everything within that map. Now not everything is the parent ofsomething else, so the next step is to filter everything out that does not have a child property(line 3). Further we have to figure out if the current element (saved in ParentWithNS), isan object or something else, we could have done this first, but checking for children excludesmore things, so for performance reasons it was done first. To check if we have an object on ourhands, we have to see if it has a type (line 4). In a semantic map just about everything is of type"NamedIndividual", so we need to specifically exclude that type to make the check meaningful(line 5). Above, in the list of relevant facts, we can see that the objects we are looking for havetwo types, by excluding the "NamedIndividual" one, we ensure that it has an actual object type– which one does not matter, because its existence already proves that we are dealing with anobject. Now we have our parent (ParentWithNS) and our child (ChildWithNS) and are readyto call the actual service.

Now we excluded the name-spaces in the facts above, but in reality they are part of the tripletsvalue and are therefore included in ParentWithNS and ChildWithNS. Since we only need theIDs for the service call, we apply another rule to remove the name-spaces and then call theC++ function attach_model (lines 6-8).

With the rule being structured like this, we ensure that it is not possible to call the C++function without finding proper IDs first, no matter how the rule is called. Since it is wellstructured and the individual parts are checked to be correct at all times, we can use it formultiple things. It can be called with any combination of variables and values, in the worstcase it will just return false.

This means the rule is able to attach a specific parent to a specific child, all the children to agiven parent or vice versa. If the rule would only check if the parent and children are part ofthe map, like this:

27

3. Implementation

1 spawn_relations (Map, Parent , Chi ld ) :−2 map_root_object (Map, Parent ) ,3 map_root_object (Map, Chi ld ) ,4 attach_model ( Parent , Chi ld ) .

then the rule would still work if it is called with the right Parent and Child values, but it wouldnot handle variables well, as it would simply attach everything in the map to everything else.But even with all the precaution, the well structured rule is still vulnerable to external reasonsof failure. Potential ways of handling this will be talked about in the evaluation part of thisthesis.

For any service provided by the developed UROSWorldControl-plugin, a query was developed,in a similar fashion. While the plugin provides a variety of control functions that are able tospawn or update singular individuals, for the remainder of the thesis, we will only look atthe spawn_semantic_map service, that translates/spawns the entire map at once. The imple-mentation of which will not be discussed further, as it is using the same concepts that wereshowcased in this section. We will however take a look at how the service needs to be setupand how it is executed.

3.3. Spawning the Semantic Map

Before we can run the spawn_semantic_map service and spawn the environment in UnrealEngine, a few things have to be set up.

The Unreal editor needs to have a project loaded that has all the needed plugins installed,namely: UROSBridge, UUtils and UROSWorldControl. In addition to the plugins, the contentfolder of the project needs to be provided with all the static meshes that are needed for theenvironment. Lastly an empty level needs to be loaded for the environment to be spawned in.Since there are two separate systems involved, we need to make sure the two are connected.To achieve this, the rosbridge_server node of the Rosbridge Suite package also needs to berunning on the ROS system. If all this is provided, the UROSBridge editor tab can be used topublish the services of the UROSWorldControl plugin to ROS and therefore connect the twosystems. Figure 3.3 shows the editor in this connected state, with an empty level loaded.

After the connection is established, the rosbridge_server logs the connection and the incomingservices publications:

1 [ INFO ] : C l i e n t connected . 1 c l i e n t s t o t a l .2 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l /spawn_semantic_map .3 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / change_mate r i a l .4 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / s e t_phy s i c s_p r o p e r t i e s .5 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / h i gh l i gh t_mode l s .6 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / spawn_phys i c s_cons t ra i n t .7 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / attach_model_to_parent .8 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / de le te_mode l .9 [ INFO ] : [ C l i e n t 0 ] Ad v e r t i s e d s e r v i c e u n r e a l / set_model_pose .

28

3.3. Spawning the Semantic Map

Figure 3.3.: Unreal Editor – Connected to ROS

Something similar is also logged in Unreal Engines console, which might seem redundant,but in the case of error, it is not. Most errors are only logged on the system they appear on.Since there are no errors on either side, we are safe to proceed with the actual translationprocess.

The very first part of it is to parse the semantic map into the KnowRob knowledge base. Toaccomplish this, we use KnowRobs Prolog call owl_parse. The rule has one parameter, whichis the path to the semantic map, afterwards the map is loaded into the knowledge base andcan therefore be queried in the Prolog console. All that is left to do is to run the implementedspawn_sematic_map rule with the map as a parameter and the environment will be spawned inthe Unreal editor. Internally the query will collect all the information from the given semanticmap and then issue the spawn_semantic_map ROS service call, that triggers the execution ofthe spawning in Unreal Engine. The execution of these calls, as well as the environment thatis spawned as a result, can be seen below.

The rule map_instance called with a variable will return all semantic maps loaded in the Prologenvironment and is used to show the map was successfully loaded.

1 ?− map_instance (Map) .2 f a l se .34 ?− owl_parse ( ’ package :// knowrob_robcog/owl/SemanticMapSpawned . owl ’ ) .5 true .67 ?− map_instance (Map) .8 Map = ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#−UDRzvDoAkCTXAdaUUdaFw ’ .9

10 ?− spawn_semantic_map ( ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#−UDRzvDoAkCTXAdaUUdaFw ’ ) .11 true .

The visual of the spawned kitchen in figure 3.4 gives reason to believe that everything wastranslated correctly as nothing seems out of place and the World Outliner shows that the gameobjects have been attached to each other correctly. Whether the translation was done correctly

29

3. Implementation

Figure 3.4.: Viewport & World Outliner of Spawned Semantic Map in Unreal Engine

and no information was lost will be tested further in the next section. The reason there is aclear visual difference in the spawned environment compared to the one shown in figure 2.3,is that the semantic map does not contain any information about lighting. If lighting were tobe added, it would need to be handled separately, as it is represented by completely differentgame objects.

3.4. Testing

In this section the correctness of the translation process will be tested, by comparing theresulting environment with the values of the initial semantic map.

Ensuring the correctness of the translation process can only be done by checking every valuein the semantic map. The two worlds run on different machines and have different standards,which would make the comparison unnecessarily difficult. Luckily the USemLog-Plugin doesprovide us with the counter part to the translation done in this thesis. We can use it to generatea semantic map from the spawned Unreal environment. After this is done, we have two se-mantic maps, one that is the original and one that is generated on the spawned environment,based on the original. In theory these two maps should be identical, if the translation processis working correctly. In reality the two maps will not be completely identical, but they shouldcontain the same information. The reason they will never be identical is because some of theIDs used to connect the parts of the map internally are generated on the fly and will thereforebe different. Their purpose however will still be the same, as they are still connecting the samevalues to the same object. Since we cannot compare the two maps word for word, we need

30

3.4. Testing

to compare them in a logical way. To achieve this, the maps were parsed into the KnowRobknowledge base and analyzed via Prolog queries.

Two different queries were designed to compare the two maps: the first analyzes a given mapnumerically, by counting the number of different class types, individuals, objects, constraintsand relations. The other is comparing all values, of all individuals, one by one. Technically thefirst query is not needed, as any discrepancy found by the first query would also be detectedby the second one as well. While this is true if the maps are identical, the first query can bevery helpful if the maps are not, as it actually provides some information about the maps,rather than just saying they are or are not identical.

1 ?− analyse_semantic_map (2 Map,3 NumberOfDifferentClassTypes ,4 NumberOfIndividuals ,5 NumberOfObjects ,6 NumberOfConstraints ,7 NumberOfRelations8 ) .9 Map = ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Or i g ina l ’ ,

10 NumberOfDif ferentClassTypes = 64 ,11 NumberOfIndividuals = 136 ,12 NumberOfObjects = 114 ,13 NumberOfConstraints = 22 ,14 NumberOfRelations = 55 ;1516 Map = ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Spawned ’ ,17 NumberOfDif ferentClassTypes = 64 ,18 NumberOfIndividuals = 136 ,19 NumberOfObjects = 114 ,20 NumberOfConstraints = 22 ,21 NumberOfRelations = 55 .2223 ?− identical_maps (24 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Or i g ina l ’ ,25 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Spawned ’26 ) .27 f a l se .

The first query shows that the maps appear to be made up out of the same combination ofindividuals. However this query does only look at the number of things, and does not checkif the values of the individuals have changed. For this purpose the second query has beendeveloped. Internally the query first checks if every individual of the map is also present inthe other. Afterwards, it compares each value of each individual until it has either found adiscrepancy or iterates through all the objects. In this case it returned false, which meansthe the maps are not identical. From the first query we already know the we have the samecombination of individuals present in both maps, which means the error that makes the secondquery fail, is somewhere in the values of the individuals.

By running some more debugging queries the failing individuals can easily be determined.Afterwards the values of these Objects can be looked at manually and as it turns out, someof the floating point numbers in the map have changed slightly in the translation process.This kind of floating point error is a common problem in the world of computer science, since

31

3. Implementation

there is an infinite amount of floating point numbers, that are represented by finite number ofbyte representations, which leads to floating point numbers needing to be rounded to somedegree.

In anticipation of this problem, the identical_maps query was developed in a way that it can begiven a numerical value that describes the maximal error we want to tolerate. In the followingit will be used to determine the biggest numerical error between the two maps

1 ?− member( Delta , [1e−7, 1e−6, 1e−5, 1e−4, 1e−3, 1e−2, 1e−1]) ,2 identical_maps (3 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Or i g ina l ’ ,4 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Spawned ’ ,5 Del ta6 ) .7 Del ta = 1.0e−5 ;89 ?− member( Delta , [2e−6, 3e−6, 4e−6, 5e−6, 6e−6, 7e−6, 8e−6, 9e−6, 1e−5]) ,

10 identical_maps (11 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Or i g ina l ’ ,12 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Spawned ’ ,13 Del ta14 ) .15 Del ta = 2.0e−6 ;1617 ?− identical_maps (18 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Or i g ina l ’ ,19 ’ h t tp :// knowrob . org/kb/UE−IAI−Kitchen . owl#Spawned ’ ,20 2.0e−621 ) .22 true .

With the help of the member rule we can adapt our query to analyze the magnitude of thebiggest floating point error. The member rule returns true if the first parameter is an entry inthe list, that is provided in the second parameter. This also means that if member is called withone variable and one list, the variable will be assigned the values of the list one by one. Thisworks because Prolog will give us all possible values of the variable that would evaluate totrue. This same logic cannot be applied to the delta value of the identical_maps rule, becausethere would be an infinite amount of floating point numbers that the Delta variable couldpossibly have. For this reason, we provide a list of possible delta values that are sorted fromthe smallest to the biggest. This way if we combine the two rules, we get a query that willreturn the smallest delta of the list, for which the maps are identical.

The query returned a delta of 10−5, which means the biggest numerical difference betweenthe maps is in the range (10−6, 10−5]. By trying all values of that range, the same principlecan now be applied to determine that the exact value of the biggest error is 0.000002 .

It is worth noting that the values affected by the floating point errors are representing eithermeters, kilograms or radians. In the case of meters the error would therefore be 2 micrometer.In other words, the errors can be neglected and the translation process was successful, sinceno information was lost.

32

Chapter 4

Evaluation

4.1. Visualizing Knowledge

Technically the environment spawned in Unreal is just an Unreal level loaded from a text file,instead of an Unreal level file. This section is going to showcase, that with the combination oftechnologies used in this thesis, it is much more than that. While there is nothing special aboutthe Unreal game objects, the key is that the environment is a piece of KnowRobs knowledge-base. With that it is possible to visualize the robot’s knowledge to a human spectator.

A few exemplary Prolog rules have been implemented. They are only examples, to show thesimplicity of visualizing available knowledge. They are in no way connected to the robot’scontrol system and therefore do not show what the robot thinks in real time. However, ac-complishing this would only require similar functions to be worked into the robot’s controlsystem. For example, every time the control system is searching for an object, it needs toquery for a possible storage place. Now that same query with an addition as simple as high-light(storage_place) at the end would already provide visualization of real time knowledge.

The first Prolog rule is able to highlight everything in the map that is of a subclass of a certaintype.

1 highl ight_everyth ing_that_ is (Map, Type ) :−2 f i n d a l l ( Ind iv idua l , get_subclass_in_map (Type , Map, I n d i v i d u a l ) , I n d i v i d u a l s ) ,3 h igh l i gh t _a l l ( I n d i v i d u a l s ) .45 get_subclass_in_map (Type , Map, I n d i v i d u a l ) :−6 rdfs_ indiv idual_of ( Ind iv idua l , Type ) ,7 map_root_object (Map, I n d i v i d u a l ) .

The find_all rule can be used to get all valid variable values for a given query. In this caseit is used in conjunction with the rule get_subclass_in_map, which will evaluate true if theindividual is a subclass of the given type and part of the semantic map. By only providing thetype and the map, but not the ID, the find_all predicate is able to provide us with a list of all

33

4. Evaluation

individuals that are of a subclass of a given type. This list is used to call the highlight servicerule provided with C++, which is calling the highlighting service in Unreal Engine.

The highlight_everything_that_is rule was called four times, with the classes Handle, Door,StorageConstruct and ElectricalDevice – the results can be seen in figure 4.1. The followingsnippet will show only the first call, but the only difference in the other three queries is thesecond parameter, which instead of knowrob:’Handle’ is the corresponding class type.

1 ?− highl ight_everyth ing_that_ is (2 unrea l_k i t chen : ’ ID_Map ’ ,3 knowrob : ’ Handle ’4 ) .5 true .

Figure 4.1.: highlight_everything_that_is Query Results

Figure 4.1 shows the Unreal level after the execution of each query. The classes used are allpart of the KnowRob ontology, in other words: any knowledge of KnowRob can be projectedonto the environment in Unreal Engine. The refrigerator is just another game object, but withthe knowledge of KnowRob, we know it is an electrical device, that is also a storage constructand has a door with a handle. KnowRob of course has more knowledge about the fridge, butthis is the information that was shown by the queries.

The highlighting of class types is a visual way of showing that there is additional knowledgeabout every game object in map. However, in the following we take a look at queries with spe-

34

4.1. Visualizing Knowledge

cific results that a robot might want to run during a simulation. For this purpose the followingrules have been developed.

1 high l ight_ l ike ly_s torage_place (Map, Type):−2 l ikely_storage_place_in_environment (Type , Map, StorageP lace ) ,3 highl ight ( StorageP lace ) .45 highl ight_device_for_act ion (Map, Act ion ):−6 owl_class_propert ies ( Action , knowrob : ’ deviceUsed ’ , DeviceType ) ,7 highl ight_everyth ing_that_ is (Map, DeviceType ) .89 highlight_handle_of (Map, I n d i v i d u a l ) :−

10 handle_of (Map, Ind iv idua l , Handle ) ,11 highl ight ( Handle ) .

All rules are split into two parts, first a rule is called that handles all the logic and then thefound individuals are highlighted. The highlighting parts are just for us to identify the corre-sponding game objects. In a simulation context, a robot would not call the highlighting rules,as it has no use for highlighting. It would however, call the logic parts of the rules and thenuse the results for further reasoning. The logic parts could very well be part of a robot controlprogram and are needed for the following example. The full implementation of the the rulescan be found in the Appendix.

Let us assume the robot is confronted with the task of baking a cake, following some recipesinstructions. We also assume it needs milk to do it. It would use the likely_storage_place_in-_environment rule, to determine the storage place of milk. Afterwards it would be trying tofetch the milk, for which it needs to open the refrigerator’s door. It has knowledge that doorsopen best by gripping the handle, so it queries for the handle by using the handle_of rule. A fewsteps later the recipe’s instructions say: bake the cake. Luckily the KnowRob knowledge basealso provides information about actions and can be queried for the devices that are needed forthe action of BakingFood. After determining that the oven needs to be used, it will eventuallyneed to open the oven, for which it queries for the oven’s handle.

Obviously there are a lot more steps involved in the process, but we will focus on the partsdescribed above. In the following, the four mentioned queries will be run and the resultingindividuals will be highlighted. The IDs in the queries have been changed to readable names.

1 ?− high l ight_ l ike ly_s torage_place (2 unrea l_k i t chen : ’ ID_Map ’ ,3 knowrob : ’ CowsMilk ’4 ) .5 true .

The first query mentioned was simply looking for the storage place of the milk, so the high-lighting query was called with the map and KnowRobs class for cow-milk. The first picture offigure 4.2 shows that in the Unreal environment the fridge was correctly highlighted as themilk’s storage place.

35

4. Evaluation

1 ?− get_subclass_in_map (2 unrea l_k i t chen : ’ ID_Map ’ ,3 knowrob : ’ IAIFr idgeDoor ’ ,4 FridgeDoor5 ) , highlight_handle_of (6 unrea l_k i t chen : ’ ID_Map ’ ,7 FridgeDoor8 ) .9 FridgeDoor = unrea l_k i t chen : ’ ID_FridgeDoor ’

This query simulates the robot asking itself where to grip the fridge door. We get the con-crete reference to the individual of the fridge door, by looking for an individual of the typeIAIFridgeDoor – afterwards the query looks for the handle of the found fridge door individualand highlights it. The second picture in figure 4.2 verifies that the correct handle was foundand highlighted.

1 ?− highl ight_device_for_act ion (2 unrea l_k i t chen : ’ ID_Map ’ ,3 knowrob : ’ BakingFood ’4 ) .5 true .

Knowrob has not only classes describing spatial objects, but also classes that represent ac-tions, one of which is BakingFood. Now if we look back at the implementation of the high-light_device_for_action rule, we can see that internally the rule checks for the class propertydeviceUsed. In this case the value of that property is the class HeatSource, which explains whyin the third picture of figure 4.2 the oven and the stove are highlighted. In the following weassume that the robot was able to find a reason to use the oven and not the stove.

This is a great example of how the knowledge based approach of KnowRob is only as good asthe knowledge it has. If the knowledge base would be more specific, with a class property likeBakingDevice rather than HeatSource, this query would work and only highlight the oven, butnot the stove. There is no need to change the query, because once the knowledge is updated,it will adapt to the new property value automatically, since the querying for the deviceUsedproperty will return a more specific class.

1 ?− get_subclass_in_map (2 unrea l_k i t chen : ’ ID_Map ’ ,3 knowrob : ’ Oven ’ ,4 Oven5 ) , highlight_handle_of (6 unrea l_k i t chen : ’ ID_Map ’ ,7 Oven8 ) .9 Map = unrea l_k i t chen : ’ ID_Map ’ ,

10 Oven = unrea l_k i t chen : ’ ID_Oven ’ .

The last query is essentially the same as the second one, but this time it is looking for thehandle of the oven and not the fridge door. This query also results in the correct highlighting,which can be seen in the last picture of figure 4.2.

36

4.1. Visualizing Knowledge

Figure 4.2.: Simulation Example Query Results

This section showed that the Unreal environment is fully connected to the KnowRob knowl-edge base. In the motivation of this thesis we talked about creating a simulation space for therobot. The spawned environment could technically be used for simulation as it is, but sincethis thesis was only about the translation of the environment, all the simulation capabilitiesare yet to be implemented. However, with the completion of this work, a robot is capable ofspawning any semantic map in an Unreal environment, provided it has access to the neededstatic meshes.

While this thesis is only the first step of providing a robot with a fully functional simula-tion space, visualization capabilities like the ones we showcased in this section could easilybe added to a robot’s control system, as the UROSWorldControl package provids not onlyfunctionality to spawn an entire environment, but also to update game objects once they arespawned. These functions could be used to update the positions of objects after they havebeen acted on.

37

4. Evaluation

Let us assume a robot is put into an real life environment, like the real IAI-kitchen lab. It couldspawn the initial environment just like we did here and then update it whenever it detects anynew objects. For example the contents of a drawer are only spawned after the robot looks intoit and detects what is in it. This, in combination with some highlighting for places or objectsthe robot reasons about, would already be an impressive way of watching how the knowledgeof the robot is developing over time. Essentially the Unreal world could be used as a way tolook into the robot’s mind.

If visualization like this would be added to the control system, the visualization calls should bechanged to use ROS messages instead of ROS services. The reason being that the messages canbe sent out even if no one is listing, as opposed to service calls which always need a runningserver. On visualization there is no need for a service response, and with messages, the controlsystem would not need to worry if somebody is actually using the messages or not. This waythere would be no need to change the execution of the control system if visualization is notneeded.

38

Chapter 5

Conclusion

5.1. Summary

The goal of this thesis was to translate a symbolic environment to a game engine. This goalis achieved in the following way: KnowRob is used to parse the symbolic environment intothe knowledge base. With that, the information about said environment is accessible in theProlog console, where queries are used to collect all information that is needed for spawningthe environment. The information is then used to issue a ROS service call, which is passedto the Unreal Engine by the Rosbridge-Interface and triggers a callback function in UnrealEngine. The callback function then spawns all the game objects of the environment and finallysets up their relations. The result of this process is an Unreal environment that is an exactrepresentation of the semantic map that was parsed to begin with.

The resulting environment is fully connected to the KnowRob knowledge base. This was show-cased by some queries a robot would need in a simulation context. These queries directlyproject their results onto the environment, instead of returning them back to the Prolog con-sole.

5.2. Known issues

In this section a few known issues and potential fixes will be discussed. The issues regardthe approach of this work, they do not have any relevance for the basic translation process,but will become pressing when more advanced features will be implemented on top of thetranslation.

In the implementation chapter it was mentioned that the SWI-Prolog C++ Interface bringsuncertainty to Prolog and external reasons of failure could never be excluded. This was notreally an issue for the work done in this thesis, but represents a fundamental problem goingforward. Since there is no uncertainty in Prolog, it is also not equipped to handle it. If arule returns false Prolog and therefore the robots control system, will think that the given

39

5. Conclusion

parameters are not supported by the rule. Now if it failed for external reasons, as opposed tofailure of the rule itself, Prolog could not tell the difference, which could lead to the controlsystem making the wrong conclusion.

For example, we assume that instead of just spawning an environment, the robot uses thetranslated simulation space to actually acquire knowledge by calling the fictional rule try_approach-_in_simulation, which is used to determine if an approach the robot is planning on executingis going to work. We also assume the call fails, due to a connection issue with the Unrealenvironment. Now the robot would possibly discard the right approach, because there is nodirect way of telling the reason of failure.

This is a general problem of this approach and will always exist to some extent, but two ideason how to handle this in future work will be presented.

One way of limiting the likeliness of failure is to implement global checks for the most likelyreasons of failure. A global connection check, as well as a check if the desired ROS servicehas a running server, could potentially be used in the control system to realize that the simu-lation would fail, without it actually needing to be called. This means the failure is detectedon a different level, which makes it clear that the failing has nothing to do with the robotsapproach, so there is no reason to discard it. The problem with this idea is, that checking theconnection, before the call, will only detect if the connection was lost already and just becausethe connection was intact at the time of the check, it does not mean it will be throughout thewhole simulation.

The second idea is similar to the first one, but more direct. The main problem is not thatthe call can fail, but that there is no way of telling how it failed. Instead of returning trueor false, a status variable could be returned that has at least three states, for example: sim-ulation_successful, simulation_unsuccessful and error. This way the control system would beable to differentiate between failing because of a bad approach and other reasons of failure.In this approach any number of more specific states could be added. This would allow thecontrol program to react to specific failure reasons. If for example the simulation service isnot running, control program could start it and try again.

There is one more fundamental problem that originates from using ROS services. The ROSservice call is a blocking call, that means the code is stopped until the service server sendsthe response. If the Unreal simulation crashes or loses connection before that happens, theC++ code would never terminate. This could potentially cause the whole control system tobe stuck, because in Prolog, queries are always expected to return at some point. This couldbe avoided by using ROS actions instead of services. ROS actions have the advantage of notbeing blocking, instead a callback function is called once they are done, also they can providefeedback about the state of the action. This means the C++ Prolog rule could be implementedin a way that is blocking, as long as progress is logged by the simulation. If there is no progresslogged for a certain amount of time and the final callback function has not been called, it isclear that the simulation has failed for some reason and the blocking can be stopped and anerror could be returned. This concept would ensure the C++ call always terminates at somepoint.

40

5.3. Discussion

5.3. Discussion

In the work of this thesis a translation process was implemented that is able to translate asymbolic environment to a game engine. This translation was tested and its correctness hasbeen shown. Even though not all the values in the resulting environment stayed exactly thesame, the largest error in the translated environment was 0.000002 and can therefore beneglected. The errors are the result of a typical floating point error that cannot be avoided.Since the game objects in the spawned environment are tagged with the IDs of the individualthey represent in the semantic map, they can be seen as an extension of that individual.This extension exposes the individual to all the potential simulation capabilities of the UnrealEngine.

The resulting Unreal environment of the translation process is technically ready to be usedas a simulation space, as all the object and constraint properties are set correctly. It is worthnoting that the RobCoG project would already provide logging and evaluation techniques thatcould be used to analyze mental simulations of the robot. However, there are still some partsmissing to make the mental simulations a reality, this will be further discussed in the nextsection, namely Future Work.

During this thesis, some fundamental problems of the approach were detected. These prob-lems are not directly relevant for the work done here, but could become relevant problems infuture work, for which this thesis is meant to be a foundation. So looking back, it would havebeen better if the suggested changes in the last section, to limit and avoid the problems, wereimplemented to begin with. The reason they are not, is simply that these problems where onlydetected once the implementation was already done and since they do not directly affect theprocess of this thesis, there was no pressing reason to change the implementation. Other thanthat there are no regrets regarding any of the design or technology choices made in this thesis.Both KnowRob and Unreal Engine are modern technologies with a bright future and it is trulybelieved that the work done here can be built upon and can serve a greater purpose in thefuture.

5.4. Future Work

The motivation of this thesis was to get one step closer to providing robots with full mentalsimulation in game engines. The semantic environment has successfully been translated to anUnreal Environment, that is technically ready for simulation and the RobCoG project is alreadyproviding the necessary functionality to analyze and make use of such mental simulation. Sothe question is what is still missing to make it a reality. According to A. K. Bozcuoglu and M.Beetz a robot needs to have a simulations system that is able to represent the world model andits self abilities. Additionally the simulation needs to be controllable by the robots control andplanning systems [1]. This thesis does only provide the representation of the world model, but

41

5. Conclusion

nothing else. In other words, the self abilities of the robot and controlling of the simulationare yet to be implemented.

For the self abilities of the robot, an accurate game character of the robot would need tobe developed that can be controlled by ROS action calls. In the introduction of ROS, thesystem was compared to a puzzle, where pieces could be replaced by any other piece of thesame shape. This concept could be applied here. Thanks to the modularity of ROS and theconnection via UROSBridge, it would be possible to have the game character be controlled bythe same ROS calls of the control system that would be used in real execution. This wouldsimply require switching out the puzzle pieces of the real robot for ones in the simulation.Instead of the mechanical arm, the simulated one would be connected the exact same way.If this concept would be applied to all sensors and physical actors of the robot, it would bepossible for the robot to switch into simulation mode and execute its plans in the same wayit usually would.

Currently the translation process does not support any light sources to be spawned in the envi-ronment, this could however be a good addition once mental simulation is fully implemented,as it would allow the robot to train itself under many different lighting conditions.

Another use case for the translated environment, that would not require much additionalimplementation, is real time visualization. While the basic idea was already explored exem-plarily as part of this thesis, the visualization capabilities would still need some additions tobe of real use, for example, displaying trajectories and markers, some of which have actuallyalready been developed as part of a different project. All these visualizations would still needbe added to the robots control system to show real time information.

42

Appendix A

Appendix

A.1. Prolog Code – Highlighting Example Code

1 highl ight_everyth ing_that_ is (Map, Type ) :−2 f i n d a l l ( Ind iv idua l , get_subclass_in_map (Map, Type , I n d i v i d u a l ) , I n d i v i d u a l s ) ,3 h igh l i gh t _a l l ( I n d i v i d u a l s ) .45 highl ight_device_for_act ion (Map, Action , Device ):−6 owl_class_propert ies ( Action , knowrob : ’ deviceUsed ’ , Device ) ,7 highl ight_everyth ing_that_ is (Map, Device ) .89 high l ight_ l ike ly_s torage_place (Map, Type):−

10 map_instance (Map) ,11 l ikely_storage_place_in_environment (Map, Type , S torageP lace ) ,12 highl ight ( StorageP lace ) .1314 highlight_handle_of (Map, Ind iv idua l , Handle ) :−15 handle_of (Map, Ind iv idua l , Handle ) ,16 highl ight ( Handle ) .1718 highl ight ( I n d i v i d u a l ) :−19 remove_namespace( Ind iv idua l , Id ) ,20 highlight_models ( [ Id ] ) .2122 h igh l i gh t _a l l ( L i s t ) :−23 f o l d l ( remove_namespace_and_add_to_list , L i s t , [ ] , Id s ) ,24 highlight_models ( Ids ) .2526 handle_of (Map, Ind iv idua l , Handle ):−27 map_instance (Map) ,28 get_a l l _ch i ldren ( Ind iv idua l , Chi ldren ) ,29 member( Handle , Chi ldren ) ,30 rdf_has ( Handle , rdf : type , HandleType ) ,31 rdfs_subclass_of ( HandleType , ’ h t tp :// knowrob . org/kb/knowrob . owl#Handle ’ ) .3233 get_subclass_in_map (Map, Type , I n d i v i d u a l ) :−34 rdfs_ indiv idual_of ( Ind iv idua l , Type ) ,35 map_root_object (Map, I n d i v i d u a l ) .3637 l ikely_storage_place_in_environment_with_highl ighting (Map, ItemType , S to rage Ind i v idua l ) :−38 l ikely_storage_place_in_environment (Map, ItemType , S to rage Ind i v idua l ) ,39 remove_namespace( S torage Ind iv idua l , Id ) ,40 highlight_models ( [ Id ] ) .

43

A. Appendix

4142 l ikely_storage_place_in_environment (Map, ItemType , S to rage Ind i v idua l ) :−43 l ike ly_s torage_place ( ItemType , S torageP lace ) ,44 map_root_object (Map, S to rage Ind i v idua l ) ,45 rdf_has ( S torage Ind iv idua l , rdf : type , S torageP lace ) .4647 l ike ly_s torage_place ( ItemType , S torageP lace ) :−48 rdfs_subclass_of ( ItemType , Subc lass ) ,49 owl_class_propert ies (50 StoragePlace ,51 knowrob : ’ typePrimaryFunct ion−conta inerFor ’ ,52 Subc lass53 ) .5455 remove_namespace(WithNS , Raw) :−56 s p l i t _ s t r i n g (WithNS , "#" , ’ " ’ , New ) ,57 New = [_ ,Raw ] .5859 get_a l l _ch i ldren ( Ind iv idua l , Chi ldren ) :−60 get_a l l _ch i ldren ( Ind iv idua l , [ ] , Chi ldren ) .6162 get_a l l _ch i ldren ( Ind iv idua l , Tmp, Chi ldren ) :−63 get_a l l _d i rec t_ch i ldren ( Ind iv idua l , D i r e c tCh i ld ren ) ,64 ( not ( length ( D i rec tCh i ld ren , 0)) −>65 f o l d l ( get_a l l_chi ldren , D i rec tCh i ld ren , D i rec tCh i ld ren , GrandChildren ) ;66 true ) ,67 append( GrandChildren , Tmp, Chi ldren ) .6869 get_a l l _d i rec t_ch i ldren ( Ind iv idua l , Chi ldren ) :−70 f i n d a l l ( Child , rdf_has ( Child , knowrob : parent , I n d i v i d u a l ) , NewKids1 ) ,71 f i n d a l l ( Child , rdf_has ( Ind iv idua l , knowrob : ch i ld , Chi ld ) , NewKids2 ) ,72 append(NewKids1 , NewKids2 , NewKidsList ) ,73 l i s t _ t o _ s e t ( NewKidsList , NewKids ) ,74 Chi ldren = NewKids .7576 i s _cons t ra in t ( I n d i v i d u a l ):−77 rdf_has ( Ind iv idua l , rdf : type , Type ) ,78 rdf_equal (Type , knowrob : ’ Cons t ra in t ’ ) .7980 remove_namespace_and_add_to_list (WithNs , Tmp, L i s t ) :−81 s p l i t _ s t r i n g (WithNs , "#" , ’ " ’ , New ) ,82 New = [_ ,Raw] ,83 append(Tmp, [Raw] , L i s t ) .

44

Appendix B

Glossary

game binary is an the binary version of a game that can be executed and does not containany readable code.

getter-function is a function that is returning an internal value of an object that otherwisewould not be accessible.

ground truth is the complete knowledge about the environment a system is acting in.

JSON API is an interface in using the JSON language that is data format for exchangingdata in web-applications.

ontology is a set of class categories that shows their properties and relations to each other.

OWL stand for Web Ontology Framework and is a XML language, designed specifically forthe semantic web.

parsing is the act of breaking a data file into small chunks of information by following a setof rules on how to interpret the data.

Prolog fact is something saved in the Prolog knowledge base that is known to be true

Prolog query is the combination of rules, facts and variables executed in the Prolog envi-ronment.

Prolog rule is a rule on how to evaluate given parameters that is implemented prior to exe-cution.

RDF stand for Resource Description Framework and is a data model, where everyhing isrepresented in triplets.

45

Glossary

ROS stands for Robot Operating System.

semantic map is a representation of an environment using the combination of RDF andOWL.

semantic web is a more machine friendly version of the internet that can be used by robotsand other intelligence to acquire information.

web-socket is socket that can be connected with over a network or the internet to establisha bidirectional connecting between two devices.

XML stands for Extensible Markup Language and is a data format for exchanging data.

46

Bibliography

[1] A. K. Bozcuoglu and M. Beetz. A cloud service for robotic mental simulations. In 2017IEEE International Conference on Robotics and Automation (ICRA), pages 2653–2658, May2017.

[2] Moritz Tenorth and Michael Beetz. Knowrob: A knowledge processing infrastructure forcognition-enabled robots. The International Journal of Robotics Research, 32(5):566–590,2013.

[3] Gavin Moran. Pushing photorealism in "a boy and his kite". In ACM SIGGRAPH 2015Computer Animation Festival, SIGGRAPH ’15, pages 190–190, New York, NY, USA, 2015.ACM.

[4] Asil Kaan Bozcuoglu Michael Beetz Andrei Haidu, Daniel Bessler. Knowrob_sim - gameengine-enabled knowledge processing towards cognition-enabled robot control. In 2018IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid,Spain, October 1-5, 2018, 2018.

[5] Jan Wielemaker, Tom Schrijvers, Markus Triska, and Torbjörn Lager. SWI-Prolog. Theoryand Practice of Logic Programming, 12(1-2):67–96, 2012.

[6] A. Billard and D. Grollman. Robot learning by demonstration. Scholarpedia, 8(12):3824,2013. revision #138061.

[7] Michael Beetz, Daniel Beßler, Andrei Haidu, Mihai Pomarlan, Asil Kaan Bozcuoglu, andGeorg Bartels. Knowrob 2.0 – a 2nd generation knowledge processing framework forcognition-enabled robotic agents. In International Conference on Robotics and Automation(ICRA), Brisbane, Australia, 2018.

47