Object Recognition and Spatial Awareness for a Spot Robotics System

Earlier this month, we were joined by Joris Sijs, Lead Scientist at TNO on the second series of Grakn Orbit. TNO is the national applied research organization of the Netherlands. They first started working with Grakn about two years ago. Up to that point, they had been working on autonomous systems, but in this case, they wanted to get as close to real world scenarios as possible.

They define autonomous systems as the combination of robotics with AI.

Situational Awareness

Situational Planning

The testing and experimentation Venn diagram is something we will have to explore later, but we could go so far as to say the ability to experiment successfully (resulting in the release of a new feature/product/etc.) is directly correlated to the speed of provisioning and quality—or how production-like it is—of synthetic data they are using.

The scenario is made up of four rooms, a hall, and a porch. There are four victims and a fireman on site. SPOT is sent in to get accurate information on the whereabouts of the victims, their condition, and to reduce noise for the rescue team.

The SPOT robot — from Boston Dynamics — is then sent into the villa to retrieve or localise victims. SPOT’s objectives include:

• locate family members
• locate pets
• distinguish a real dog from a toy
• SPOT should assess when the situation requires human intervention based on victims’ condition or other variables like location, or proximity to danger, etc.

The Hardware Setup

PTZ camera — pan tilt zoom camera mounted on top of SPOT. These components are mounted on top of a mini PC. This mini PC has a lot of computing power, as most of the computations is done on the device itself to reduce latency.

The image is then passed to an image recognition module, which is then used to do some room characterisation such as identifying doors, windows, and other structural elements. The association module then takes these detections and associates them so that they can create better tracks instead of individual detections.

Traditionally in robotics, ROS and Python are used to enable communication between the modules in the system. If you take one of your modules or software components and you want to connect it to Grakn, you need to create a dedicated ROS client for Grakn as well, along with using the native Python client.

Grakn ROS Client

A setup consists of: a camera, image recognition, an association layer, a planner and a controller. ROS facilitates the merging of that input date into your database. Let’s break down the flow:

• the Grakn client subscribes to the image in order to feed it back to the database
• the dog is then published on the ROS bus
• the association module where the speed and position of the dog are written into the database
• the planner can then read from the database (object velocity) and vice-versa

How Do You Go About Building This?

There are two main components: the ROS wrapper and the Grakn client session.

The ROS wrapper handles the publish / subscribe mechanisms, the Grakn client session is built on top of the native Python client. What are the requirements for each?

• The ROS wrapper initialises the client and the topics to publish or subscribe to.
• The Grakn client session should also be able to create a keyspace. This is necessary as in robotics we want to start with an empty keyspace. We want to have a fresh scenario for the dynamically changing environmental situations that the robot encounters.

After a while, when things may have been running for some amount of time, a new ROS message might come in with new information. Here, they use Grakn utilities, which is a library that helps with common Grakn operations.

Joris and his team are currently looking at ongoing work around automated query generation.

This slide shows us how a read function works. The data is then retrieved, and put in a nice format that can be given to the ROS wrapper which publishes it again as a ROS topic. This gives the planner access to all the variables for the robot to observe.

Here we see that they have defined a living-being entity. These two are then the parent entities for: adult, child ; and pet. A living-being has a physical state and mental state. Specifically, SPOT is tasked with identifying the mental-responsiveness of an identified living_being.

We can also see how an instance of this model looks, referring to the lower section of the slide above. They have some people and a dog, fluffy. Notice that in this instance there are two adults, man(with name Sam) and man(with name George). Sam is the fireman, and George is the father of the family.

We can see that Sam is mentally responsive and George is not mentally responsive. How do we create the mental responsiveness attribute?

Let’s say the mother is mentally responsive and the father is not. This is modelled using Grakn’s inference rules. Joris and his team are currently looking at ongoing work around automated query generation.

Small but Important: Spatial Awareness

Setting the scene in the scenario.

You can see all the relations in green and the entities in blue. At first glance, it’s not as large or complex as you might expect.

Looking at the slide below we see that they model rooms as polygons. A polygon is a mathematical concept that represents a closed shape made up of connected line segments. Each room is represented as a polygon, and structural parts like walls and doors are part of that polygon.

Modelling this in Grakn we get the entity space and closed-space, and real-estate which is a type of closed-space. A room is a type of real-estate. We also have structural-part entities like walls and doors.

If it is in the kitchen and needs to exit the room via the kitchen door, it must know the position of the door. In this way, using polygons are handy to have if you want to know where the doors are.

We saw how Joris modelled a specific room in order for SPOT to move within it; but how do we model the composition of a building in the schema — we can see how his diagram might look in Graql:

space sub entity;
closed-space sub space;
real-estate sub closed-space,
    plays composition;
room sub real-estate;
composing sub relation,
    relates composition,
    relates component;
structural-part sub entity,
    plays component;

They have three rooms: kitchen hall living. These rooms are composed of structural-parts: kitchen-wall, kitchen-door, hall-wall, living-wall, living-door.

As Joris explains, we can make use of Grakn’s hyper-graph and rule-based reasoning to create room connections (relations) based on commonly associated structural-parts.

If we know that the kitchen-door is a structural-part that is a player in a relation (c#2 in the slide above) with the room kitchen, as well as the room living, you might infer yourself, we can then make a relation between the rooms: kitchen and the living, via the kitchen-door and living-door. This is an example of a transitive relation in Grakn.

This gives SPOT the knowledge that it can move from the kitchen to the living room via the kitchen-door and living-door. This is an example of a transitive relation in Grakn. This spatial awareness is combined with SLAM or other navigation techniques to help SPOT understand its environment.

Here we can see what this looks like in Grakn Workbase — Grakn’s IDE:

Modelling Negation

How do we address the fact that SPOT doesn’t yet know where the living-beings are located when the robot enters a building. How do we model this lack of knowledge in our database?

In Grakn, we use a searching relation and not-locating and actually-locating. These allow us to address the negative; once a space is searched, capturing in the database that a living_being is not located in that room. We don’t need SPOT re-checking rooms it has already searched.

What About Adding New Knowledge?

In real world scenarios, new facts are presented during the mission. Rather than adding this new knowledge to the database as an instance, you should update the underlying knowledge. You want to add this to the schema as a new type or relation.

This is happening on a regular basis. Doing this in a suitable and automated way is essential. Grakn’s dynamic schema — able to be updated at any time without needing to do any migrations — makes this quite simple.

Concluding his talk, Joris talks about what excites him about this work:

Also, knowledge graphs are hot. So combining those, in my sense, gives me the fever. You can see some of my fevers here.

Joris and his team are currently collaborating with James Fletcher, a PhD student at the University of Bristol, to continue advancing this work. You can watch the full talk here.

Many thanks to Joris and his team at TNO for their enthusiasm, inspiration and thorough explanations.

---

Originally published on Medium by Daniel Crowe on October 27, 2020. Read the full article