Situated Virtual Reality for Telerobotic Control

1. Introduction

The objective of this project is to experiment with a new way to teleoperate mining operations based on 3D models. The idea is to substitute completely the view of the physical model by the 3D model and to teleoperate machinery without direct video feedback. Therefore, the challenge was to create an application which is a sufficiently good representation of reality to perform the teleoperation task.

2. Approach

Figure 1 The robot arm and its environment

For this first stage of the project we decided to begin the tests with a simple robot arm and a simple task to accomplish. The task consists of controlling the robot arm to push rocks to a chosen destination in a limited time using a 3D representation of the scene (Fig1).

 

So a few questions needed to be answered: How to control the robot arm, how to generate and animate the 3D model, how to track the rocks and how to check the faithfulness of the 3D scene. All in real-time.

 

3. Solutions

The application we developed from April to August 2010 included three different types of technology which are 3D modelling, game design and object tracking. The Figure 3 gives an overview of the application architecture.

 

This project started with the modelling of the objects which compose our scene: the robot arm, the board and a default shape for the rocks. All the parts have been modelled with Maya 3D software.

 

Then we imported the 3D models into Unity3D game engine which is the heart of the application. It is where the animation and the main part of the application logic take place. The scripts are mostly written in JavaScript but a few of them needed to be in C# for all the UDP communications with OpenCV, Facelab and the robot arm.

 

We used the OpenCV library to track the positions of the rocks on the board. The program is in C++ and consists of detecting the objects with the most irregularities. For that we process a video stream coming from a single Axis branded camera (Fig. 2). Thereafter the number, position and the size of each rock is saved in a datagram and sent by UDP on a port monitored by Unity3D. The default 3D rock model is loaded within Unity3D, placed and resized for each rock detected.

 

Figure 2 Rock tracking steps: video stream source, perspective correction, image processing, 3D in Unity

Two methods are available to control the virtual camera which represents the view location in Unity3D: The Logitech DualAction gamepad and/or the head and eye tracking via Facelab software and stereoscopic infrared cameras. We can choose either to use our eye gaze to tilt and pan the camera or a tip tracking method which places the tip of the 3D robot in the centre of the screen with or without delay to smooth the motions. The camera position can be changed using the simple joystick on the gamepad or moving our head right and left.

 

The gamepad is also used to control the robot arm position. Technically it changes the robot arm model position first and during the next frame of the animation a datagram is sent to the physical robot arm to match its position to the model. When the physical robot arm position differs from the model, which will occur if there is physical obstruction of the robot, a second 3D model is displayed reflecting the real position of the arm.

 

Finally everything behaves in real-time based on the application frame rate of 60 fps. The physical robot arm copies the model robot arm displacements and the rock models copy the physical rock displacements.

 

Figure 3 Application architecture overview

4. Experiments and first conclusions

In August 2010 we tested the application on a group of 15 subjects. We are still analysing the results but the first basic conclusion we can make is that all the subjects, no matter the computer game experience background, were able to push between 4 or 7 rocks per minute into the hole. The scores vary depending on the control method tested but it is already clear that user centric directional control of the robot works better than robot centric directional control for most participants.