1. Introduction
The virtual reality assisted remote maintenance and diagnostics system
is a communications tool between an expert and the maintenance people working
at a remote site. The system to be serviced or installed could be a diesel-engine,
a turbine at a power plant, paper machine, etc. The expert views the target
with a stereo vision system. The expert can point and measure distances
with a laser pointing and measuring device. 3D virtual information can
be augmented on the live video sequences in the form of CAD-drawings, process
measurements, component data etc. The expert sends instructions to the
maintenance crew by voice communications, laser pointing, work sequence
video and animation transmissions, and by sharing "white boards" and other
application programs.
System features
-
Video and voice communications
-
Virtual model of the maintenance object: parts names, links to virtual
objects, www-links, documents etc.
-
Augmented reality: combination of virtual objects and real time video
-
Remote camera control, remote data acquisition and visualisation, Remote
laser pointing and measurements
-
Hands free and free roaming of the maintenance personnel
-
Sending and sharing of documents, drawings, images, animations, www-pages
-
Remote capture of still images
-
Stereo video and stereo virtual: stereo augmented reality
|
Virtual reality
Virtual reality (virtual environment, synthetic environment, cyberspace
) is artificially constructed environment, which can be sensed and interacted
by humans using various different equipment to mimic human senses and human
behavior.
The virtual reality is generated using computers, software and attached
peripheral units such as: graphics displays, head-mounted displays (HMD),
3-D sound systems, tactile sensors, stereo vision, head tracking systems
etc.
How do we use virtual reality
The object to be serviced or repaired is modeled in 3-dimensions. The
expert can interact with the model and navigate in the model using
a keyboard, a mouse and/or joystick. By pointing to various objects of
the model the user gets additional information in the form of new virtual
objects, text, animations etc. When the user points for example to the
top electronics enclosure of the "bottling process", she will be shown
a schematic diagram and a textual description of the various fuses.
 |
 |
| Figure 1. Expert views the image from monitor or
with head-mounted display. |
Figure 2. Maintenance person can receive instructions
from expert |
Augmented reality
In a visually augmented reality the user is not completely immersed in
the virtual world. Instead, she can see the real world with virtual objects
superimposed on it. The superimposed objects can be visual 3-D virtual
objects, text, graphics, or various symbols and signals.
How do we use augmented reality.
For remote installation and maintenance support we superimpose the virtual
model on top of real time video. The expert sees the augmented view of
the object. The information consists of visualized real time measurements,
textual descriptions, part names and numbers, links to maintenance information
database. The user can:
compare the real world with the model
see visualizations of the real time measurements, service protocols etc.
obtain in depth information of the object; animations, data
 |
 |
| Figure 3. Camera view of bottling process. |
Figure 4. A virtual model of the bottling process. |
 |
 |
Figure 5. The virtual model is overlaid
on top of the video scene. The model will stay correctly
on top of the camera scene, even when the camera is turned or moved. |
Telepresence
Telepresence links the remote real world (with sensors and actuators) and
the senses and interactions of the human operator. The system functions
as if the human operator were actually present in the remote location.
How do we use telepresence.
The expert can control the camera movements (mouse, joystick, head-tracking)
and observe the remote location by viewing the real time video and still
pictures and by hearing the sounds. The expert can communicate with
the operating personnel using speech, using a common white board for drawings
and pictures, and by pointing to objects using a laser pointing device.
The expert can also see the real time remote site measurements such as
the laser distance measurements and the machine automation system measurements.
To enhance the quality of vision dual cameras for stereo viewing are
used. High resolution still pictures (using one for mono and two cameras
for stereoscopic pictures) can be grabbed from the remote video and sent
as files to the expert. The stereo pictures are viewed through a head-mounted
display.
 |
 |
| Figure 6. Expert controls camera by mouse or joystick
movements or with head-tracker |
Figure 7. Controls are transferred to the turning
camera head, which also contains the laser pointing device. |
2. Overview of current system
The project develops a portable or semi-portable prototype system, which
consists of an expert station and a remote station. The remote station
comprises of communications unit, a dual-camera/laser pointer unit, and
the operators portable communications unit. The expert station comprises
of a head mounted display with head tracking system, joystick or mouse,
a 3D virtual model viewing and generation software with graphics-video
overlay capabilities, a telecommunications computer with videoconferencing
software, and LAN connections to expert databases.
.
| Remote communications unit:
Ethernet, ISDN and TCP/IP communications
Codec based video conferencing
Video frame grabber
Power Supplies
Pan-tilt unit for camera movements
Dual cameras for stereo viewing
Laser pointing and measurement system
Control and measurement system server software:process measurements
Video frame grabbing server software
pcAnywhere host
|
Camera head and camera head computer:
Packed in suitcase
"Shoe box" computer for camera, laser and process controls and measurements
Wireless LAN Access Point and Ethernet Hub
Power Supplies
Pan-tilt unit for camera movements
Dual cameras for stereo viewing
Laser pointing and measurement system
Control and measurement system server software: laser, cameras, pan-tilt
unit, process measurements, video frame grabber. pcAnywhere host.
|
.
| Wearable computer and accessories
A belt attached Pentium computer with mouse and optional keyboard
A head-mounted-display with microphone and ear phones
|
A miniature video camera
H.320 video conferencing software
pcAnywhere host
|
| Expert workstation
Ethernet, and TCP/IPcommunications
Video frame grabber for inputting video into the virtual world.
3D graphic accelerator cards (2 for stereo viewing)
A HMD unit for full immersion(stereo viewing)
Expert station user interface software: user controls(camera, laser,
data-acquisition), measurement visualization, real time video and virtual
objects mixing, links to process documents
|
Expert communications unit
Ethernet, ISDN and TCP/IP communications
Codec based video conferencing
H.323 based video conferencing
|
3. Partnership
The research work is carried out by Helsinki
University of Technology Automation Technology Laboratory in co-operation
with VTT-Automation (Automation section
of the Finnish State Research Center). The work is mainly financed by the
Technology
Development Center of Finland. The 8 participating companies partly
finance the the research. They will be able to utilize and to commercialize
the system and the technologies developed during the project. The participating
companies are: Cybercube Oy, Remtec
Systems Ltd, Prosys PMS Ltd, Kvaerner-
Masayards Inc., Fortum
Oil and Gas, Wartsila NSD Corporation,
Fortum
Power and Heat, Sandvik
Tamrock,
Valmet Paper Technology.
4. ARCDEV voyage
The system was demonstrated in Polar Sea onboard oil tanker m/s Uikku last
year. Report
of demonstration
Panu
Harmo's picture gallery of ARCDEV voyage (in finnish). Copyright by
Panu
Harmo, Helsinki University of Technology, Automation Technology Laboratory
TKK Automation Laboratory
Last update 9.4.1999
2.7.1999
New
in this page:
