Building Virtual Worlds

Moshell - Fall 98

Lecture 19: Hardware for VR & Computer Graphics

The problem of producing a VR system that doesn't make users sick, and which does useful work, is not yet solved. But everyone agrees that the system's speed remains a key unsolved problem.

Speed, in a system with feedback loops, "is everyone's problem". The graphics system is often singled out as the bad guy, but in fact a head mounted system is equally vulnerable to tracker delays. So let's think about how time works, and then how trackers work.

Types of Delays

1. Frame rate - the obvious time cost. If a system updates 50 times per second, that means that it is 20 milliseconds from the presentation of one sample to the next. This means that when the data arrives, it could be between 0 and 20 milliseconds old (just because of the frame rate.... we'll discuss other problems later.) This is like the postman arriving every day at 3 PM. If you miss today's mail, you have to wait for tomorrow's, even if you only missed it by a brief interval.

You can be reasonably sure that "0 to 20 milliseconds" is more likely to represent a 20 ms real delay, than a zero ms delay. Something has to actually do some work in there, and it can't be done instantly. But there's another problem - the frame rate of the tracker will not in general match the frame rate of the image generator. A diagram will be presented on the blackboard to discuss the problem of beat frequencies or heterodyning between system components running at different frequencies.

2. Latency - the subtle time cost. Once data has been prepared, it still has to get formatted, sent through a serial or parallel port, read in by the software of the image generator, and used to re-orient the synthetic camera. These latencies add up, because the process is a pipeline. We normally speak of tracker latency, transmission latency and image generator latency as the three main components.

Types of Trackers

Magnetic: The original system was developed by a man named Polhemus, and was called the Polhemus Isotrack. That system was quite slow, with frame rate around 30 hz and latency of 50 ms. The second generation system, the Polhemus Fastrack, was better but even so was only marginally acceptable. Its advertised latency was 10 ms. But in fact, to achieve this latency you had to be operating with the sensor within about a foot of the emitter. The problem: filtering.

The "Flock of Birds" from Ascension attempted to overcome some of Polhemus' technical difficulties, and gained substantial market share. Its emitter is a black cube about 10" square - much larger than a Polhemus. They are both 6 DOF trackers- they report the absolute X,Y,Z difference in position between the source and emitter(s), as well as roll, pitch and yaw. The XYZ comes from time-of-flight information, and so the wires from the sensors and emitters back to the control box must be of a specific length. The angular information comes from the relative strength of the magnetic coupling between three pairs of coils of wire, oriented three different ways.

The flux gate compass is another cunning trick - the earth's magnetic field can actually provide orientation information. But you need some other source of XYZ absolute location information. FGCs are available cheaply from Radio Shack, and Tom Clarke at IST has experimented with this technique.

Advantages: magnetic signals run right through and around human beings with little interference. So you don't get shadows.
Disadvantages: magnetic materials and even nonmagnetic ones like aluminum, will distort the fields. This can be reduced by using DC signals rather than AC ones - though in fact they are using "slow AC" rather than pure DC fields. If a mag field is not changing at all, it doesn't induce a current in the sensor. Slow, in this case, means one or a few pulses per sample instead of 50 or 100 pulses per sample.

Acoustical: a triangle of clickers (piezoelectric) emit ultrasonic pulses; a triangle of microphones picks them up and the relative positions and orientations are computed.

Advantage: very cheap technology - used in consumer games, and the infamous Mattel Power Glove.
Disadvantage: Acoustic shadows. But the Power Glove was cunningly designed so that you pointed it at the TV running your video game, and (since you had to see the TV) the line of sound to the sensors hanging on the TV was always clear.

Query 19.1: Why did the Power Glove fail in the marketplace? (If you weren't at my lecture, you had better ask a friend!)

Optical: Twinkle some lights (usually infrared), pick them up on sensors and compute position and orientation.

Advantage: light is less subject to interference than RF (radio frequency) or low frequency magnetic fields, or sound. Also light based systems can be much faster than mag-systems, whose cycle time is limited by the frequency used; and audio systems which are limited by time-of-flight of sound.

Disadvantage: shadowing means you need more sensors & emitters to provide redundency.

Mechanical: Unlike acoustical and magnetic systems, mechanical systems are "closed" against noise. Also we can compute the value of a mechanical angle (measured by shaft encoders or potentiometers) VERY quickly. But you are now carrying around some kind of linkage arm, like an old fashioned dentist's drill.

In the BOOM displays, the boom arm carries the weight of a high resolution CRT display and also provides mechanical tracking.

Inertial. The coolest idea for cheap tracking is to use gyroscopes and accelerometers. Tiny lightweight accelerometers are made from silicon crystals and are used for airbag deployment, among other things. Gyros can tell you which way something is facing. High quality accelerometers and gyros are at the heart of Inertial Navigation Systems which control the flight of aircraft,ships and submarines worldwide.

Consumer grade inertial systems suffer from drift - they are always integrating a derivative (acceleration) to get velocity, and integrating that to get position. So they accumulate error. But - like flux gate - they have the advantage of not needing an emitter.

Filtering and latency. The electromagnetic environment in a typical building is noisy. There are electrical power lines, as well as racks of electronic equipment emitting all kinds of magnetic fields. There are radio and TV stations, cellular phones, utility trucks' CB radios, pagers and all kinds of other signal sources. A single sample is quite likely to have added to it, some amount of junk or noise. So you have to take several samples and average them together, to reduce the noise.

The problem of course is that any average cannot incorporate FUTURE signals; it can only be based on the past. A plain average of, say, the last 3 samples, would have two defects:

a) It would give too much weight to old information
b) It would take no account of the known information about the direction and rate of change

For instance, if you had three measurements of 8, 10 and 12 cm, the average would tell you to predict that the next value would be 10. But common sense would tell you that the next value would probably be 14.

A technique which performs this kind of prediction with theoretical optimal performance is called a Kalman Filter, Its inventor is a professor at UF (but we'll forgive him that unfortunate career choice...) Such filters make their predictions by computing the first and second derivatives of the signal, and then combining averages of the data with averages of its derivatives. For instance, one might predict something like

p(t+DT) = p(t) + p'(t) DT + p''(t)(DT)**2

where DT is the time step, p'(t) is the first derivative of p(t), and p''(t) is the second derivative (or the first derivative of p'(t)). DT is typically much smaller than 1. In fact, more sophisticated formulas are usually used.

Query 19.2: Hand compute the successive differences and second differences for this series, in which DT=0.1 second.

80 60 40 30 25 22

Use the above crude filter to predict p(t5+0.1), where t5 is the time of the last sample (22).

The Bishop Speedup. Polhemus and Ascension systems traditionally computed all 6 DOF and then delivered the information in a bundle. However, Gary Bishop at the U. of North Carolina and one of his grad students, realized that you could transmit the updated information piece-by-piece as it was computed. This would mean that the flying virtual object (e. g. a model of a user's hand) could have its X position corrected while its Y position was still using old data.

A simple minded application of this principle would transform a smooth-but-late series of accurate (X,Y,Z) points into a fast-but-inaccurate sort of zigzag track, consisting of (Xnew, Yold, Zold) and then (Xold, Ynew, Zold) and then (Xold, Yold, Zold). So it was necessary to incorporate another Kalman filter in the imaging system. This one estimates a new Y and Z to go with the accurately updated X; then computes a new X and Z to go with an updated Y, etc. The technique somewhat resembles the SIMNET dead reckoning concept: "keep going in the direction you were going, until told otherwise."

The Infinite Virtual Walk was an experiment we tried back in 1993, when the Toy Scouts were just forming. Hear the glorious details in my lecture!

Query 19.3: What's a String Racer and how did it contribute to the naming of the Toy Scouts?

Query 19.4: What's a Grey Code? And what has it to do with tracking human positions?

Control Devices

In class we will discuss the following hand-held input devices:

strain gauge gloves
pinch gloves
force feedback gloves
    pneumatic pouches
    piston systems
    hydraulic systems
    motorized systems

Spaceballs (named BEFORE Mel Brooks got the idea....)
dongles or 'potatoes' (or 'potatos', if you're Dan Quayle)
magic wands

PhanTOM Tool Handle Interface

Head Mounted Displays

Key concepts:

exit pupil aperture
field of view
interpupillary distance (IPD)
enclosure (older "helmets" were hot and heavy.)

CAVE displays
ImmersaDesk and Mirage Displays

Sound Systems

Stereo headphones with convolved sound
Quadraphonic headphones
Supersonic Heterodyne Sound (new stuff!)

Query 19.5: Explain what a heterodyne, or 'beat frequency' is. If you wanted to produce a 1 khz sound but wanted to use physical energy with wavelengths in the range of 1 cm, explain how supersonic heterodyne could be used and specify plausible driving frequencies. (The speed of sound is around 300 m/s at sea level.)

Back to the course index
Back to the course syllabus
Back to the previous lecture
Onward to the next lecture