Mark
Reaney
An
experimental theatre project has been underway at the University of
Kansas’ Department of Theatre & Film and University Theatre. Dedicated to examining the potential of
virtual reality technologies in live theatre, our projects have progressed from
laboratory work to the development of practical VR scenographic tools[1],
to fully mounted VR/theatre productions.[2] Each experimental project that we
undertake inspires yet another.
And, in the past year we have begun to see other production groups
repeating our techniques and further develop them. Because of the growing interest, and in order to better
organize our efforts, we have formed the Institute for the Exploration of
Virtual Realities or i.e.VR.
Our
most recent experimental production was to incorporate VR technology into a new
production of Arthur Kopit’s WINGS.
By presenting scenic elements as virtual worlds composed of disjointed,
fractured images and sounds we sought to devise an exciting and innovative
method of communicating the main character's chaotic state of mind as she
suffers a stroke and fights for recovery.
By
immersing an audience within these scenic worlds we would make the experience
immediate and powerful. Rather
than resting comfortably in the role of observer, each audience member would
experience Emily’s ordeal personally. As in any traditional production, we can build a empathy
with her by seeing how she reacts to her situation. But with an immersive presentation, we can also empathize
directly by experiencing the conflicts in her perceptions and the resulting
fears and trepidations.
So,
the focus of this next progressive step was to focus and improve the audience
immersion within the fictive world.
Immersion in various forms is an important aspect of both VR and many
types of theatre. It is an area
where the two fields work similarly.
VR simulations and theatrical works are increasingly effective when
barriers between the spectator and the simulations are reduced.
The Interface
For
the earlier production of THE ADDING MACHINE, the virtual scenery was presented
to the audience through rear-screen projections. The projections were real-time computer graphics and they
appeared to be three-dimensional because of 3-D glasses worn by the audience,
and yet the actual method or presentation was not much different from other
productions using rear-projections.
It was a conscious decision to give the first merger of theatre and VR a
more theatre-like appearance. For
WINGS however, we were committed to create a work that had the components of a
more stereotypical VR experience.
The plan was to fit each member of the audience with the trademark of
virtual reality technology, a head-mounted display.
The
head-mounted displays (HMDs) that were employed were unique in design. I-Glasses!, made by the now defunct
Virtual i-O Inc., use a system of half silvered mirrors placed at an upward
facing angle before the user’s eyes. A small LCD screen is mounted horizontally above each eye
facing downward, much like the bill of a cap. When the on-stage actors are brightly illuminated they can be
seen through the nearly transparent mirrors. When the stage lights are dimmed and the LCD’s are
bright, the images in the LCDs are seen reflected in the mirrors. By manipulating the illuminate levels
on stage and in the HMDs, the wearer is able to see only the actors, video
images alone, or video superimposed over the view of the stage. Because the video images were
stereoscopic, the virtual objects seemed to occupy three-dimensional space in
front of the viewer. By carefully
controlling the stereo parameters of the images, the objects could appear to be
next to, before or behind the actors on stage.
Built-in
stereo headphones gave us a unique opportunity for sound design. Just as the video component rendered a
view of Emily’s visual perceptions, the headphones allowed the audience
to experience her auditory world.
Because the earphones did not completely cover the ear, HMD sounds could
be layered over the speech of the live performers. Additional speakers in the house created another layer of
audio to the mix.
Like
some of our earlier projects, this experiment was partially sponsored by the
USITT. Thanks to a generous grant
from the USITT’s New Initiatives Fund, additional support from KU’s
College of Liberal Arts and Sciences, and equipment loans from Virtual i-O, we
were able to equip an audience of 60 with these HMDs.
The
images that comprised the virtual scenography were created in several
ways. The primary source was
real-time computer generated images.
Using a Macintosh computer and Virtus WalkThrough Pro software, we
created virtual environments that could be navigated by a crew member in
real-time. This software is
capable of outputting video in stereoscopic mode and so is ready-made for use
in the HMDs.
The
second image source was offstage actors that appeared as ghostly images
superimposed over the on-stage actors.
To achieve this effect a pair of binocular-mounted video cameras were
placed offstage left. These two
cameras were mounted on a single tripod about 3 inches apart. The two video signals were time-synced,
then electronically combined into a single stereoscopic video signal that could
be fed to the HMDs. Through the
use of this device we were able to see images of doctors and nurses, friends or
relatives as Emily would see them.
Being 3-dimensional, they seemed to occupy the stage with her. But,
being translucent, they may have been a hallucination or a memory.
A
third camera served as another image source. This camera and its operator were positioned behind and
above the last row of audience.
This camera was focused on the stage and like the other video sources
could be fed into the HMDs. A
single camera was used at this position because being so far from the playing
space, stereoscopic methods would have been ineffective. As Emily’s identity becomes
fractured and she loses the sense of herself, images from this camera allowed
us to superimpose another image of her over herself. By panning the camera we could make the ghostly Emily move
to either side. By zooming it
became smaller and seemingly more distant or appeared in immediate
close-up.
This
unique staging dramatically reinforced the moments when the character to
referred to herself in the third person and added new implications to those
moments when she talked to herself.
For the director, a new method of blocking a scene was discovered. Through this device actors can be
brought into extreme close-up, adding poignancy to a scene The same character
can also be pushed into the distance to illustrate alienation or emotional
distance. In one scene Emily was
flanked by a pair of doctors but seems unaware of their presence. In that scene the camera zoomed in until
Emily was isolated in the image seen in the HMDs. Therefore the “reality” held the doctors but in
shared mind’s eye of Emily and the audience, she was alone.
The
final image source was videotape that held some prerecorded image from the
other sources and some footage that was shot on location at the area hospital
and airport. By pre recording some
sequences we were able to capture some special effects and intricate blocking
that would not have been possible in live performance. Location shooting allowed us to
experiment with the use of realistic, cinematic images. The video recording also had the
practical application of providing continuous video images while the computer
operators switched to a new virtual environment.
While
there were a variety of possible image sources, the presentation of those
images also had several facets.
The playing space was divided into upstage and down stage portions by a
black scrim. This scrim not only
allowed actors to appear and disappear from Emily’s consciousness, but it
was used as a projection screen for a three-gun video projector mounted above
the audience. Upstage of the
playing space, a rear projection screen was placed. Behind it, a second video projector, this one an LCD pad,
was positioned. Since these
projectors could not utilize the stereoscopic video signals being sent to the
HMDs, another computer was needed to run virtual environments in a monoscopic
mode for them. Video signals from
the rear-of-house camera and one of the two binocular offstage cameras could be
relayed to the projectors with no loss of clarity.
With
all the pieces in place, our environment of Emily’s mind was composed of
five layers or realities. In the
first, live actors could appear in the down stage playing space and made to
appear and disappear through manipulation of light. This layer was inhabited primarily by the actress playing
Emily although other characters used it in those moments when she is most aware
of her surroundings. The second
layer was the area behind the scrim in which characters appeared independently
of the first layer. This was used
mostly as dimly seen characters in the hospital flit in and out of
Emily’s perception. The
third layer was the HMDs. Here
strong images were superimposed over the entire scene. This layer was used extensively as it
was the most powerful illustration Emily’s mental state and best brought
the audience into rapport with her.
Projection on the mid-stage scrim provided the forth layer. This frontal projection also covered
the actors in the down stage area, wrapping them in the images. This layer was used to reinforce the
HMD images and to introduce new images of its own.
The
last layer was the rear projection behind the entire playing area. This layer could also be used to
reinforce other layers, but it could also be used to depict more traditional
backgrounds. This was useful in
later scenes in which Emily’s awareness of her surroundings is becoming
more clear.
Of
the five layers, three relied on the five video image sources. These were: a computer rendering
virtual worlds stereoscopically, a second computer depicting virtual worlds
monoscopically, offstage cameras, a rear-of-house camera and a VCR. By using video splitters and switchers,
any of these sources could be sent to any of the three video layers. The selection of video sources for each
layer required additional equipment and crew.
Altogether,
the control booth was very crowded.
In addition to the usual stage manager, light board and sound operators
there were two computer operators, a VCR operator, and two operators working
the video switchers for the projectors and the HMDs. Control of the HMDs also included a video fader that let us
manipulate the relationship between HMD images and the view of the live
actors. In order to insure that
all the operators could view the stage a platform had to be built in the booth
for the control stations in the second tier.
This
variety of media required a new style of scenography. Early in the process it became evident that in addition to
an increased sense of immersion in the fictive world, this system afforded a
great deal of flexibility in how the scenic elements could be presented. To take advantage of this flexibility
we purposely avoided making hard and fast decisions about what images would be
presented in each layer at any given time. Instead we designed virtual environments, edited video and
chose camera angles that would be generally appropriate for a section of the
play. Then, during rehearsals, we
were able to try different combinations, mixing and matching video sources with
HMDs and projectors. As rehearsals
progressed, we found options that worked better than others and a set deal was
formed. However, several scenes
remained in an improvisational form.
The chaotic scenes wherein Emily first experiences the stroke and later
relapse gave our stage manager and crew a chance to improvise a new matrix of
images with every performance.
The Effect
After
the audience received brief instruction on wearing the HMDs the performance
began. The character of Emily was
discovered sitting on a chair in the down stage playing space, reading a
book. The ticking clock skipped a
beat, the reading light behind the scrim suddenly disappeared, Emily seemed to
be in some distress and then calamity followed. The stage plunged into darkness and the HMDs and projectors
began showing fractured scenes of Emily’s house, cars, hospitals,
ambulances, flashes of color, abstract objects, remembered places and strange
faces. At that moment, the
audience was launched into Emily’s role, experiencing disorientation as
she did With her, they waited through the suspense and confusion.
For
the next hour and a half, the story played out through interlaced images,
sometimes restricted to the electronic means, often played by live actors
alone, but mostly through a combination of these elements. The response to the use of HMDs was
encouraging. People were
fascinated by the experience. They
seemed particularly enthralled with their ability to visually compose the scene
by moving their heads and thereby the images in the HMDs.
Throughout
every performance people could be seen moving their heads slightly to create
new relationships between the computer generated images and the view of live
actors. In post-performance
talk-back sessions they confirmed how much they enjoyed being able to
“play director”. At
the beginning of the performance these movements were self-conscious but as the
play progressed the audience became more adept with the new technology and the
act of finding an interesting composition of images became fluid and
natural. In many instances scenes
were designed to steer the audience toward a particular composition. Images that were going to be
superimposed over live actors were created to include areas of black through
which the live action would best show up.
Other scenes were designed so that shifting the HMDs made little
difference. Rapidly moving virtual
worlds afforded little time or reason to adjust the scene.
Although
audiences were, by-in-large, thrilled with the HMDs, most performances saw one
or two members of the audience remove them for some period. When asked, the reasons varied. A few found the HMD uncomfortable,
possibly due to incorrect adjustment.
Others found the experience to intense and wanted a respite. And some were curious to see how the
production appeared without the HMD’s mediation.
As
usual, WINGS has raised issues that will need to be addressed in yet another
production. Each experiment forges
a new link in the growing chain of understanding. i.e.VR will continue to test new methods and new
technologies in an effort to increase the power of live-theatre. Our next production will focus on the
immersive qualities of scenic spaces composed of extremely photo-realistic
computer renderings. TESLA
ELECTRIC by David Fraser will be the vehicle for the experiment. It will open in February, 1998 in
the University of Kansas’ William Inge Theatre.
For
more information on the work of i.e.VR visit:
www.ku.edu/~mreaney