DEEP/PLACE: site-based immersive history
DEEP/PLACE is a flexible system for site-based, immersive, interactive installations that allows participants to explore historical and cultural information in a hybrid physical/virtual space. This multidisciplinary collaborative artwork merges materials from discrete domains; navigation is through an intuitive gestural interface. The experience provides an interplay between the real and the virtual, allowing the past to infuse the present.
We have joined together as an interdisciplinary team of collaborators to explore, through technology, experiences of place and history that are engaging and interactive. As part of this endeavor, we began with a set of core guiding principles that would govern our investigations. The first principle is the experience of a hybrid physical and virtual space in which to situate site-specific enriching information such as socio-cultural and historical artifacts and audiovisual elements. In other words, we began with the premise that we would curate a wealth of site-specific information about a selected place, and that the end user would engage with the material not solely through a computer, but rather, by way of a digitally enhanced and augmented experience in the place itself. The second principle is that we would develop a customizable system that can be configured and deployed to different sites with the ability to define semantic categories to organize the curated media content. The third principle is that the user interface will be as simple and intuitive as possible in order to promote the users' engagement and provide a relatively transparent mode of interaction. The fourth and final principle is that the information architecture of the system will be non-linear and contextual so that the user of the system can navigate through the site-specific information by means of gestures within the space.
DEEP/PLACE features an expanded interactive audiovisual space consisting of diverse media elements. This multidisciplinary collaborative artwork merges materials from discrete domains—such as architecture, cultural geography and geology—in an immersive site-specific experience. Participants explore the multifaceted information by navigating a rich media landscape through an intuitive gestural interface. The media landscape is represented by a system of interconnected nodes of site-based information that include spatial and geological information, archival blueprints and images, 3D models, video and recorded audio material. A live camera feed of the site into the virtual space connects the built architectural space to the digital multi-modal history. The overall experience provides an important interplay between the real site and the virtual experience, allowing the past to infuse the present.
The DEEP/PLACE system is a flexible and customizable environment that will allow others to design their own site-based interactive explorations, populated with relevant media. The underlying structure provides coherence and ease of use while allowing the designer great freedom to individualize the experience for the particular site. The user communicates with the computer application through a gestural interface that consists of a wireless multi-sensor glove and position trackers. It implements a 3D virtual environment in which the rendition of the media elements within this space are projected onto a large screen and played through loudspeakers.
The process for a typical deployment of the system includes an archival research phase followed by digitization, recording and modeling of artifacts. In the next phase the designer organizes these media elements into thematic and chronological clusters called nodes. The final stage of the design involves the organization of media elements within the nodes, which are 3D subspaces in the virtual environment. Once the design phase is completed the system is placed at the chosen site in a location that best underscores the dual experience, that is, the interplay between the presence of being at the real site and the virtual experience. The visuals are rendered in real time and projected onto a screen that is located directly in front of the participant; the size of the screen and the location of the participant are chosen to balance the immersive aspect of the rendered content with the presence of the immediate surroundings of the site.
In Figure 1 we provide a schematic overview of the system. The media database contains assets collected during the research phase. The organizational network represents the overall design of the content and includes information about all media elements. The rendering engine implements the virtual environment by combining information from the organizational network, digital assets, the gestural interface and the on-site camera to produce the visual and auditory output.
The main organizational construct in the system is the node. Conceptually, each node represents a theme and a chronological moment in the history of the site. Each node may include multiple media objects: possible media formats include images, audio (either spatialized or ambient), recorded video and 3D models. Within the virtual space, each node occupies a defined volume. Each piece of nodal media, as appropriate, can be positioned and sized within this volume, as specified by the designer via a text file, creating a visually dynamic layered information space. The design can specify a variety of properties for each media object in the virtual space such as: theme, chronological period, type (image, audio, etc.), location and size.
Within the organizational network, nodes are linked conceptually in two manners: chronologically and thematically. The designer determines the thematic subjects and the temporal divisions based on how s/he wishes to organize and aggregate the elements in the media database. If one thinks of the themes and chronological divisions as a two-dimensional planar grid, the nodes are objects in this grid which, in turn, contain multiple media objects. In other words, each node represents both a theme and a moment in time. The system is designed so that all nodes do not have to be populated on this grid. The designer also has the ability to designate multiple nodes that depict a theme and the same chronological division. In this case, when the user arrives at a node, one of the possible nodes will be randomly chosen by the system.
In addition to designating the nodes, the designer can also choose background images and colors that correspond to themes. There is also the ability to import filtered and manipulated video from the live camera feed. The system is structured so that, once the designer has prepared the media and conceptualized the organization of nodes, it is straightforward to implement.
As a system focused on realizing site-specific installations, the DEEP/PLACE system is portable and can be set up in any site location. The main components include the interactive system (with a gestural recognition glove and position trackers for the user's hand), video projection, stereophonic audio and an on-site video camera. The user learns to make a number of navigational and exploratory gestures with the glove and attached trackers, moving through historical time and across the chosen themes for the site. The user explores the rich media landscape by navigating within and between nodes. Exploration within the node involves moving the viewpoint to a position and proximity of a media object. Object-specific audio is spatially triggered during this exploration. In contrast to node-specific ambient audio, object-specific audio may be heard based on the user’s proximity to the objects that have attached audio elements. During the exploration a live camera feed that has been filtered contributes additional media.
The experience opens with a node that contains media (video, image, audio) elements within a volume of space. The user can navigate through this volume (through gestures) and examine the media more closely. When the user has finished with the current node, a change of gesture (using finger pointing) will cause the user to navigate to a different node: a gesture in the horizontal direction will cause the user to move backward or forward in time (the movement will be either left or right) and the same gesture moving away or toward the user will cause the theme to change. As the user navigates to a new node the old node will fade and move away as the new node, populated by its media, comes into view.
The gestural interface consists of a wireless glove and 3D position sensors. Gestural movements allow the user to explore the media within a node. Gestural movements also allow the user to travel to the nearest adjacent nodes, either chronologically (the theme will stay the same but the user will move forward or backward in time) or thematically (the user will stay in the same time period but change themes). Other gestural movements also help orient the user by returning to an initial view of the current node or moving back to the system’s starting node. A timeline at the bottom of the screen helps orient the user chronologically; background images and color orient the user thematically.
First Realization of the DEEP/PLACE System
For its initial deployment and configuration, we authored the DEEP/PLACE system to focus on Harkness Chapel (Connecticut College, New London, USA) with three main themes: architecture, culture and land. Harkness Chapel is a landmark building on the Connecticut College campus that was designed and built in 1938 as a reflective gathering place. As a public site, it has a rich history of performances and events. Much of the materials we have found on the architecture of the building are in the form of letters, architectural pencil drawings, and some black and white photographs. Current material within the architecture theme include video and audio and a virtual tour of the interior public and crawl spaces and the bell and its tower. Archival materials for the architectural theme include schematic drawings and photographs of the construction of the chapel. We have included in the culture theme a large collection of letters, telegrams, and notes from meetings during the time of construction between the architect, Roger Gamble and the donor, Mary Harkness, in 1938. There are also many time-based audio and video clips of performances. The geology (land) materials include GIS (Geographical Information System) data of land use, historical maps and information about the geological rock formations. We have produced layers of 2D images and 3D models that show the site within a larger land mass. This context includes both the surrounding area and also a visualization of what is under the surface of the earth in this location.
We have elected to position the screen in front of the altar in the chapel. This choice allows the main architectural elements of the chapel (windows, ceiling, organ, beams, arches, organ grille) to be visible to the participant in addition to the panorama of the screen (see Figure 2). The background visuals of the virtual space are informed by the images obtained from the on-site camera. The wireless camera is positioned inside the chapel so that the changing light conditions are captured and used as part of the digital background in order to accentuate the quality of light coming through the windows throughout the day. This connection portrays the interplay between the actual site and the virtual one. We have also used color codes to signify our themes (architecture, culture, land) to provide thematic orientation to the participant.
DEEP/PLACE is a flexible site-based interactive installation that can be configured to allow participants to explore site-specific historical and cultural information in a hybrid physical/virtual space, creating new opportunities to enhance apprehensions and relations of place. We have created the first iteration of this system by designing an explorable topography of site-specific historical/archival, geological, architectural and audiovisual elements based on Harkness Chapel in New London, CT, USA. This collaborative project involves three core faculty (in Studio Art and Computer Science) at Connecticut College as well as students within the Ammerman Center for Arts & Technology. Expertise in history, architecture, geology and geographical information systems was provided by affiliated faculty at Connecticut College.
The authors would like to acknowledge the invaluable assistance of two Connecticut College students, Amy Barrett and Hannah Plishtin. Amy worked on the gestural interface and Hannah worked on the archival material. Both students assisted in the design development.