Art and Play in Interactive Projections: Three Cases

 We examine how three art-related projection projects approached issues of viewer participation, interactivity, user input and artistic expression differently. Each project presented video projections to a non-specialist audience with software controlled interactivity. One objective was to create an ambient play experience in a public space – something without a beginning or an end that participants could join and leave casually.


Terms of Engagement

The three interactive video projects presented here were led by the authors and emerged from an academic research context. The projects started with different assumptions about the user, different communication goals and different production and collaboration strategies. Although they share some features with simulations, they all avoid standard gaming conventions; there are no levels, no overt objectives, no winners or losers. Additionally, the core requirements of each project were linked to the contexts they emerged from: experience design, art practice and healthcare delivery.

In each of these three productions, Tentacles, Trio and The Art of Waiting, the viewer is encouraged to participate in unstructured play. As with every interactive experience (and in fact, most other things in life) there is the initial satisfaction resulting from simply figuring out how one’s decisions, gestures and actions cause reactions and create effects in the surrounding environment.

Tentacles is a large public projection with game-like user controls accessible through an iPhone. [1] In Tentacles, launching the app on your phone instantiates your creature on the screen. The main control interface on the phone allows you to move your creature around in the projected space. Some things you encounter make you bigger and some make you smaller. You might find yourself the sole inhabitant of this watery world, or you might share it with other players standing nearby. But what you do beyond that is up to you. We observed that some players try to grow their creature by directing it towards nourishing bits floating around, only to find that the bigger they become the more sluggishly they react to the controls. Others entangle with their peers, either affectionately or aggressively, thrusting themselves about like virtual egos. Finally, there are players who seem to delight in simply moving around on the screen, perhaps dancing, flirting or simply pleasing onlookers with their grace and style. This spontaneous performance could be the form of action which most connects the player with their creature. Proudly aware that they are watching their own avatars, we have often observed players with their free hand outstretched towards the screen, pointing out their movements to their human friends, but also appearing to want to touch their creature, in a way reminiscent of Michelangelo’s image of God reaching to Adam in the Sistine Chapel.

Trio is an interactive video art installation displaying three folk musicians playing a song together. Viewers using mobile phones can switch between different musicians to create alternate arrangements of the song. In Trio, like Tentacles, there is much happening on the screen before the viewer even chooses to interact. Three large projected musicians sway gently as they strum and pluck their country music song. A prominently displayed phone number encourages people to engage by dialling in, but with little explanation of what they might expect to happen next. Unlike Tentacles however, Trio presents a dense layer of detailed instructions once the viewer has logged in. Prompted by a long rambling poem on the other end of the phone line, the viewer learns they can control the images by substituting one musician for another:

“Press 1 to reach out to Iriz. Press 2 to connect with Steven. Press 3 to tickle Diego. 1 for bass; 2 for uke; 3 for squeezebox, 4 for zither, 5 for slower, 6 for faster, 7 for heaven, 8 for eleven, 9 if you want to call Golan Levin, 0 for naught, 0 for naught…” [2]

At first the parameters seem straightforward: substitute one musician for another by pushing the buttons on your phone. But beyond that, the rules are invented by the players. Some might try to press the right button sequence to put together an all-girl band. Or a bearded band. Or create a grunge version of the ensemble. At the same time other players in the crowd are thwarting your plans because they have their own agendas. The loose structure of the play interaction encourages this sort of spontaneous improvisation and ad hoc gameplay. Whatever value the content had for the viewer is supplanted by the thrill of empowerment over the interaction.

In the third project, The Art of Waiting fused the productive impulses of the designers with those of the players. [3] A group of university art and design students worked with researchers at a children’s rehabilitation hospital to produce interactive experiences for a large-screen projection in a clinic waiting area. The requirements were unusual. A nine square-meter area with an array of 100 densely packed pressure sensors in the floor created an input sensor that would be equally accessible to children with motor impairments including those using wheelchairs and assistive devices. Even parents and attendants could engage from their seats at the periphery of the area by reaching a toe in and touching the floor.

The designers of this interactive environment had therapeutic goals in mind. [4] In order to calm children before potentially stressful medical appointments and to empower mobility restricted children, the interactions needed to reward slow or static behaviours as much or more than energetic behaviours. Collaborative or social actions were also considered desirable. And since many visits involved more than one period in the waiting area, persistence within the interactive experience would create a sense of familiarity and comfort for children returning 30–40 minutes later.

The third year art and design students created two fully functional interactions which addressed these demands. One depicted a sloped grid of 100 squares, each corresponding to one of the in-floor sensors. A player pausing on a sensor would cause a virtual plant to begin growing on that spot on the screen. Moving to another sensor would cause the original plant to shrink and start another one growing. But if a player stayed long enough in one spot, their plant would become more permanent, persisting over a period of time proportional to the time spent creating it. Slowly moving across all of the sensors would create a virtual forest and several players working together could come closer to achieving maximum density.

As with the previous examples, the richness of the imagery suggests that the ‘game’ is fully fashioned and that one simply has to discern the rules. But once again, the activity ends up encouraging unstructured play, and social rules of engagement are negotiated in real time among the participants.

User Control – Input Paradigms

While all three of these installations share a common presentation form – a single projected video image controlled by software – they each use different techniques and strategies for collecting input from the users. Tentacles and Trio use handheld devices and The Art of Waiting uses environmental sensors.

Tentacles was originally designed to be controlled with an iPhone or iPod Touch and was later extended to include the iPad and Android devices. The application presents a graphic interface that reflects the images projected on the large screen. Users drag their fingers across their touch screens to steer their creatures around in the projection. The further they move from the centre of their handheld screen the faster their creature moves. There is visual feedback on the small screen to indicate direction and speed.

In addition to the visuals, however, there are sound components which go further to link the large screen to the small screen of the device. A background soundtrack plays in conjunction with the large image, augmented by smaller musical elements which play asynchronously on each user’s device. When you engage, your handheld unit springs to life, emitting sounds which intersect with the musical soundscape, calling to and enveloping passersby and proliferating as more people join in. At this point players and non-players become acutely aware that the creatures on the large screen represent participants who are in the crowd all around them. The multiple sound sources, like the multiple participants holding their small devices, combine to form one single social entity, which is only partially revealed on the main visual and aural display in front of them.

The custom device application for Tentacles is meant to extend the experience from the large screen to the small screen and to afford control features specifically designed for this interaction. In contrast, Trio uses the familiar paradigm of a phone call directed to an automated response system. Pre-recorded messages prompt uses to input their control choices by pressing the number keys on their phones. While hardly intuitive, this form of interaction is so familiar to anyone who has ever tried phoning a company or institution that users quickly move on and begin exploring how their choices affect the projected images. Separate from the longwinded recorded message, patterns correlated to the user’s input begin to emerge. The leftmost musician on the screen can be swapped with two others by using the leftmost buttons on the keypad: 1, 4 and 7. The centre musician is controlled with the centre keys: 2, 5 and 8. And the right with the right: 3, 6 and 9. This discovery frees the user from having to try to understand the complex verbal instructions. Additionally, the one anomalous key: 0 turns out to create a short, stuttering effect as though pressing it came close to crashing the system.

The decision to use the simplest phone interface was based on several considerations. First was the techno-social fact that not everyone has chosen to invest in a touch screen phone and their data subscriber plans (about $80/month in Canada). Second, the imagery in the work features a cultural celebration that is somewhat out of sync with contemporary media culture – i.e. folksy, amateur musicians – and seemed incongruous with a slick, technical presentation. Finally, the separation of the recorded message from its functional value allowed it to take on its own poetic role within the overall experience. Once users figured things out and became engaged with controlling and altering the large public images, and once they had shared the experience with friends and strangers, they would often return to the spoken text on the phone, as though to a little private performance in their ear. As with Tentacles, Trio uses the handheld device to create a small scale, private experience within the larger shared public interaction.

So many interactive installations rely on sensor input (rather than device control) that it is not uncommon to have young people walking up to a screen and start waving their arms around expecting it to respond. The Art of Waiting chose to use in-floor sensors for user input for reasons related to the specifics of the user group. The installation is to be installed in the clinic waiting area of a children’s rehab hospital. Users of the waiting area range in age from infants to 18-year-olds. Many have physical or cognitive disabilities and most are waiting with their parents or an attendant for a medical examination, treatment or diagnosis. The goal of the project was to create an interaction that would be accessible to almost all users of the space, and that would be a calming activity in a potentially stressful situation. [5] Relatively few of the users were likely to have cell phones or other devices. Switches and touch screens were considered too inaccessible to users with limited motor abilities and also created risks associated with infection or contamination. Motion sensors typically use some kind of camera which was considered inappropriate for a sensitive medical facility. The floor sensors were chosen as the most accessible means of input, regardless of ability. A grid of one hundred 30-centimetre square tiles is installed under a carpet in an area bordered by chairs. With this configuration, every visitor to the space is automatically providing input to the system. Even a passive engagement – merely being present – causes actions and transformations to take place on the projection screen.

Scale = Public = Shared = Social

The interplay of scale in the first two installationsthe small screen in the palm of one’s hand contrasted with the large public screen on the facade of a buildingparallels other central human experiences. The intimacy of touch, for example, is contrasted by the dominance of projected, broadcast visual stimuli, while the screenthe signforms a kind of text waiting to be read. Your personal space simultaneously shrinks and expands as the tiny gestures you make with your fingers are magnified for all to see.  Public and private stand in stark contrast, highlighting dichotomies like wireless and wired, perception and cognition, knowing and being.

Operating from within the crowd, viewers or players had the opportunity to step onto the stage of the projected environmentto display themselves in action, engaged with other virtual beings. Movements, gestures and displays become part of this spontaneous public performance, suggestive of the activity on a dance floor, where typical rules about decorum, reservation, engagement with strangers and physical contact are suspended. Each private, gestural experience is amplified publicly as a by-product of being within a crowd.

Taking action in public in this way constitutes one layer in the creation of community. Our behaviours and others’ meld to generate simultaneous effects, creating a joint awareness that forms the cornerstone of our collectivity.

In all three installations play is presented as a free-form, creative activitya childlike enthrallment with exploration, skill-learning and sharing. The scale and location of the displays encourages parallel play and the growing awareness of the activities of other players nearby. The public nature of the experience creates the opportunity for ambient performance, where other players’ awareness of you subtly influences and rewards your behaviour. Finally, these factors combine with the ambiguous structures and activities built into each project to encourage social play and collaboration in an emerging, shared activity.

Gamesor rules-based playemerge later in life and become the standard in the adult world. But the works presented here offer a simpler experience to their usersone that is direct and immediate.

References and Notes: 

  1. Tentacles' official Web Site. (accessed September, 2011).
  2. Geoffrey Shea, Trio, 2010, (accessed September, 2011).
  3. E. Biddiss, P. McKeever and G. Shea, “The Art of Waiting – Interactive Displays in Healthcare Settings” (paper presented at CHI 2011, Vancouver, May 7-12, 2011). (accessed September, 2011).
  4. P. Jessee, H. Wilson and D. Morgan, “Medical Play for Young Children,” in Childhood Education 76, no. 4 (2000): 215-218.
  5. C. Spielberger , R. Gorsuch and R. Lushene, STAI Manual for the State-Trait Anxiety Inventory (Palo Alto, CA: Consulting Psychologist Press, 1977).