media art

T/Act - social empowerment through interaction with media artworks

This paper presents results from research made through a collaborative design process with selected individuals with severe physical disabilities. The work encourages and enables creative expression by the participants beyond everyday norms. Can a disruption of institutionalized conditioning according to class, education, gender and physical abilities be orchestrated by careful design and presentation of interactive artworks?


Our current lifestyle is focused and reliant upon media technologies. Our lives are organised through and by technology, such that we can easily forget the importance of physical social interaction rather than which are mediated by online social networks. Instead of being empowered by technology, humans are enslaved to its seductive powers. Is it possible to move away from this focus on the technological and rather discuss the act of using the interface and the product of that action and the content? Does access to media technology in itself empower the participant, particularly if that person is herself on the margins of society? The Eye Writer project is a superb example of open source media technologies being used to empower a specific individual (Tempt One) and others with a similar disabilitating disease (ALS). [1] As Tempt One himself states:

“Art is a tool of empowerment and social change, and I consider myself blessed to be able to create and use my work to promote health reform, bring awareness about ALS and help others.”

It is clear that the act of empowerment for Tempt One comes through a combination of access to the technology, the ability to once again create graffiti art, and his possibility to have a presence in the public city environment through the large scale urban projections of his tags. Each element is very specific to the individual in question. In the research described in this paper the author attempts a broader area of investigation. Can the use of media technologies enhance the possibilities for people with disabilities to express themselves creatively on equal terms with able bodied people?
This paper presents ongoing research into the effects of physical interaction with audiovisual systems through a discussion of the results and observations from collaborative design workshops organised for a group of people with disabilities. The author, as a media artist, had not considered working with people with disabilities until a visit by a group of students from Beaumont special school to the Lanternhouse International arts centre in the north of England where he was undertaking a residency. As these students with severe cerebral palsy were encouraged to touch and interact with the installation which was on display for them, it became apparent that the colour, form, sound and overall interactive environment they were confronted with provided a powerful and provocative stimulus, causing emotional reactions which surprised their carers. A follow-up visit to the college showed that although well equipped with musical instruments, media and audio software, most solutions were generalized rather than individually tailored to each student’s needs. This approach may work for the able-bodied person where we all have approximately the same physical abilities, but for a person with disabilities this can be totally inappropriate and very frustrating for all involved. Together with musician Alan Fitzgerald the author proposed to develop bespoke electronic interfaces for a small group of students. In particular it was hoped to examine the following question: If a unique interface is created specifically for a particular individual, can an examination of the use of this interface lead us to answer questions regarding interface design in general? Unfortunately at the time it was not possible to carry out this project in England, but since the beginning of 2011 the author has been investigating similar themes through participatory design workshops with people with disabilities belonging to the Taika Dance group in Turku, Finland. The majority of the participants are electric wheelchair users and have severely limited use and control of their physical bodies, while some have more mobility. They have their own social networks, yet as a whole they can be regarded as on the margins of society with little voice or visibility. Does access to media technology and the ability to create visual and audio performance lead to a wider social empowerment in society for people like these with disabilities? Does the same effect happen for the wider public at large when they are able to interact deeply with a media art work?

Through a participatory design process, the aim of the workshop sessions has been to develop personal interfaces which might be thought of as bespoke electronic musical instruments made for each individual. Due to the practical difficulties involved with all aspects of the collaboration – logistics, communication, and basic bodily needs – progress has been slow, but fruitful. As this group of people have had no prior possibility to make sound or music, the process started with getting to know each other via “off the shelf” solutions. A midi keyboard and controller were used to provide an immediate experience of actually creating different sounds. Using Max/MSP and Reason software, samples and sound parameters could easily be modified. Sounds were also recorded from the participants own voices and mobile phones to use as samples. Even at this simple level, the experience of hearing one’s own voice played back and modified to create interesting or weird sounds was stimulating for the group. Participants soon felt confident to contribute their own ideas and suggestions for the sounds.

The next level of interaction involved gradually introducing different types of electronic sensors and interfaces, allowing the participants to experiment and play with sound in ways that were totally new for them. The author is familiar with using analogue sensors for data collection, interfacing through the Arduino microcontroller to PCs. Now it was necessary to develop methods of using the electronics so that they would not restrict the users’ limited physical movements. Fortunately there are many small footprint solutions readily available on the market. The selected solution was to use short range radios to send the data to remote PCs. The X-Bee radio together with an Arduino Fio has so far proven to be the best solution, as radios can be networked to send data simultaneously to one PC. The type of sensors used range from simple flex and pressure sensors, accelerometers, and compass modules, to perhaps the most useful, the 9 DOF Razor IMU which provides angle of orientation data in all directions. [2] The emphasis on hardware development had been on the novel use of existing electronic components and not the actual development of new technology per se, although this does include the creation of custom sensors and switches using soft circuitry for example. The exploitation of small wireless devices means that the usual restrictions caused by signal wires are removed, and any impediments to the physical body are minimized. The approach used is to concentrate on the movements that the participants are able to make, rather than design an interface that they would have to adapt to.

The focus is on ABILITY rather than DIS-ability. They play according to their own abilities, and can focus on developing that skill. The aim is to discover appropriate forms of interface and sound according to each person’s physical abilities and musical interest. The dynamics of social interaction between the members of the group is also mediated by the technology. It can be observed that there is an eagerness to be the one performing. At the current stage of the project only one or two people have been able to use the interfaces simultaneously. Now that the physical abilities of each of the members have been understood, appropriate personalised interfaces are under development.

As much as possible the motivation for the design of these interfaces comes from the participants themselves as they experiment with the prototypes. One example is a control interface made as a cushion for a wheelchair user – she can control media and play sounds by shifting her weight on the chair. Made with Arduino and Open Frameworks, the interface is very sensitive, intuitive and fun to use. It can be thought of as a dance mat for wheelchair users, yet it is equally useable by the able-bodied. This is at the core of the research: through the development of new media interfaces for a small group of very particular people, gain insight into empowerment through human interaction with audio visual systems in general. Even though the participants have sensory systems different to the regular population, the goal is to make this difference invisible through the medium of the art performance. With the Taika Dance group the aim is to perform publically at the end of 2011.

The use of computer mediated technologies opens up further possibilities for social interaction. Networked technologies, such as video, audio and telematic control of devices allow these physically challenged participants to interact with others over large distances (such as Finland-UK). There is the potential to enable people with disabilities to collaborate remotely and perform highly advanced works to a geographically dispersed public audience. The use of telematic and virtual spaces allows flexibility in developing personal navigable space for each participant – finding the comfort zone for each individual is extremely important when they may not feel comfortable exposing their physical self to a live audience, but a tele-mediated performance maybe an exciting and liberating alternative. The author can foresee other groups of users/participants such as older people making use of these same systems to create their own networked performative works, mixing the security of their personal space with the empowerment of performing to a virtual audience online.
Collaborative performance shifts interaction and participatory behaviour onto a social level. The research aims to develop a methodology for observing the changing role of creator-interactor-viewer and the effects on the social interaction of the participants. How does narrative structure and a shared sense of social space lead towards development of temporary community? In the case of the Taika Dance group, the participants are already known to each other, but through the performative act they are able to transform their own self-image and their perceived role in society. They become activators of their own destiny for that moment in time – they are no-longer abject objects on the margins of society but proud performers in their own right. These works enable investigation of enactive engagement in collaborative activity with playful, participatory artworks, environments and performances. These include accessible and easy use – easy control interfaces that give inexperienced users control over creative acts and allow them to explore artistic experience through their natural body movements and perceptually guided actions.
The dialectical method facilitates the benchmarking of the generalist approach with that of the highly defined individually focused approach. By focusing on people with special needs (brain damage, physical handicap) in this case, the research adds to the discussion of reactions to interaction stimuli and control in the average adult human. Just as the blind person’s sense of hearing is amplified, so it may be that someone with severely limited movement can actually have an acute sense of control over a range far too limited for the normal person to perceive. Work by Saranjit Birdi with special needs patients in the UK supports this proposition. [3] The bespoke device or environment designed for the individual also acts as a window into their world, as we are able to experience the physical or virtual world through their interface, their experience. In particular Merleau-Ponty’s discussion of the body schema illustrates how examination of a unique individual helps us to understand the wider landscape. [4]

As is alluded to in the title of this paper, the motivation for the research is to understand if and how social empowerment can be orchestrated through interaction with media artworks. Can a disruption or disturbance of institutionalized conditioning according to class, education, gender and physical abilities be affected by careful design and presentation of the interactive artwork? It is vital that the interactive experience invites and encourages social interaction between the participants themselves, as it is only through social activity that the self-image can be positively developed. Can the artwork create a community of presence, an opportunity for living in the moment leading to unpredictable (inter)activity within the social group? The artistic TAZ (Temporary Autonomous Zone) acts as a revealing agent within society using the tools of poetic terrorism to disrupt the status quo. [5] Hakim Bey’s concept of the Temporary Autonomous Zone has been proposed by Geert Lovink as a model for network based communities of interest. [6] Having worked extensively with 3D virtual communities in the past, the author can say that the behaviours observed in physically interactive environments can be identical to those seen in the TAZ of virtual communities. The physical artwork (environment, installation) becomes a point of focus for social interaction AND empowerment, as the normal rules of engagement within the public (museum) space are temporarily ignored in favour of those created by the participants themselves.  We are forced to reappraise the traditional models for spectator vs. artist, as new tools and technologies allow the barriers to interaction to become transparent. The role of the artist or designer changes to become that of a facilitator or producer for a larger group of participants. In fact, the artist creates the situation, and the possibilities for others to bring to life, and accordingly the role of the artist as the author becomes less significant. Curator and theorist Nicolas Bourriaud regards that we have passed into a new “altermodern” era where artistic production is concerned with the weaving of “relationships” between people and things, where the artist “viatorises” objects to build narratives through “post production” techniques – the re-use of artefacts, sampling, a mixing of cultures and signs. [7] The discourse, the social activity, becomes the work itself.

By contrasting the generic with the specific, this research has set out to uncover new information about the benefits, desire and motivation to interact with complex technologically driven systems, as well as proposals for rules and methods for the creation of artistic communities of presence. The work together with Taika Dance encourages and enables creative expression by the participants beyond their everyday norms. The eventual goal is to have an understanding of how to enable deep audience participation in live performative events and interactive environments through their interaction and control of audiovisual and robotic systems.

References and Notes: 

1. The EyeWriter Project website. Free Art and Technology (FAT), OpenFrameworks and the Graffiti Resarch Lab: Tempt1, Evan Roth, Chris Sugrue, Zach Lieberman,Theo Watson and James Powderly. (accessed June 28, 2011).
2. An inertial measurement unit, or IMU, is an electronic device that measures and reports on a craft's velocity, orientation, and gravitational forces, using a combination of accelerometers and gyroscopes. (accessed June 29, 2011).
3. Saranjit Birdi, Thisability (2010) Online documentation and artist statement, (accessed July 4, 2011).
4. Maurice Merly-Ponty, “The spatiality of one’s own body and motility” in Phenomenology of Perception (Abingdon and New York: Routledge Classics 2008, 1945, eng 1962), 112-177.
5.  Hakim Bey, "The Temporary Autonomous Zone, Ontological Anarchy, Poetic Terrorism" (Autonomedia 1985, 1991) (accessed July 4, 2011).
6. Geert Lovink, “The Data Dandy and Sovereign Media, An Introduction to the Media Theory of ADILKNO,” Lecture for the Fifth International Symposium on Electronic Art, Helsinki, 24 August 1994.
7. Nicolas Bourriaud, Relational Aesthetics (Paris: Les Presses du Reel, 1988, eng 2002); Altermodern (London: Catalogue Tate Triennial, 2009).


Dialogues with Decay: Tracing Narratives of Data Space in Pat O’Neill’s "The Decay of Fiction"

This paper outlines the formative dialogues that emerged during production of the experimental film The Decay of Fiction and its interactive counterpart Tracing the Decay of Fiction: Encounters with a Film by Pat O'Neill.  The project was a collaboration between filmmaker Pat O’Neill and The Labyrinth Project–a research initiative on database documentary directed by media theorist Marsha Kinder at the University of Southern California.


In 1993 experimental filmmaker Pat O’Neill was introduced to the Hotel Ambassador.  Built in 1921 in the center of Los Angeles’ Wilshire corridor, the formerly grand hotel was a famous nightspot that hosted the Oscar award ceremonies and became a magnet for dignitaries and Hollywood luminaries in its heyday.  Later made infamous as the site of Robert F. Kennedy’s assassination in 1968, the Ambassador closed in 1989 and was left vacant, its abandoned spaces periodically leased as a location for movie shoots.  Today the site is home to the Robert F. Kennedy Community Schools.  O’Neill was given a tour of its spaces long after its doors were closed to the public and inspired by this urban ruin, he began filming, capturing the way light would move through its surfaces.  This collection of shots (captured using a combination of computerized motion control and time-lapse photography) was edited with a temporary soundtrack from noir films and became the basis for O’Neill’s 2002 film The Decay of Fiction.  While working on the film in 1997, O’Neill was invited to collaborate with The Labyrinth Project (a research initiative on interactive narrative at The University of Southern California directed by media theorist Marsha Kinder) on a digital media project that would be based on his film.  This collaboration resulted in an interactive DVD-ROM published in 2002 called Tracing the Decay of Fiction: Encounters with a Film by Pat O’Neill. [1] This paper will outline the history of how the digital iteration emerged from O’Neill’s film and explore the dialogues that developed while these two forms of The Decay of Fiction were being produced.  

Since 1963 Pat O’Neill has been creating a body of experimental films that cannot be easily categorized as belonging to any singular strand of cinematic style. His work exhibits an intimate mastery of image processing techniques conventionally used to produce special effects in cinema, but the layered landscape of sound and moving images he composes extends beyond the limited language of traditional effects. [2] Rather than creating seamless optical illusions, O’Neill foregrounds the gaps between his densely layered imagery to orchestrate a different kind of illusion – one of unlimited associations in the mind.  A poetics of associative meaning is awakened when watching O’Neill’s films and it is this matrix of imagined trajectories that corresponds to one of the characteristics shared in film and digital media discourse – non-linear narrative.  

During a workshop hosted by The Labyrinth Project at USC’s Annenberg Center for Communication in 1998, media scholars and artists were invited to discuss the possibilities and challenges in creating an interactive non-linear narrative based on The Decay of Fiction.  O’Neill (who had no prior experience working in digital media) was invited by Marsha Kinder to collaborate on this project because their close friendship created a foundation of mutual respect and trust from which one of the first Labyrinth Projects emerged. Moreover, Kinder had written about O’Neill’s work in the 1970s and wanted to collaborate with someone whom she admired and whose work had great potential for interactive database narrative. The goal of the collaboration was to make an interactive work that would be emotionally engaging while both experimenting with and retaining the pleasures of cinematic narrative. Kinder defines narrative “not just as the idea of the well-made story with a three-act structure but…as a cognitive way for contextualizing the meaning of perceptions. It’s a combination of data that’s selected from a bunch of different databases and put together in interesting ways.  And I think Pat’s film have that kind of structure.” [3] In the computer world this modular architecture of non-linear narrative typically takes shape in the form of a database.

In his book The Language of New Media, media theorist Lev Manovich traces the origins of the database to computer science and defines it as “a structured collection of data. The data stored in a database is organized for fast search and retrieval by a computer and therefore, it is anything but a simple collection of items.” [4] This idea of data as a “collection of items” corresponds to the way in which O’Neill collects material for his films, identifying himself as “a kind of scavenger that looks through a lot of existing material and finds items that spatially or in terms of feeling have connections to the basic piece.” [5] The modes of selection made by the computer or by O’Neill, can be considered non-linear in that items are chosen randomly from a constellation of possibilities. The difference, however, is that computer data exhibits no intrinsic value or associative meaning. The machine does not choose, judge or make cognitive connections between items but reduces them all to identical, sterile bits of information.  Although the database evokes a very contemporary notion of computerized consciousness, it connotes a lack of corporeal presence and an absence of the kinds of sensual pleasures found in O’Neill’s narratives. Rather than a database that fetishizes computer consciousness, I would like to imagine an embodied data space – one that is grounded in human consciousness mapped onto spatial trajectories.

The orchestration of space is central to the narrative trajectory presented in The Decay of Fiction. Space takes precedence over action, as opposed to action forming the underlying architecture in most traditional storytelling. The 73-minute film traces a pathway through the decrepit hotel, dripping with the kind of nostalgic traces that abandoned spaces evoke, a pathway that O’Neill describes as a “choreography for the camera.” [6] This feeling of nostalgia is heightened by the superimposition of noir film soundtracks onto the contemporary ruins of the hotel spaces evoking what O’Neill calls “the decay of storytelling or how storytelling merges with the environment or with a space that’s foreign to it but somehow attracts it”. [7] It is as though these stories are written onto the body of the hotel and watching O’Neill’s scenes of sped-up-time, we are witnesses to this decay of fiction.  

Conceiving the network of noir inspired action that would take place in the environment occurred after the empty spaces had already been captured on film. Having recorded the camera moves using a customized motion control system, O’Neill was able to later shoot his actors while repeating the same camera movement. Then through the process of optical printing, the foreground action and background spaces would be combined to form a composite image in which black and white figures inhabit a contemporary landscape shot in color. By compositing the present with the collective memories of a vintage era of Hollywood’s past the Ambassador’s remains become a repository of cultural history and imagined interactions represented in the film by a layer of ghostly fictional characters playing out noir inspired narratives. These narratives are periodically interrupted by animated interludes that seem to emerge from a parallel dimension formed from a repository of surrealistic moments. These parallel spaces converge as the film culminates in a carnival parade of performers whose dance of overlapping bodies blurs the boundaries between past and present. On one level, the film is an imprint of the hotel as artifact – a fossil of the past housing not only the imagined fictions invented by O’Neill but the public and private histories of Los Angeles’ memory.  

In the DVD-ROM, the hotel’s history is an additional dimension that is only hinted upon in the film. In the film, as the camera pans across the hotel’s ballroom we are reminded of the Robert Kennedy assassination as we hear excerpts from the speech he gave shortly before his death. Though this is one of the few moments in the film where historical memory materializes, it maintains a peripheral distance to documentary that haunts the edges of the screen. In the DVD-ROM, as the viewer navigates through the ballroom, additional material about the assassination is made accessible through a click of the mouse. This documentary material includes news clippings, archival footage surrounding the shooting, radio broadcasts of conspiracy theories and contemporary interviews with historians and witnesses offering their insights on this historical trauma. These documents are embedded within the surfaces of the navigable space where they remain hidden unless activated. The film de-emphasizes the “artifactual” dimension of the Ambassador hotel as a repository of history and although O’Neill initially struggled to incorporate his historical research of the space into the film script he abandoned his efforts, realizing that he “wasn’t doing a documentary but a choreographed camera move with action.” [8] Collaborating on the interactive iteration of The Decay of Fiction opened up the possibility to include not only historical material O’Neill had intended to include in his film, but additional material researched by Kinder, myself and others during the four year period in which the DVD-ROM and film were being produced. These included moving image archives of social events and publicity stunts hosted at the Ambassador, photographs of the hotel and its surrounding neighborhood from 1920 to 2002 and contemporary audio interviews providing different and sometimes conflicting perspectives on the history of the area. All these alternate layers of narrative possibility exist on the fringes of O’Neill’s fictional spaces and can be accessed at any time. While exploring the hotel in the DVD-ROM, one encounters a diversity of data spaces and it is up to the viewer to choose which allegorical vector to follow.

Tracing the Decay of Fiction expands upon the uncanny qualities of place and memory that are present in the linear film by transforming the film’s linear spatial trajectory to non-linear spatial navigation or “navigable space” – what Manovich identifies (along with database) as “another key form of new media.” Although Manovich refers to 3-D computer generated virtual space as a model to illustrate the exploration of navigable space, the same description could be applied to illustrate spatial exploration in Tracing the Decay of Fiction. In the hotel, the viewer can activate any of its static interior spaces by placing the cursor over the edges of the screen, animating the still image into motion and following the camera moves that give the illusion of spatial navigation.  Near the end of production on the DVD-ROM, Rosemary Comella invited Manovich to view the navigation system she had designed and programmed for the interface allowing the viewer to move within and between the hotel spaces. Manovich discussed his conception of “navigable space” but admitted he had never seen it realized in this particular way.

Another attribute that differentiates the DVD-ROM from the film is its use of “the image as interface.” Manovich states that “The new role of an image as image-interface competes with its older role as representation.…a computer image is situated between two opposing poles – an illusionistic window into a fictional universe and a tool for computer control.” [9] The role of the image as both “window” and “tool” is illustrated in the DVD-ROM when the viewer pauses on a still while following a camera move through the hotel.  Doorways, walls and windows become links to historical and fictional interludes. However, I hesitate to assert that the use of image as representation and interface are in opposition here. Rather, the immersive beauty of O’Neill’s cinematography is foregrounded and the navigational devices are intentionally integrated to minimally distract the viewer. Furthermore, in the work of experimental filmmakers like O’Neill the image does not always function as  “illusionistic window.” Rather, the illusory quality of the image is frequently challenged using the very techniques used to maximize illusion in the special effects industry. While discussing the industrial apparatus that O’Neill re-uses to develop his own aesthetic, he states, “the by-products of the processes of special-effects work.…that which undermines the illusion. That seems to be a very powerful thing – the illusion and the denial of the illusion, both present at the same time.” [10] Similarly, the binary poles of image as illusion and instrument are not in competition with each other in Tracing the Decay of Fiction, but are “present at the same time”.

Another nodal point of dialogue between the film and DVD-ROM is the differing role that montage plays in each. In the film, the viewer is introduced to O’Neill’s choreographed sequence of narrative spaces – the viewer sees an authored mix and a pathway arranged by its author.  In Tracing, the viewer participates in the mix and editing is replaced by navigation and choice.  In the film, a sequence of moving images is arranged over time while in the DVD-ROM the moving images are spread out into space. The viewer’s choices in that space are arranged into a spatial montage. Spatial montage is made explicit on the DVD-ROM by allowing the viewer/performer to control the “choreography of camera movement” from a selection of rooms in the hotel. By moving the computer mouse over indicators on the edges of the screen the viewer can control specific camera moves or slide into adjacent spaces. Alternately, one can choose a different space by using the original blueprints of the Ambassador’s architect Myron Hunt as a map to locate the spaces to explore on the DVD-ROM.  While navigating the camera moves, the ghost-like characters that inhabit these spaces can be activated over the moving image with a click of the mouse. During the production of the DVD-ROM, there was a debate about whether to include the noir characters or to leave the navigable spaces uninhabited in order to retain the uncanny quality of the hotel and invite the viewer’s consciousness to inhabit its spaces. [11] It was a question of how closely the DVD-ROM sequences should mirror the film. While describing the period when he was first filming the Ambassador, O’Neill said, “as you walk around an empty building – especially when you know about who inhabited it and what happened there – you always expect you’re going to run into these people as you go around a corner. I mean its haunted in your own mind…so it was this quality that I was trying to figure out how to synthesize.” [12] While this haunted quality is made explicit in the film’s layering of ghost-like characters, in the DVD-ROM the viewer can select either option – either she can explore the empty space or inhabit it with characters from O’Neill’s fiction. Periodically, however, the choice of combining foreground action and background space is automatically generated whenever an “earthquake” occurs. At these moments, the viewer loses control of the interface and a random collage is generated from the database of foreground and background elements. By alternating the layer of characters off and on or by viewing a randomly generated mix of multiple layers, the emptiness of the hotel takes on a heightened quality of mystery as you realize in your mind, that there is another hidden narrative frequency that haunts these spaces.

In summary, I have outlined the formative dialogues that emerged during the production of The Decay of Fiction and its digital hybrid Tracing the Decay of Fiction. First I explore the role of non-linear narrative in both projects and show how the complex network of associations created in O’Neill’s films correspond conceptually to the way non-linear narrative is structured in new media. This structuring takes the form of a database, a collection of items, or an index. The indexical nature of O’Neill’s process is reflected in his view that his films are like a journal, a synthesis of disparate units, “a collection of entries all by the same person but at different times and places.” [13] I propose extending the notion of the database to a data space in order to address the role of spatial navigation and spatial montage.  Spatial navigation as a mode of organizing narrative footprints resonates in O’Neill’s observation that his films serve as “a record of an individual who wanders the land and from time to time stops to comment on it.” [14] I also explore the dual nature of image both as interface and mode of representation both in the film and DVD-ROM. All of these dialogues converge in the ruins of the Ambassador and in the end the hotel becomes a metaphor for the exploration of data, narrative, memory and history. Finally, it is a tracing of decay that is embodied in the haunted orchestration of spaces written on celluloid and encoded in digital space.

References and Notes: 
  1. My role in the DVD-ROM was as co-director along with Pat O’Neill and Rosemary Comella.  This involved working on the conceptual development, research and production including the digital compositing, interface and graphic design of the collaborative project.
  2. It should be noted that longtime collaborator George Lockwood contributed his expertise as sound designer to many of Pat O’Neill’s films.
  3. Labyrinth Project workshop, USC Annenber Center for Communication, February 28, 1998.
  4. Lev Manovich, The Language of New Media (Cambridge, MA, The MIT Press, 2001), 218.
  5. Labyrinth Project workshop, 1998.
  6. Ibid.
  7. Labyrinth Project workshop, 1998. The addition of noir dialogue came through the kinds of signature chance encounters O’Neill welcomes in his work.  During early editing, O’Neill was listening to the television when he heard a passage from the noir film “The Big Sleep” and realized that the dialogue fragments had an evocative affinity with the spaces he had shot.
  8. Labyrinth Project workshop, 1998.
  9. Manovich, 290.
  10. David E. James, “An Interview with Pat O’Neill,” Millennium Film Journal, nos. 30/31 (Fall 1997).
  11. From a conversation with Rosemary Comella in 2006.
  12. Labyrinth Project workshop, 1998.
  13. James, “An Interview with Pat O’Neill.”
  14. Ibid.

Precarious Flux

Mixing theory with applied perspectives this paper generates a series of questions and describes how contemporary social technologies have significantly changed our practical reality, a reality where human experience and technical artifacts have become closely intertwined. The paper's conclusion explores the ontological consequences of this change and the potential in establishing 'Precarious Design' practices and methods as a response.



The physicist David Bohm (1980) posited that the “world is full of movement and becoming, in which any thing, caught at a particular moment, enfolds within its own constitution the history of relations that brought it here.”[1] This is an intriguing position and one that in spirit captures the inherent limitations of a singularly defined experience and gives prominence to the notion of complexity and “being in flux.” Some years later and based in another field entirely, design thinker and curator Antonelli (2008) opinioned that “… core human experience is rendered more urgent by the speed at which technology is moving…“and that a great number of us “…routinely live at different scales, in different contexts, and at different settings Default, Phoneonly, Avatar On, Everything Off on a number of screens, each with its own size, interface, and resolution, and across several time zones.” [2] This modern way of existing is often described as appealing, stimulating and empowering. Various social media platforms, digital agencies and technology developers all strive to assist, to connect us through these multiple interfaces and time zones. The corporate/civic/personal rhetoric of social media is driven by the positive (see Fig 1.).  However a few notable counter voices have arisen. The Baroness Greenfield inspired a fierce bout of media anxiety in April 2009 after she published an article in the Daily Express titled “How Facebook addiction is damaging your child's brain: A leading neuroscientist's chilling warning” in which she claimed that there were (probably) lasting neurological effects from frequent exposure to social media websites.  Since 2007 there has been increased pressure from the American Medical Association for the American Psychiatric Association to include internet addiction, video game addiction, e-mail/text messaging along with sexual preoccupations in the upcoming 2012 Diagnostic and Statistical Manual of Mental Disorders (DSM V), the standard diagnostic text used by psychiatrists worldwide, and on the 16th of February this year Physiologist Kathy Charles, writing in the New Statesman, likened once again the social network to something compulsive and destructive, claiming that: "Facebook keeps users in a neurotic limbo, not knowing whether they should hang on in there just in case they miss out on something good."  This paper will attempt to avoid the techno triumphalism of being better ‘connected’ and similarly the paranoia around digital media’s so called deleterious mental effects on users. An idea that will now be revisited and debated in British parliament after many sections of the British press blamed social media arenas for aiding the London, Birmingham, Nottingham, Bristol and Liverpool city looting problems on the 6th, 7th and 8th of August 2011. Rather this paper accepts that we are now in an age where cultural and technological change has created a new reality of sustained rather than temporary movement. Although Bohm and a plethora of thinkers, scientists and artists have either instinctually known, or through intellectual observation made peace with this worldview, this paper shall explore the problems and potentials that arise when applying such a notion to the everyday user of technology in society. 

Context Theory

The majority of users remain uncomfortable around confusion, and if we do encounter it we still expect confusion to be of a fixed and short nature, it’s a basic tenet of the knowledge is power aspiration. Human Computer Interaction (HCI) was a field that historically had not been concerned with representing complexity or mirroring the precariousness nature of our existence. Rather HCI’s goal was to gain user satisfaction, to make digital tools more receptive to our needs. Media Art arguably has never contained any stable goals or ideologies and is populated by a mix of pioneering creative technologists (Robert Hodgin, Jer Thorp), obsessives (Joshua Davis) and niche collectives (Antirom, c5corp).

There is a danger associated with being in flux, it suggests change, pressure, and movement even Bohm’s ‘becoming’ implies an end rather than an endless becoming. A fundamental of good HCI design was to eliminate or reduce user confusion, to allow us to be stable, to have permanence, to perceive and connected to and transact with the systems and devices that we need. User legibility was preferred over ambiguity. Design thinking was all about making things better a form of modernist 'heroics’. Recent developments in the field have seen a turn towards experience design, described as a situation where experience and technology are intrinsically enfolded. Lucy Suchman an anthropologists specializing in the digital described the “relations of human practice and technical artifact [have] become ever more layered and intertwined. At the same time that the technological project is one of congealing and objectifying human activities, it is increasingly also one of animating and finding subjectivity in technical artifacts. The assimilation of lived experience to technique goes both ways, which only makes the project of re-imagining technological objects the more urgent.” [3] This turn to experience with a need for engagement with dense multidisciplinary methodologies will inevitably expose the field to the poststructuralist problem of endless subjectivity. In traditional dialectical tension to Design, Art has traditionally functioned as society's most deliberate and complex means of self-expression and as such is comfortable with subjectivity. When artworks follow anti-hierarchical ideologies (post Dada, Fluxus et al.) a certain level of dissonance can be felt by the user as they struggle to identify what are the expectancies of them given the lack of explicit rules. The ensuing paradigm of uncertainty, ambiguity and ambivalence in early twentieth century Art preceded Bauman’s turn of the century notion of ‘liquid modernity,’ [4] a new modernity in which fractured timelines were normal, where social structures were no longer stable, and a state of being where fixed concepts like ‘career’ and ‘progress’ could no longer be meaningfully applied.

Context Corporate

Today’s torrent of societal change and unrelenting uncertainty has left many industries sluggish and effected many corporate identities (consider the state of journalism, publishing, music distribution, retail). The speed to change is recognized (no longer new) but many are now feeling the impact of this sustained change. For Designers questions now arise about their role in solving societal problems and how do they confront the idea that a theory of everything is needed, that everything must be considered before anything can be addressed? The traditional path to a clearly defined problem and solution becomes a challenge, perhaps even futile in this period of dense movement and uncertainty.

Context Ludic

Cultural theorist Huizinga (1938) conceptualizing play stated that within a game you are “…spatially and temporally segregated from the requirements of practical life”. [5] Whereas now to be spatially and temporally segregated (from workmates, lovers, family and friends) are now common requirements of practical life. Perhaps there are merits to re exploring the ludic discourse above and beyond the past the preoccupation with narrative aesthetic and revisit the psychology of role playing. Role testing or  playing is expected to be transitional - done in our youth, but within the social network and digital gaming contexts we can extend this process. What happens if you can’t manage our new ‘practical life’? In this scenario what does suspension of belief and or self now mean. The gaming analogy can also be distinctly felt within the observations of Jenkins [6] who described a move away from transactory culture into participation where play is becoming a default method in engagement and knowledge attainment (and almost universally seen as a good thing).

Context Social Skills

Western society has been through an adjustment; we have adjusted to accelerated change, hopefully learned most of the tools required for this new practical reality. Graduating from the novice state towards the intermediate level we psychologically lean on our tools (Charles 2011) to such an extent that dissociation anxiety has become a popularly understood term  describing our contemporary difficulty when we do not have access to our connective technologies. There have been recent attempts at convergence, by bringing together all our feeds, our emails, texts, tweets into one interface. [7] This attempt to unify the users experience is a logical but perhaps anachronistic goal when each media instance fosters a different cognitive connection, simply blending these mental conditions this may not be a unifying reductive solution. We ‘need’ these tools to provide different things for us: acts of sharing (Blog, Twitter, Podcast, MySpace, YouTube, Flicker, Vimeo), discussion (Twitter, Newsvine, StumbleUpon, Youtube) and connecting, re connecting (Facebook, MySpace, LinkedIN, Friendster) are all distinct practices. Each of these different functions requires significant effort to immerse and different commitments in terms of assessing, changing and publishing content; each interface (after all conceived as a discrete experiences) cues the user into different mindsets. Bringing them all together may result in a useful Meta view of activity but perhaps not in itself a unifying experience that can solve the more fundamental mental and emotional conditions of confusion, noise and anxiety.

Context Depth

Mike Bergman, credited with coining the phrase ‘DarkNet’ has said that searching on the Internet today “can be compared to dragging a net across the surface of the ocean; a great deal may be caught in the net, but there is a wealth of information that is deep and therefore missed”. [8] Intermediate media users can quite effectively create a closed private network of devices used for file and content sharing such as the encrypted messages sent via BlackBerry to various mobs during Britain’s August 2011 riots. There is also evidence that users are finding new modes of communication and semantics [9] such as the increase in personally curating our entertainment and nested linguistic meaning. Whereby a message or sentiment is embedded, disguised or hidden within a linked text or video, which can only be truly understood by select users who are aware of the specific total (online and offline) context of the user. The deep and cognitive Web is several orders of magnitude larger than the surface or representational Web. This level of subjective and structural complexity means that the Internet still constitutes a free activity where we can move around, sign-in, explore, search, look, understand and comment without a sense of sanction. What then if anything constitutes expert usage, traditionally denoted as prolonged intense practice through experience and education, in a world full of multiplicitous digital experience? Anatonelli (2008) addressed the role of design in a world which humans have surpassed their Enlightenment roles as neutral observers and have become ‘actors on the very forces of nature’. Implying that to effective one must be active within the complexity. She also saw a need for users to develop personal elasticity, that being: “the by-product of adaptability and acceleration, elasticity means being able to negotiate change and innovation without letting them interfere excessively with one's own rhythms and goals.” This paper also considers if to be expert now means to be agile and surface than deep. Huizinga’s (1938) seminal quote goes on to extend the description of games in that we are also “bound by a self-contained system of rules that holds absolutely.” As stated acceleration of the Internet and its online culture left the corporate world far behind, conventional advertising strategies were ineffective, the environment was to a large extent unstructured. New rules were slow in coming; what remained meanwhile was self-governance. We explored our user role and sense of self in a freer system of ethics and behavior – many used alter egos, role  playing and exploration of promiscuity and the taboo [10]. What constitutive and regulative rules now control our behavior?

Context Semiotics

If we remain in a gaming mindset, then the conventional wisdom is that life shall intrude, that there is a porous magic circle. In offline, online and everything in-between people are crossing this reality / non-reality threshold all the time in both directions, “carrying their behavioral assumptions and attitudes with them” (Castronova 2005). [11] Practically testing or breaking through the offline and online bubble has resulted in some contentious legal interpretations such as the recent conviction that Paul Chambers received for his ‘tongue-in-cheek’ tweet about blowing up Robin Hood airport in Britain back in January 2010. A vigorous debate has ensued around appropriate contextualization of Chambers actions. The presiding judge interpreted that Any ordinary person“ would interpret the tweet as alarming. [12] The notion of ordinary is now in itself a fraught concept. What is ordinary social insight when at different scales, in different contexts, and at different settings? There are still no clear regulative rules that prescribe acceptable social conduct/communication within social media (the UK Digital Economy Act 2009 is mainly interpreted as copyright protection). The Chambers case shows how problematic words without appropriate context are.   

Context Physical

Service design and codesign go some way to addressing the HCI context of complexity but from a procedural and often corporate view. When attempting to congeal and objectify contemporary human activities is not surprising that Design has moved away from an industrial to emotive centered approach. An excellent example of this is the ‘The We Feel Fine’ project (, an emotional search engine started in 2009 whose goal is to collect the world’s emotions to help people better understand themselves and others. Having come through the other side of dematerialization, re-materialization is also becoming more prominent. The uptake in programming projects such as Processing (Java) and openFrameworks (C++) by non-computer scientists is making engineering physical and digital interaction more achievable (see ‘Pigeon d’Or’ by Tuur Van Balen). Another example of rematerialization is Tim Kring’s augmented reality game ‘Conspiracy For Good’ which confidently attempts to make a virtue out of the game fiction / social reality divide. This paper proposes that such practices could be considered as Precarious Design.  

The Precarious Designer

When asked to reflect on the art of the first ten years of the millennium, art critic Hal Foster [13] focused on the ‘precarious’, art which functions as a social political critique, work which foregrounds its own schismatic condition, its own lack of shared meanings, methods, or motivations, Art if you will, that captured a sense of cultural vertigo and liquidity. Applying Foster’s description, a Precarious Design paradigm could be a community of precarious designers who create experiences and or artifacts from a position of living and observing and testing within acknowledged and accepted precarious contexts. Such a designer accepts acceleration, recognizes the fluxing user position by being one. As with Foster’s precarious art precarious designers can function within a post – conceptual space where there is no distinction between works of self-expression and works of social critique (i.e. they are part and parcel of the same activity). Precarious design by collectivizing or collating works could also give life and voice to the broader fluxing context, objectifying places within the complex digitalphysical continuum of our current reality a chaotic continuation of uncertainty.


Both applied and artistic practices are striving to synthesize and express what constitutes a core human experience and develop methods to survive and succeed within our fluctuating context of sustained extraordinary change. In a sustained world of acceleration the aims of design become interesting. If we fully embrace Bohm’s implicate possibilities then, as Suchman writes: “Integration, local configuration, customization, maintenance and redesign on this view represent not discrete phases in some ‘system life cycle’ but complex, densely structured courses of articulation work without clearly distinguishable boundaries between.” [6] User experience should no longer be explored in terms of a singular moment but also over longer periods, or indeed we need to consider that the different interfaces work as differing forms of personal histories. What then becomes significant is establishing what people are actually doing and what people need to do.   

Problematically this practical reality is both without perceived sanction and seems to offer limitless individual agency; however, we are not free of corporate or political and legal influence and ramifications. Users need help in delineating new cognitively useful, safe and or dangerous personal and legal boundaries. Given the cognitive freedom of the ersatz ludic space, existence feels quite different. Without rules our identities and ontologies need support. In addition to the recent strategic investment of the designer as facilitator or conduit in multidisciplinary methodologies (Britain’s Design Council), this paper offers up the notion of the Precarious Designer, who by way a personal insight via a personal/niche epistemology, is well positioned to conceive of these new expressions and being lighter of foot is able to dance along with the inevitable redefining moments within society and technology. 


References and Notes: 

David Bohme, Wholeness and the Implicate Order, (Routledge, London, 1980), 149

Paola Antonelli, (ed.) Design and the Elastic Mind, (New York: MoMA, 2008)

Lucy Suchman, May 2010,  

Zygmunt Bauman, Liquid Modernity (Polity Press, Cambridge, 2000)

Johan Huizinga, Homo ludens,  (Routledge, London, 1938).

Henry Jenkins, Confronting the challenges of participatory culture, (MIT Press, Boston, 2009).

Tariq Tahir, May 4th 2011,

Michael K. Bergman “The Deep Web: Surfacing Hidden Value” JEP, Vol 7, Issue 1, (2001)

John Naughton, May 2011,

Emma Norton, March 2010,

Edward Castronova, Synthetic Worlds: The Business and Culture of Online Games, (University of Chicago Press, Chicago, 2005)

Robert Booth, Sept 2004,

Hal Foster, “Precarious”, 2009,

(All URLS accessed 12/8/1)



Network culture, media art: cultural change dialectics

The emergence of Network Culture represents a fundamental paradigmatic shift in society.Networked connection replaces abstraction in terms of cultural production.This paper investigates the influence of Actor-Network Theory as discursive strategy that explicates public spheres such as the New Media art world  creating a new form of Virtual social space. 


 The emergence of Network Culture represents a fundamental paradigmatic shift in society, as it resituates the concept of mediation as a default condition, in everyday life and the art world. Here, networked connection replaces abstraction in terms of cultural production. Actor Network Theory redefines the relations of all elements in a milieu in terms of their effects on the system, and not on their base significations. This results in a heterogenous semiotics of symmetry, where the material, intellectual, and social effects of agents within a network differentiates their degree of agency. This can be seen in terms of Latour’s systemic “artworld”, consisting of a fractal aggregate of subordinate sites, such as galleries, festivals, websites, and blogs, as well as various overlapping social spheres of influence. This paper will investigate the influence of Actor-Network Theory as discursive strategy that explicates public spheres such as the New Media art world as creating a new form of Lefebrian social space, that of the Virtual.  This will be done by considering two New Media art works, Face to Facebook and Wikipedia Art as examples of cultural production that utilize virtual space as site of social agency under the ANT paradigm.

The Emergence of ANT

It has been said that with the rise of network culture, that the shift in focus occurs from the online community to that of the “smart mob”; that this represents a decentering of concentration from the concept of located/dislocated (place-based and place-less space) to that of a flat, relational network.  In Science in Action, [1] Bruno Latour theorizes the concept of Actor-Network Theory, in which he deals with agents in a network of relations as being mediated, symmetric, ubiquitous, and material-semiotic. Ubiquity is assured in that there is nothing outside the network; once someone or something engages an ANT network, it becomes an actant and therefore part of the network.  Material-semiosis relates to the actant’s relevance having to do not only with its signification, but also with its material circumstances. For example, someone contrasting silk versus nylon hose as social intermediaries would not deal solely with its metomyny as signifying class structure. But taking both as mediators through the material circumstances of silk versus nylon and thus constructing meaning from this as well, these objects become relevant. This is merely to say that meaning in an ANT network is a constant state of socio-cultural mediation and negotiation and is dynamically dependent on the circumstances of the whole network and its architectonic of meaning, rather than merely with its base significations. Latour also states that elements within an AN are regarded as equivalent, or described in the same terms. This includes human, non-human, and material actants, and therefore creates a ‘flat’ space of signification, and does illustrate networks of equal actants, such as flash mobs or online communities (as in our examples), rather than systems that derive difference from less complex sets of criteria. 

In terms of ANT, perhaps we could conceive of the network as an heterogenous infinitude of equivalent spaces, but it may be useful for us to consider the nature of constructed space within an AN in order to consider artworks that engage networked culture.  Henri Lefevbre, in The Production of Space, [2] posits a unitary theory the division of space into three types, the physical, the social, and the mental. These are derived from Aristotelian “Becoming” to Kantian space to contemporary epistemology as theoretical underpinnings of his construction of space. He critiques Chomsky and Derrida is not addressing the mediation of the "the abyss between the mental space on the one side and the physical and social spheres on the other." [3] In our case, we would like to suggest that, given the need to bridge this abyss, that we could theorize a fourth space; a Lefevbrian Virtual space as being the combination of the mental and the social. This is the space of our given artworks.

But what of the environment-net they inhabit? To construct an AN within which works like Face-2-Facebook and Wikipedia Art operate, we will look at Howard Becker’s conception of an “art world.” [4]  Congruent with ANT, Becker asserts that the work of art is a system within a system which is the product of a set of a complex sociological propositions that are dependent upon one another, which we could consider as analogous to an AN. There is no homogenous “art world” per se, but heterogenous fields of spaces, milieu, individuals, and works. If we allow ourselves to extrapolate to the virtual space in our network, we can see that this expanded artworld exists as a series of overlapping and nested spaces (example, an exhibition is situated by being on a certain website or in a certain gallery, and is then contextualized by the curator and critics, and seen by the audience). There are not just overlapping spheres of museums, galleries, fairs, curators, gallerists, critics, patrons, visitors, but also social media, blogs, forums, maillists, and tweet feeds.  Furthermore, our examination of artworld-as-network only denotes the mediated relevance of the milieu (spaces virtual and/or otherwise) merely as construct or our range of interest. The art world, per se, exists within the expanded fields of society, so stated not to imply it existing in social space, but art world as embedded in that larger environment. As such, we could expand the AN to encompass the globe, its societies, all objects, and their dynamic relations. For the sake of discourse, it is assured that this is beyond the scope of our inquiry, and we shall remain largely in the criteria of examining a limited art world network in a virtual space.  But even so, we will see that our examples (Face2Facebook and Wikipedia Art) bleed outward from their points of origin into the larger Lefevbrian spatial landscapes. 


Face-to-Facebook [5] is a project by Paulo Cirio and Alessandro Ludovico that “stole 1 million Facebook profiles, filtered them with face-recognition software, and posted them on a custom-made dating website sorted by their facial expressions characteristics.” Face-to-Facebook “screen scraped” various data (name, country, groups) along with profile pictures from which a mock dating site called The artists create a virtual function of agency by harvesting the million faces (social) and processing them to create relations between them (Mental), for a critical Website for wider viewing (virtual agency). Cirio and Ludvico make explicit the theory of agency relating to the idea that facial cues are chosen and used as markers for sexual attraction, although this is a basic metric of mediated relevance in the AN. On Face-to-Facebook, the faces are analyzed using facial recognition software and grouped based on arbitrarily categories determined by the artists, "climber", "easy going", "funny", "mild", "sly" and "smug".  In Dan Jones’ essay, "The Love Delusion," [6] states that “men typically overestimate the sexual interest conveyed by a woman's smile or laughter”, grouping the faces creates a dialogue by implicit discourse of sexual attraction and explicit critique of privacy in social media.  Cirio and Ludovico create a critical Actor Network in virtual space as critical device to explicate to make visible the cultural terrain created by social media. Furthermore, the impact of the site on larger networks is shown by the degree of press the site has created, although this is only a visible indicator of the relational activity within this Actor Network construct.

Wikipedia Art

Another project that plays with differing levels of agency in virtual space is Kildall, Stern, et al’s Wikipedia Art. In Wikipedia Art, [7] Scott Kildall created a performative citation in virtual space as a combination of a mental statement (citation) in the social space of Wikipedia, creating a virtual gesture.  It was placed on Wikipedia as “art composed on Wikipedia, and thus art that anyone can edit”.  Its header reads:

“Wikipedia Art is a conceptual art work composed on Wikipedia, and is thus art that anyone can edit. It manifests as a standard page on Wikipedia - entitled Wikipedia Art. Like all Wikipedia entries, anyone can alter this page as long as their alterations meet Wikipedia's standards of quality and verifiability.  As a consequence of such collaborative and consensus-driven edits to the page, Wikipedia Art, itself, changes over time.” [8]

This listing lasted for a full fifteen hours until its deletion from the Wikipedia site, but not without widespread discussion throughout communities like, [9] Art Fag City, [10] The Whole9, [11] and others. The importance of Wikipedia Art was not so much the work itself but its gestural aspect as networked performance that questions the social networks of consensus.

If we could look at Wikipedia Art as an Actor Network, the project would appear as a series of subnets nested within/overlapping one another as a series of spheres of agency or influence. First, the initiators, Kildall and Stern, represent a home node in the network, with rhetorical conspirators (Sherwin, Coffelt and Lichty) representing another sphere. Socially, the project engages the Wikipedia community and the Beckerian art worlds of the online community and Whole9 and Art Fag City blogs, which in turn have bled to many other venues such as the Transmediale festival and London’s HTTP Gallery. Considered as a larger “art world” aggregate, this network influenced the larger society by invoking the rage of Wikipedia founder Jimmy Wales, and gaining notice of the Wall Street Journal. [12]


While we have explored the concept of ANT, Lefevbre’s division of space, and how the two contribute to the construction of “art worlds”, the emergence of networked culture creates unique modes of artistic expression. These consist of taking a metacritical role in examining virtual (mental/social) space as medium through the intervention of social media.  Face-to-Facebook, Wikipedia Art, Google Will Eat Itself and others probe networks of agency to critically show the shape of networked culture.  The fact that it does have a tangible effect reveals the reality of modes of agency within the network, and reveals the critical landscape of the Actor Network.


In closing, this essay has sought to explore Latour’s concept of Actor Network Theory (ANT) to describe a paradigmatic shift. That shift is from that of locative/embodied discourse to that of purely networked culture; that is, the shift from associating culture with places and things to purely that of symmetric networks of equivalent mediated agents.  As opposed to the widely conceived hierarchical nature of the artworld, ANT conceives of “art worlds” as conglomerates of spaces, ideas, and social contracts that define milieus and works. Artworks described by ANT critique virtual milieu by engaging artworlds-as-networks, and as such explore a virtual Lefevbrian space by bridging the abyss between the Social and Mental spaces through engaging online communities. These critical pieces explore the relations between actors in social situations in context of “art worlds”, with the tangibility of result of their agency being the response in the general audience or public sphere of the mass media. We can examine, using ANT, networked art such as Face-to-Facebook and Wikipedia Art, and as such, describe networked cultural production in its own terms, as demanded by ANT. The rise of networked culture has created a fundamental network shift within society, and ANT is a valuable tool in understanding the relationships created by art within networked social environments, and networked culture as it expands into the future.

References and Notes: 
  1. B. Latour, Science in Action: How to Follow Scientists and Engineers through Society (Cambridge, MA: Harvard University Press, 1988).
  2. H. Lefevbre, The Production of Space (New York: Wiley-Blackwell, 1992).
  3. Ibid., 6.
  4. H. S. Becker, Art Worlds (Berkeley, CA: University of California Press, 1984).
  5. P. Cirio and A. Ludovico, Face-to-Facebook, 2010, (accessed Sept. 1, 2011).
  6. D. Jones, "The Love Delusion," New Scientist, March 2007, (accessed Aug. 25, 2010).
  7. S. Kildall and N. Stern, Wikipedia Art, (accessed September 2011).
  8. Ibid. 
  9. C. Moss, "Wikipedia Art," February 17, 2009, (accessed Aug. 30, 2011).
  10. P. Johnson, "Wikipedia Art Lasts All Day!" February 16, 2009, (accessed Aug. 30, 2011).
  11. J. Coffelt, "What is Wikipedia Art?" February 14, 2009, (accessed Aug. 30, 2011).
  12. G. Mijuk, "The Internet as Art," The Wall Street Journal, July 29, 2009, (accessed Aug. 30, 2011).

Aesthetics of Voice

This paper introduces key themes in relation to vocal aesthetics: voice as intersubjective, paradoxical, uncanny, intimate. It asks how voice is determined by and determining of spatial relationships and the way this affects memory and place. The paper explores specific examples from media arts to explore the performativity of voice and an’authenticity effect’ of media voices.


A number of key themes reverberate through the aesthetics of voice: paradox, uncanniness, intimacy, intersubjectivity, performativity, and memory and place. I’ll begin with a brief introduction to these themes, turn next to modalities and techniques of voice in media art, and finally end with a discussion of voice’s 'authenticity effect.'

Theoretical Introduction

Media artists have long been exploring the potentials and complexities of electronically mediated voice. From Yoko Ono screaming and coughing; to Alvin Lucier sitting and stuttering in his room; to Janet Cardiff leaving the room and the building to take us on sound walks; to Susan Philipsz with her locational singing – to name but a very few. Despite the richness of media art and the resounding way that sound studies burst into prominence in the 90s, vocal aesthetics has remained rather sub voce. And, although it has been several decades since Roland Barthes first invited attention with his concept of “grain of the voice,” strangely, little general theoretical work on voice followed.

There was of course Derrida’s significant work in the seventies, when he deconstructed the “self-presence, immediacy, identity, interiority” of vocal speech. [1] While this moved past invocations of voice as true, unmediated and authentic, the attention to vocal speech in relation to writing may well have played into a sort of binary, which inadvertently diverted theoretical attention. In recent years, however, critical writing has sidestepped such a binary, attending instead to an ambiguity of voice, a supplementarity, which resonate with Derridean concepts and draw out potentials of voice on which Derrida himself did not focus.

Key figures in this critical revival are Adriana Cavarero, Mladen Dolar, and Steven Connor, who share a sense of the complexity, the uncanniness, the difference and the paradox of voice. They recognise the in-between quality of voice – hinging between the physical (sonorous) and non-physical (signifying). [2] [3] As Steven Shaviro explains, in relation to Dolar’s work: [4]

"Dollar… argues and demonstrates that the phenomenon of Voice is in fact far more uncanny and slippery, and already inclusive of difference, than Derrida gives it credit for. The voice always stands in between: in between body and language, in between biology and culture, in between inside and outside, in between subject and Other…. the voice is both what links these opposed categories together, what is common to both of them, without belonging to either." [5]

In its uncanniness – in Freud’s sense of unheimlich or unhomely – voice carries a trace of its ‘home,’ the body of the speaker, but leaves that home to perform speaking. Steven Connor is particularly intrigued by the uncanny voice of the double; he works with the doppelganger figure of the ventriloquist’s speech because for him it evokes “the imaginary production of a secondary body, a body double: a ‘voice-body.’” [6] [7]

The doubling of voice undoes a ‘unitary’ subject – not just in relation to an individual subject, but it also disturbs a separation of two ‘unitary’ subjects. Here the work of Adriana Cavarero is particularly relevant, empahisizing the relationality of voice, in order, as she says, to prise apart any ‘unitary’ quality in Derrida’s figure of Speech. Cavarero engages with the alterity, relationality, and intersubjectivity of voice in order to get past the presence, which rightly, in her view, worried Derrida. [8] Relationality also resonates with the spatial relationship that voice creates, a shared space, a relational space, a doubled space – to which I’ll turn next.

Spatiality: Voice, Memory, and Place

While it’s a basic understanding within sound studies that sound operates not only through time but also in space – performatively in/forming the space it traverses – once again too little attention has been payed to how this plays out with voice in particular. Just as voice can be thought as a hinge between the sonorous and signifying, it also hinges between bodies and the spaces they inhabit: “…bend[ing] and connect[ing] rather than dividing…. facilitat[ing] openings and intertwinings (of doors, concepts, subjects, experiences, materials) rather than discriminating one side or one thing from another.” [9] And so voice hinges bodies and spaces, mediated and personal memory, memory and place; it connects speaking and listening bodies physically and affectively with each other as well as/through the spaces they share.

To explore this, I’ll discuss two works at Cockatoo Island, Sydney – a location resonant with its history as a prison and an industrial site and now an art and event site, tourist destination and camping ground. The first work is Susan Philipsz’ “The Internationale,” originally sited in an underpass in Ljubljana, during the 1999 European Biennial, Manifesta 3. The work was restaged in Cockatoo Island for the 2008 Biennale of Sydney ('Revolutions - Forms that Turn') in Turbine Hall – a large, abandoned post-industrial space. This is one of those works that takes you by surprise: What is the work? What is it doing here? Almost instantly, however, it seems completely sensible and at home. Yet, uncannily, of course, not at home, and thus disturbing our own location in the space. Philipsz’ singing of the classic workers’ anthem taps into the Hall’s physical memory, echoing through the space and through us. We feel ourselves differently in that space, sensing, somehow remembering its industrial history, as her frail, obviously recorded voice re-inhabits the space as if with workers’ memories – as if those memories haunt her voice, as if those memories haunt those enormous spaces, as if we overhear it in that space – a private, intimate voice wafting on that memory.

As always, Philipsz sings in a lone (lonely?), unaccompanied, palpably untrained voice – thin but reverberant. Here, it is an electronic voice emerging from one lone old speaker, calling you into the space as you approach. The effect of Philipz’ recorded voice, evidently with little or no ‘art’ or ‘professsional’ postproduction and no deployment of sophisticated speakers, is almost like an old radio abandoned somewhere in the Hall. 

"When I make the recordings, it is important to keep the breaths and pauses in between, so that the song sounds natural and intimate. The idea is that when you hear a voice taken out of context in this way, your own sense of self becomes heightened while at the same time, you begin to experience your surroundings in a new way." [10]

For Philipsz this sort of production gives listeners the possibility to imagine it as their own voice. [11] For some it is a melancholy lament for revolutionary workers’ hopes, for some it is exposed, fragile and brave, for still others, stirring and sad... Like any “cover” version, Philipsz is inhabited by and inhabits this song, bringing her own emotional response to it, but leaving us room to have our own.

Another, very different work – though like Philipsz’, complex and ambiguous thanks to the voicing as well as being strangely in and out of place – is Richard Grayson’s work, Messiah, in the 2010 Sydney Biennale. Messiah is located in one of Cockatoo Island’s long tunnels – a transitional, hinging space, moving from the expansive and bright outside into a dark, intense, echoic domain. Off that tunnel, in a small low ceilinged dark room, plays Grayson’s sound/video work.

Richard Grayson, who is interested in belief systems and the way that he sees Theology taking over from enlightenment rationalism, had emailed The Midnight Amblers, a group of musicians from Erskinville, Sydney, to rewrite and perform Handel’s Messiah as country music – revoicing and rescoring it so that we could hear the words anew. He asked the band to perform the work in a back yard, recording it with various DIY video cameras. When Grayson edited in Berlin, the sync turned out to be a bit off. He embraced this ‘failure,’ which served well to “foreground the artifice,” He wanted to “bring back the weirdness and the spookiness… of something we take in a way as granted.” [12] In its room strewn bizarrely with classic country and western hay bales, the out-of-sync mode – where sound does not seamlessly sub-serve the visual – add to the disturbance and heightened sense of awareness of the materiality of each sense. Itself dislocated from concert hall to suburban back yard to funky Berlin and now to this ‘art’ space, Grayson’s Messiah is dislocating – mediating and enhancing our sense of the layered memory of the work and of this place.


In both of these works the voice is not just performing, it is performative – performatively bringing forth memory of the place in which we find ourselves. The concept of performativity helps to take us beyond voice as straightforward performance or as the emission from some fixed unitary subject. Performativity is suggestive of the way voices DO and create, the way they change something rather that present or represent. Recalling Cavarero’s attention to the relationality of voice, I would suggest that voice performatively evokes this relationality, bringing it into being rather than expressing it. What I’d like to emphasize here is the complicated and intimate intersubjective relations, which are staged as performative electronic voices lure the other, across space, even across the familiar space of the Internet.

The example I’ll present is the networked performance of visual artist Barbara Campbell. The title of her durational project, 1001 nights cast, [13] played with the Tales from 1001 Nights (The Arabian Nights) and with the fact that the net cast of this performance involved a webcast and a cast of over two hundred writers, who submitted their stories online for Barbara to perform each night. When you logged on to her evening performance, you saw only her mouth, giving prominence to the voicing, which storytelling involves. Framed by a story of a bereft bride wandering the world and greeted by strangers who give her stories “to heal her heart,” the webcast opened with a view of Barbara’s tongue, pierced and wounded – like the heart of the bride – with a numbered tongue stud that signaled the number of the day/performance. I was fascinated by that mouth and tongue – a bit like a Chinese acupuncturist, searching it for deeper meanings, to see what it told about her, as it told her/other’s stories. I asked Campbell about the choice to frame her mouth and foreground her tongue, in a conversation we had for Maria Miranda’s forthcoming book, Unsitely Aesthetics. We discussed the mouth as visceral evidence of the physicality of the storytelling, and the importance of voice.

"Barbara Campbell: Of course the other thing about my tongue was that it had the tongue stud in it which had the number of that night … I previously didn’t have any kind of piercing…. that was another kind of reminder to me that I was carrying the project around or that it was inhabiting me or I was inhabiting it because to have a piercing in your tongue is very much like carrying an open wound because the tongue with all those enzymes from the saliva is always trying to heal itself…" [14]

This pierced and wounded tongue not only performed the stories but also performatively brought forth an intimate and wounded affective space that inhabited us and that we inhabited together during the ephemeral nine minutes performance.

Modalities and Techniques of Voice

Technique is somewhat like performativity – it shapes the object and technologies to which it ostensibly responds. Elsewhere I have discussed a number of bodily techniques or habits that shape voices as well as various modes of voice that throw the normal voice into relief – from the broken and stuttering voice to the scream.* Here I’ll focus on the ‘ground zero’ of voice, breath.

Breathing is both a technique that enables and shapes voice as well as a mode in the whispered or breathy voice. As an example, I’ll reference my own collaborative artwork. For a number of years Maria Miranda, and I have been working on a project called Talking about the Weather. In this work, we’ve been collecting breath, beginning with performative encounters with strangers on the streets. The project was animated by a desire to get the world’s biggest collection of breath and use it to blow back global warming.

As Tim Flannery said in a few poetic lines, which were the inspiration – literally and metaphorically – for this project, the intimacy of breath is not just between people but also between people and the planet:

"The air you just exhaled has already spread far and wide. The CO2 from a breath last week may now be feeding a plant on a distant continent, or plankton in a frozen sea. In a matter of months all of the CO2 you just exhaled will have dispersed around the planet." [15]

Breath is a particularly intimate and alluring mode of voice. Like full-throated voice, of which it is an essential condition and part, breath is compelling and intriguing in that it is both bodily and not – it starts in one body and then connects to and communicates with another. I should emphasize that I am not talking here about the commodified intimacy of TV ads that have become all too familiar. Rather the performative intimacy and breath that I am interested in and that I hear in a range of artists works, is more strange and in-between. It is a way of speaking about a shared affective, inter-corporeal space that is beyond that of two separate unitary subjectivities. Following Alfred North Whitehead’s idea that “‘the body is only a peculiarly intimate bit of the world,’” cultural geographer Sarah Whatmore understands the way in which “the corporeality of the body and of the world fold through each other.” [16] Breath, in its affective movement, performatively calls forth the space around us – it is “the very engagement between body and world from which these feelings arise.” [17]

Returning to the streets where we collected breath… With the weather feeling so frightening when we started this work in 2006, we needed to talk about it – obsessively, incessantly – to connect with people through this talk. We were asking people to contribute the breath they would use to talk about the weather to our project. The project felt to us like a very intimate communication in that we were asking people to contribute something personal and vital. We asked first for a minute of their time and then for their breath. We were calling upon and calling forth their generosity – and doing it in the middle of their everyday life, going somewhere on the street or in the park – we were inviting strangers to enter an imaginative, performative zone with us.

As you can imagine, gathering the world’s largest collection of breath is a big task and we realized we needed to extend our search as widely as possible, so in 2008 we took it into Second Life. Not having played in such worlds as a gamer, my experience there was unexpected. While at first appearance, and certainly if you haven’t been “in world,” Second Life may look like a familiar cartoon world, but this fails to account for the strangely magical, compelling and intense feeling of being there, including an intimacy with avatars, one’s own, and others’. While preparing for the breath collection events, I wandered around Second Life and noticed its voice activation mode. However after one attempt, I quickly realised that I did not want to use it, because it actually broke the intimacy of the connections between me and my avatar and others’ avatars. That is, voice activation, my own voice and others’, took me out of the world.  In a sense, this is obvious – the avatar has its own specific materiality and therefore needs its own voice. The implication of this is that in order not to break the intimacy of the Second Life experience, what is needed is either a silent voice, full of potential, but not actualised, or a particular voice for your avatar, that you choose or make, like all its other body parts and clothing. That voice would speak from the avatar’s ‘embodiment,’ not ‘yours’, and thus maintain the intimacy. Thus, in our own work in Second Life, which we experienced as a public place in which to collect breath, we chose to work with the specificity of avatars’ connections and still have the affect of voice by using only breath rather than spoken word.

Authenticity Effect

To end, I’ll briefly discuss what I call an ‘authenticity effect’ of voice. Although in the heyday of postmodernism and in the early days of the Internet, artists and everyday users seemed to revel in the disruption of identity and authenticity, now with social media and YouTube, there has been a prevalence of direct address and ‘at home’ videos that many read as a desire for and return to authenticity. What I sense in this, however, is a performative voice, and, as Cavarero would remind us, when voice works performatively, it is not necessarily a call to (or from) essentialism or authenticity. What we have here instead, I suggest, is voice performatively evoking authenticity – an authenticity effect.

Contemporary artist TV Moore provides my final example here. Timothy Vernon Moore delightfully invokes the network between subject and machine as he goes by his very own, proper name’s initials, TV – ‘no pun intended’, as he says. In his work, we can find stunning examples of what Nicolas Bourriaud insightfully understood as ‘postproduction’ (Postproduction, 2002) or what Mark Amerika inspiringly explores as remix culture (REMIXTHEBOOK 2011). After months of playing around in the Australian Broadcasting Corporations archives, TV Moore found a documentary about marginal people, earlier recognised as ‘vagrants’ or hobos. In the 60s, they were a different sort of nonconformist or outsider or prototypical artist-- alienated youth outside ‘normal’ structures of work, family, home. What TV Moore did for the work “The Forgotten Man” (2006) was perform the script of all the people in the documentary, from the snooty bureaucrats to the youth themselves to the ABC narrator. He re-performed their words, lip-syncing with amazing closeness. And so you watch this video in wonder and wonderment… whose voice are you listening to, why do they all have the ‘same’ voice? Is it the same? The lip sync is so neat, so 'authentic' yet...

And then you wonder, what has become of these characters, now ventriloquized, haunted, inhabited by TV Moore – or is it he that is inhabited by them? ‘Inhabiting’ is one of the key ways that Bourriaud understands the working of Postproduction artists. 

"Artists actively inhabit cultural and social forms…. By refilming a movie shot by shot, we represent something other than what was dealt with in the original work. We show the time that has passed, but above all we manifest a capacity to evolve among signs, to inhabit them." [18]

TV Moore voices an inhabiting of the forgotten man and all who discoursed around him, to mediate them and our memory of them, but also to displace them and himself. In this way, TV Moore’s work not only provokes wonder – about the forgotten man, about history, about documentary and about authenticity – but also provides a stunning final example of the intimate, uncanny and paradoxical aesthetic potentials of voice.


I discuss a number of these ideas at length in "Introduction: The Paradox of Voice”  and “Doing Things with Voices: Performativity and Voice” in Voice: Vocal Aesthetics in Digital Arts and Media, eds. Norie Neumark, Ross Gibson, and Theo van Leeuwen (Cambridge, MA: MIT Press, 2010).

Norie Neumark
La Trobe University

References and Notes: 
  1. Steven Shaviro, “A Voice and Nothing More,” Stephen Shaviro's Web Pages, 2006, (accessed June 12, 2012).
  2. Mladen Dolar, A Voice and Nothing More (Camrdige, MA: The MIT Press, 2006), 7, 8, 61, 96, 102, 103.
  3. Amelia Jones, “Space, Body and the Self in the Work of Bruce Nauman” (paper presented at Artspace, Sydney, December, 2006).
  4. Mladen Dolar, A Voice and Nothing More (Camrdige, MA: The MIT Press, 2006).
  5. Steven Shaviro, “A Voice and Nothing More,” Stephen Shaviro's Web Pages, 2006, (accessed June 12, 2012).
  6. Steven Connor, Dumbstruck (Oxford: Oxford University Press, 2000), 35-42.
  7. Steven Connor, “The Strains of the Voice,” Birbeck's official Web Site, 2004, (accessed June 12, 2012).
  8. Adriana Cavarero, For More Than One Voice, trans. Paul Kottman (Stanford: Stanford University Press, 2005).
  9. Amelia Jones, “Space, Body and the Self in the Work of Bruce Nauman” (paper presented at Artspace, Sydney, December, 2006).
  10. Manuel Segade, “Memory Trigger,” HighBeam Business' official Web Site, September 22, 2007, (accessed June 12, 2012).
  11. Susan Philipz, “In Conversation” (speech at Melbourne Town Hall, August 18, 2011).
  12. Richard Grayson, “Sin of the World: Grayson's Messiah (Video),” COFA Talks Online, June 3, 2010, (accessed June 12, 2012).
  13. Barbara Campbell, 1001 Nights Cast,” 1001 Nights Cast's Web Site, (accessed June 12, 2012).
  14. “Conversation" in Maria Miranda, Unsitely Aesthetics (Berlin: Errant Bodies, forthcoming 2011).
  15. Tim Flannery, The Weather Makers (Melbourne: Text Publishing Company, 2005), 22.
  16. Sarah Whatmore, Hybrid Geographies (London: Sage, 2002) 118 119, 160, 161.
  17. Sara Ahmed, “Communities that Feel: Intesity, Difference and Attachment,” in Conference Proceedings for Affective Encounters: Rethinking Embodiment in Feminist Media Studies, eds. Anu Koivunen and Susanna Paasonen, 10-24 (Turku: Media Studies, 2000), (accessed June 12, 2012).
  18. Nicolas Bourriaud, Postproduction (New York: Lukas & Sternberg, 2002), 18, 53- 54.

Neuro-Technology and Augmented Perception

Science fiction films suggest methods for contextualizing concepts and concerns surrounding new perceptual technologies.  Through revisiting two films, I will highlight major themes of neurological perceptual representation and procedural translation associated with brain computer interfaces, while suggesting their relevance to contemporary trajectories in augmented perception and mixed reality.



Traditional science fiction films suggest methods for contextualizing concepts and concerns surrounding new perceptual technologies.  Through revisiting two well-known science fiction films, Brainstorm (1983) and Strange Days (1995), I will highlight major themes of neurological perceptual representation and procedural translation associated with brain computer interfaces, while suggesting their relevance to contemporary trajectories in augmented perception and mixed reality.

Brain sensors are integrated into diverse contexts ranging from imaginative portrayal in films, actual use within interactive art works, and as scientific tools for investigation.  Throughout each, the apparatus stimulates narratives of penetration, record, distribution, replication, simulation, and reproduction of interior perceptual experience. These narratives are not unique to brain sensors, but rather continue along a technological trajectory, including forms of perceptual instrumentation and methodology as diverse as the X-Ray and psychoanalysis. 

Perception as Media  

Brainstorm opens its trailer with the provocation: "Suppose it were possible to transfer from one mind to another the experience of another person, any person, any experience.”  This major fantasy surrounding this transfer of experience implies that the device will mediate an immersive experience that is paradoxically both immediate (1) and scientifically observed.   The device's methods reflexively call attention to the neurological perceptual process as a mediating act that can be extended and replicated.  

“Immersed in media experience, conscious of mediated experience, we no longer experience any realm of human existence as unmediated, immediate, “natural”. We can only imagine such an experience (now aware that imagination, too, is an “imaging,” a mode of mediated representation).” (2)  

The physiological act of perception is presented as a cinematic media process, accessed computationally through a scientific device, such as a brain computer interface, and distributed cinematically.  Sensory information is recorded as electrical data signal, stored, transmitted, and reconfigured from one processing center to the next along a one-to-one pathway.  In Brainstorm, the biological brain is intercepted and translated by an electro- mechanical headset prosthesis. In Strange Days, the brain-computer-interface, called the SQUID (superconducting quantum interference device) references, at least in name, early brain imaging technology associated with clinical MEG (Magnetoencephalography), first developed in the 1960s. (3) Unlike actual SQUIDS, the devices in the film and in David Cronenberg films, such as Existenz (1999), are partially composed of biological matter. Nonetheless, processes of perception are still treated as computational electrical sensory streams, transmitted via digital or analog media distribution channels (tapes or CD’s). 

Within scientific contexts, imaging of electro-magnetic data from the brain is most frequently used to dissect and understand sensory and perceptual processes.  However, computational neuroscientists have also been working to simulate imagery directly from collected sensory data. (4)  EEG (Electroencephalography) Neurofeedback is a process whereby electrical neurological data is recorded and replayed for a user, translated into audio-visual signals, or as with LENS (Low Energy Neurofeedback System), fed back to the brain as an electrical signal.  While chemical brain alternations are widely accepted psychiatric therapies, active electrical stimulation systems such as DBS (Deep Brain Stimulation), ECT (electroconvulsive therapy), and TMS (transcranial magnetic stimulation), are more controversial.  Other experimental researchers have directly stimulated the brain to induce and study perceptual phenomena, such as autoscopy, the sense of an out-of-body experience. (5) 

Augmented Transpersonal Experience 

In the films, the source of pleasure seems to arise from the expansion of perception achieved through adaptive integration of the implant of another’s perception within oneself.  Instead of striving for a complete substitution of perception or full cinematic immersion in the entertainment material, the devices promise a greater pleasure of going beyond both self and other into territory which can only be facilitated through integration with the device. The novelty of perceptual difference might increase awareness of one’s own interpretive biases and filters, propelling the user into a state of augmented meta-awareness beyond the confines of the individual self.  

Both films use the fantasy of experiencing another's perception to engage with the desire to know what happens at the moment of death and beyond.  In Brainstorm, this ability to see beyond the confines of one’s own perception culminates in a scientific and spiritual quest to experience extended frontiers beyond mortality and physiology itself.  This desire to engage in the most forbidden of experiences, seeing death before actually dying, is presented as an alluring scientific, philosophical, and entertaining goal. In Brainstorm, the main protagonist scientist, Michael, excitingly exclaims: “I’m scared, but the thing is, I like it. I want more. It’s a chance to look scientifically at the scariest thing a person ever has to face.” 

Representational Modes

Brainstorm maximized the special effects of its day, including extended 70mm show-scan projection technology and enhanced graphics, to represent the transcendent experience of dying first as a poetic reflection on the brain as computer database memory mainframe, before travelling through psychedelic abstraction, and then finally dissolving into hyper-space.  Douglas Trumbull directed Brainstorm after producing other effects sequences representing the sublime through of technology and space, including Close Encounters of the Third Kind, 2001: A Space Odyssey, Star Trek: The Motion Picture, and Blade Runner

Strange Days presents the perception of dying as a pleasurable desire in more pathological terms.  Snuff tapes, in which an individual’s perception is recorded as they die, are a prized form of entertainment. During the film’s homicides, victims' deaths are shown on screen, while the perceptions of the perpetrator are continually recorded and simulated for the victim - homage to the voyeuristic pleasure of augmented terror presented in Michael Powell’s film, Peeping Tom (1960). Here, desire concentrates on an intensification of physiological expression associated with fear of death and the otherwise unrepeatable embodied terror of dying, rather than transcendence or dissolution of individual material experience. 

In Strange Days, the clips end at death, asserting that perception is either an on/off binary, or ceases to be able to present information related to its own destruction. The clips are displayed in factual photorealistic form without attempt to treat perception subjectively or abstractly, even in moments intended to represent intense feeling and death. Instead, perception is displayed as classical first person point-of-view cinema. While the dominant movement within contemporary visual effects continues towards graphical photorealism, alternate trajectories do exist, emphasizing abstracted subjectivity, exaggerated hyper-realism, and other forms of stylization serving affective perceptual effect. (6) In terms of brain computer interfaces, movements of presentation might span the gamut from desires to replicate or simulate objective reality, to alternate attempts to diminish or amplify more affective, subjective, or thematic aspects of perception for narrative aim. 

In conclusion, examining notable science fiction narratives is a valuable method for extrapolating conceptual trajectories of less traceable interactive media art forms.  In particular, Brainstorm and Strange Days demonstrate how perceptual processes and mediation devices have been represented, contextualized, and integrated with popular media forms, while suggesting thematic issues relevant to working with augmented perceptual devices. If perception is framed, translated, taught, modified, and augmented in part through existing neural technologies, what new speculative design contexts might be employed to encourage the development of devices that augment perception in ways beneficial to society?   

References and Notes: 
  1. Jay David Bolter and Richard Grusin, "Remediation," Configurations 4.3 (1996): 311-58. 
  2. Vivian Sobchack, Screening Space: The American Science Fiction Film, (New York: The Ungar Publishing Company, 1987), 237.
  3. David Cohen, Edgar A. Edelsack, and James E. Zimmerman, “Magnetocardiograms Taken Inside a Shielded Room With a Superconducting Point-Contact Magnetometer,” Applied Physics Letters 16, no. 7 (1970): 278-280.
  4. Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, and Yukiyasu Kamitani, “Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders, ” Neuron 60, December 11 (2008): 915-929. 
  5. Olaf Blanke, Theodor Landis, Laurent Spinelli, and Margitta Seeck, “Out-of-body experience and autoscopy of neurological origin,” Brain 127 (2004): 243-258.
  6. (Sobchack, 283). 




Users become Re-creators: Enhancing Experiences through Mapping

In this paper, we present a new understanding of interactive installations that goes beyond action-reaction communication between actor and installation. The goal is to enhance the user’s experience and engagement as well as the reflection about the creator’s initial intention. We employ ”Mapping” as method to redefine the user’s role from consumer to”re-creator” within a specified scope, set by the creator.



On the crossroads of technology and arts, we consider User Experience Design as a promising approach to empower the design process of media art pieces and ensure the actors’ engagement and reflection. We focus on interactive installations as media art pieces; the actor provokes a system’s reaction by his actions (e.g. full body movement).

Since most installations only provide a closed action-reaction framework – some employ programmed randomness to include surprising moments – we argue towards a new understanding of the actor’s role. The goal is to enhance the user’s experience by creating a deeper engagement and immersion on the one side and a reflection process on the other side.

Digital Experience Design

In context of HCI, works like “User Experience with the CYBER graphics terminal” (1974) from Edwards & Kasik have constituted the term user experience for the first time. [1] During the 80s the term was rarely used. In the 90s its popularity started again with the movement of Don Norman (“User Experience Architect”). Since the new millennium, the term user experience is spread through various disciplines in the field of HCI. Hence, the quality of a digital product does not rely on usability only, but also on aspects like aesthetics and emotional bonding. In the near future previously separated disciplines will merge due to the shift from performance- and task-oriented systems to experiences with and through digital products. [2]

Hassenzahl says: „[...] experience emerges from the intertwined works of perception, action, motivation, emotion, and cognition in dialogue with the world (place, time, people, and objects). It is crucial to view experience as the consequence of the interplay of many different systems. [...] While many processes together produce experience, emotion is at its heart and has an accentuated position. One may go as far as saying that emotion is the very language of experience.“ [3] 

According to this, our understanding of experience is a stream of thinking, acting, feeling, rating and reflecting of external and internal influences. It is an inner self-reflection. An experience is a composition of this stream into a closed and personally meaningful entity. Experiences give our actions meaning, they are remembered, communicated and act as a motivator or de-motivator. It can be named precisely and has a start and end point. As a matter of this, an experience changes over time.

Besides taking a look at experiences on a meta-level, it is also important to take a look at the particular factors that are directly related to our interaction with digital products, because user experience design does not only involve the product, but also the user, their activities and the context in which the interaction takes place. We understand user experience as a symbiosis of these four basic elements. As a matter of this, we extend the term experience design by the word digital to Digital Experience Design to accentuate the interaction with a digital product. The use of a digital product in contrary to analog products is essential for the general framework of interaction. Therefore we stress on this particular aspect.

The Digital User Experience is based on four key aspects: the characteristics of the product, the user, the activities and the context of use. The digital product is characterized by its tangible (pragmatic) and intangible (hedonic) qualities. The appraisal of functionality, performance and usability takes place on an objective level, whereas beauty, emotion and meaning take place on a subjective level. The user is driven in particular by his intentions (goals), perception, (pre-) knowledge and culture. In this process, self-reflection and perception/cognition takes place. The user evaluates and rates the own acting and the experience as well as compares the current experience with previous ones. Before the user interacts with a product he has expectations of the product (expected experience). This experience will change over time. The user starts with some expectations before the first contact with the product, which he might have gotten from product description, photos/videos or review. During the use, these expectations can be met or not. Based on these experiences new expectations evolve. After the use a first opinion is created and new expectations for future interaction/use arise. Places and spaces, their objects and people (subjects), events and environmental influences (e. g. light and weather) characterize the context of use. An essential factor is the time, because all characteristics of the context and activities depend on it; they can change during the day and over months and years. This does also include the requirements and needs of the user; they also change over time. Consequently, it is only possible to measure the user experience at a given point in time.

In conclusion we can say, that user experience is a dynamic phenomenon, which changes over time and influences or future experiences. The context of use influences the user, the activity and the digital product. Thus, the experience can be influenced for example by poor lighting conditions, that results in reflections on the screen, the activity by tight and crowded places and spaces, and the user can be distracted by high traffic, pedestrians or other objects (e.g. vehicles in road traffic). An activity provides the connection between the digital product, the user and the usage context.

Mapping to enhance experiences and reflexion

to map: to assign (as a set or element) in a mathematical or exact correspondence <map picture elements to video memory> (Source: [4]

With the term mapping we refer to the process of assigning a set of controls to a given functionality of an installation to modify the action-reaction principle of an installation. This gives the creator the possibility to enhance the interaction and user experience.

“Interactions are reciprocal events that require at least two objects and two actions. Interactions occur when these objects and events mutually influence one another.” [5] Thus, the user can influence a process and change the systems’ behavior. However, the interaction is limited to a set of rules, defined by the creator. Mapping on the other side modifies or enhances these fixed rules of interaction. It makes decisions about if, when, and how to respond to incoming inputs or which channel to use for the response. Additionally mapping is always connected to a certain topic (e.g. movement, sound). It builds in itself a closed entity.

More in general, a distinction can be made between installations in which a continuous mapping of action (input) onto reaction (outputs) is used and installations in which a sort of dialog takes place. These two conditions can be thought as boundaries of a continuum of possible interactions. According to this, we can distinguish two different types of mapping: 1. direct mapping implies some kind of directness in the action-reaction process without any dynamics; 2. indirect mapping includes some kind of logical model or reasoning that selects a reaction-algorithm in correspondence to the current context. Hence the reaction is selected from a collection of possible reactions. Due to this static concept the user experience decreases when the user fully understood the underlying mapping rules. 

Therefore we propose to change the role of the actor from an active user to an active re-creator. A re-creator is empowered to manipulate the mapping rules, which allows her/him to reflect on technology, aesthetics and experience. Technology becomes a visible artifact of the installation. Performing the mapping process motivates the re-creator to explore the action-reaction framework and underlying rules. Adding personal meaning enhances the experience even more.

In traditional interactive media art works the designer defines a set of actions and reactions and maps them according to the designer’s mental concept (see fig. 1). The user on the other side perceives the installation as a black box. He/she has to figure out how the installation works (action → reaction) and how the designer applied the mapping. With the lack of modifying the mapping, his/her experience of interacting with the installation is decreased. This can lead to one of the following three states: The user gets bored: He/she has figured out the installation’s underlying concept and mapping; any further interaction is boring; the interaction time is too limited/short. The user gets unexcited: He/she understands the static installation concept, but its static behavior makes the interaction unexciting. The user gets frustrated: He/she cannot find out or understand the installation’s underlying concept and mapping.

By applying an dynamic mapping concepts to the interactive media art work, the designer’s and user’s role are changed (see fig. 2):

The designer still creates a set of actions and reactions, as well as a mapping (as shown in figure 1), but in addition he/she defines certain rules and boundaries. The designer defines the different perspectives of the installation in order to reflect upon its being and purpose. He/she can adjust this experience by defining the boundaries of the mapping. The re-creator exploits the installation and modifies the mapping within the given scope through an additional interface.

However it is always a tightrope walk between level of freedom and level of art. The more mapping is applied, the more the designer divulges the installation to the user. On the other side, the less mapping is applied, the more the installation can focus on a certain aspect.

An Example: Der Schwarm

Some examples of mapping are presented in the installation Der Schwarm. [6] A flock of swarming light spots projected on the floor reacts to free body movements. The response of the swarm intelligence to the movement of the interacting user is represented through behavioral patterns. A pattern defines a set of swarm parameters such as movement direction, velocity and graphical representation. Free body movements are tracked and its velocity and position are mapped to the flock’s behavioral patterns and position. Quick movements by the user evoke a fleeing or aggressive flock of light spots, while slow movements make the light spots react calm and friendly. 

An enhancement of the installation Der Schwarm is an auditory display [7] that creates sound, every swarm particle creates a sound. The mapping is realized through the employment of Albert Mehrabian’s three-dimensional emotion model (PAD), which has advantageous properties for digital systems and is already been applied to link properties of sound and emotions. [8] [9] Mehrabian’s representation oriented system is defined through the axes valence (pleasure vs. displeasure), arousal (arousal vs. non-arousal) and control (dominance vs. submissiveness). [10] At first we mapped three major swarm behavior parameters to the axes of the PAD model. Then we assigned three sound parameters to the axes. The result is one possible mapping of swarm behavior parameters to sound parameters. An interface enables the user to modify the mapping, so any combination of the three swarm behavior and sound parameters is manually adjustable. 

In this example the user becomes a re-creator. S/he is enabled to change the mapping within a defined scope and explore its effects by full body interaction. The re-creator on the one hand can modify the underlying mapping rules and understand the installation’s basic idea. The re-creator’s empowerment to configure certain functions and manipulate the action-reaction framework of the installation can foster hers/his experience. On the other hand, the designer is enabled to provide restricted insights into his media art pieces. This allows her/him to draw the re-creator’s attention to certain parts of the installation to deliver her/his intending message.


In this paper we have proposed a new concept for enhancing the user experience with interactive installations. With the installation Der Schwarm we have a proof-of-concept. Applying this approach allows an enhanced experience for the actor and provides new possibilities for the creator to reach the actor. The actor becomes a part of the whole process (not the product). Thus we are not only able to enhance the experience, but also to strengthen the engagement of the user with the installation as well as the immersion and reflection.

References and Notes: 
  1. E. C. Edwards, and D. J. Kasik, “User Experience with the CYBER Graphics Terminal,” Proceedings of VIM-21, Denver (October 1974).
  2. D. Krannich, Mobile System Design (Norderstedt: BoD GmbH,2010), 126.
  3. M. Hassenzahl, Experience Design: Technology for All the Right Reasons (San Francisco: Morgan & Claypool, 2010), 4.
  4. Merriam-Webster Dictionary Web Site, (accessed May 24, 2012).
  5. E. D. Wagner, “In Support of a Function Definition of Interaction,” in The American Journal of Distance Education 8, no. 2 (1994).
  6. A. Hashagen, H. Schelhowe, and H. Seelig, “'Der Schwarm' - An Example for Interaction of Computer Science and Performance Studies,” Proceedings of ISEA 2008, Singapore (July-August 2008).
  7. A. Hashagen, N. Hajinejad, and H. Schelhowe, “Dancing Sound: Swarm Intelligence Based Sound Composition through Free Body Movements,” Proceedings of ISEA 2009, Northern Ireland, (August-September 2009).
  8. C. Peter, “Emotion Models and their Implications for System Design,” in Emotion in HCI: Joint Proceedings of the 2005, 2006, and 2007 International Workshops (Stuttgart: Fraunhofer IRB Verlag, 2008).
  9. J. Loviscach, and D. Oswald, “In the Mood: Tagging Music with Affects,” in Emotion in HCI: Joint Proceedings of the 2005, 2006, and 2007 International Workshops (Stuttgart: Fraunhofer IRB Verlag, 2008).
  10. A. Mehrabian, Pleasure–Arousal–Dominance: A General Framework for Describing and Measuring Individual Differences in Temperament,” in Current Psychology: Developmental, Learning, Personality, Social 14, no. 4 (Winter 1996).

Beyond Locative: Media Arts after the Spatial Turn

William Gibson no longer writes about cyberspace in the future, but instead about locative art in the atemporal present. Having emerged in the mid-’00’s from media arts, locative media are now part of the consumer technology and popular culture. This panel discusses the value of this concept in relation to debates at the intersection of urbanism and media studies, and considers the (non)existence of a locative avant-garde.
Tuesday, 20 September, 2011 - 09:00 - 10:30
Chair Person: 
Marc Tuters
Tristan Thielmann
Mark Shepard
Michiel de Lange

Chair: Marc Tuters

European Media Art Network / European Media Artists in Residence Exchange

The Program includes a showcase of selected EMARE Productions of the past 15 Years presented by Peter Zorn, EMAN / EMARE Manager and Prof. Mike Stubbs, CEO of FACT, Liverpool.
Friday, 16 September, 2011 - 13:00 - 14:30
Peter Zorn
Mike Stubbs
Peter Zorn
Mike Stubbs

Chair: Peter Zorn
2nd Chair: Prof. Mike Stubbs

European Media Art Network
European Media Artists in Residence Exchange

Museums, Archiving, and Interactivity

artMUSE goes MaX – how virtual exhibition technologies arises media art in Europe by Martin Koplin et. al./ Historical Orchestra: A Research for a More Engaging Museum Experience by Ferhat Sen/ Soft Clouding by Morten Søndergaard, Thomas Markussen, Barnabas Wetton, and Ivan Dehn/ From Archive to Retroscope – pushing forward resource integration by Catherine Moriarty and Chris Wild/ The Museum Machine - or - A Database Approach to the Representation of Space by Andreas Kratky and Juri Hwang
Tuesday, 20 September, 2011 - 17:00 - 18:40
Chair Person: 
Daniel Wessolek
Martin Koplin
Ferhat Şen
Morten Søndergaard
Thomas Markussen
Barnabas Wetton
Ivan Dehn
Catherine Moriarty
Andreas Kratky
Juri Hwang
Lívia Rózsás
Chris Wild
Helmut Eirund
Ann Van Nieuwenhuyse
Svetozora Kararadeva
Iwona Bigos
Irena Ruzin
Hans Hermann Precht
Reha Dişçioğlu

artMUSE goes MaX – how virtual exhibition technologies arises media art in Europe

by Martin Koplin, Livia Rozsas, Helmut Eirund, Hermann Josef Stenkamp, Ann Van Nieuwenhuyse, Svetozora Kararadeva, Iwona Bigos, Irena Ruzin, and Hans-Hermann Precht

Syndicate content