Reconsidering Media Art Dynamics

Push/Pull is an interactive artwork for exhibition, but also an object of study. The paper considers the social dynamics specific to the reception of such a research-based artwork, including its gaming dynamics. The collaborators on the project are interested in the role of user experience interviews in the development of such works, and in general how making artwork in a research context might best be reflected in its presentation.


Push/Pull is an interactive artwork conceived and executed between 2005 and 2009, with Melanie Baljko, John Kamevaar and a team of people in the context of a collaborative research project under the rubric “Lo-fi”. Kim Sawchuk has also collaborated in the project in designing user experience protocols and carrying out interviews with us on a number of occasions. [1] Push/Pull is an artwork for exhibition, but also an object of study through which we examine some of the social dynamics specific to the reception of research-based artworks. In particular, we are interested in the role of user experience interviews in the development of such works, and also how the multi-step progression of making artwork in a research context might best be reflected in its mode of presentation. Push/Pull was built through a process that included several user-tested prototypes, and what we learned at each stage was incorporated into the work. The prototypes were interactive scenarios staged predominantly at the Ontario Science Centre in Toronto. They were very task-oriented in comparison to the final artwork, but the movement dynamic of the interaction carried over to Push/Pull: physically moving around the circular sculpture, as well as back and forth in front of its four panels, is required to activate the visual and aural expressions of the work. At the panel that is most like a game, the interactant has to virtually dance around to stay connected with her or his moving cluster of LEDs.

The strengths of the Lo-fi collaborators have shaped Push/Pull. Because of the modeling and computational processes that subtend biotechnology research, the topic of my art practice in the 1980s, by the mid-1990s the focus of the work shifted toward artificial life. The resulting body of work concentrates on artificial agency, and is largely interactive, both features of Push/Pull. Melanie Baljko’s principal focus of research is computational models of conversation, a sub-specialty within computational linguistics. Modes and strategies of human conversation are used as the model for the complex interactivity of Push/Pull. The Lo-fi project also includes the expertise of communications scholar Kim Sawchuk, specifically, her work on audience response to new media art. Dr. Sawchuk’s work traverses the humanities, social science and the arts community. Dr. Baljko and Dr. Sawchuk have developed multi-methodological evaluation that merges qualitative and quantitative approaches. John Kamevaar brings to Lo-fi a noise aesthetic and a history of live performance, through the experimental sound group Kaiser Nietzsche that he formed in the mid-80s, and many years of performing with the Toronto-based improvisation ensemble CCMC.

Social Intelligence and Gaming in Push/Pull

Push/Pull is programmed in such a way that layers of artificial agencies are presented to participants – from the agency of the system itself to abstract entities composed of a few lights and electroacoustic sounds. In effect, Push/Pull invokes a heterogeneous population of low-fidelity agents: human-representative agents are differentiated from artificial agents by colour in the LED displays and by sound, or their “voice” (the sounds are taken from the natural and media world, but micro-sampled and extensively processed).

Representation of artificial agents in interactive artwork invokes the topic of artificial social intelligence – the study of social relations that include artificial entities. Social intelligence in general means the capacity to understand experience in direct relation to how others understand it. Thus, for example, empathy plays an important role because it is the imagining of another’s experience. Empathy has had a strong presence in modeling social intelligence in artificial media; it is considered by many robotics researchers to be a core feature of modeling both life-like emotional response and embodied learning, and thus essential in making more “realistic” robots. But characteristics of artificial social agents are not meant to appear simply to mimic humans or other animals. Rather, these agents when they interact with humans in a mixed social world have to develop their own unique kinds of “mental images” of the human participants they interact with: sensor data enables the agents, whether robotic or software-based, to recognize humans and locate them within a set of co-relations. Lo-fi proposes that such a heterogeneous system of agents calls on biomimicry, because it plays out as a conversational exchange, but with the virtual agents themselves as far away as possible from mimicking humans.

The intention in our modeling of social intelligence is to shed light on issues such as the threshold of representation required to assign agency and to invoke artificial sociality. Because the interactant appears in Push/Pull as an agent in the same form as the artificial agents, all of the agents have an elemental subjectivity that is attributed by the human interactant, and the interactant-representative agent is a rudimentary avatar. Gaming theory is very a propos to the dynamic in the work, because modes of intersubjectivity in game play necessarily take into account the avatar as social agent. Since video games are built on the premise of a first-person player (in combat games, the FPS or first person shooter) the avatar is the quintessential artificial other, in that it represents the self as other and psychically embeds the player within the game action. This is also the entry point to sociality in Push/Pull. Unlike high-definition reality video games, the game-like aspect of Push/Pull doesn’t give a more literal presence for the interactant in the work, but it does evoke similar subjective and intersubjective dynamics in the imaginary of the interactant.

In fact the agents in Push/Pull align closely with very early minimalist gaming representations – think Pac-Man or Tetris from the 1980s. This kind of abstraction within the gaming paradigm is especially pertinent to exploring questions of how people can be induced to enter intersubjectivity with artificial agents in the first place. Mark J.P. Wolf postulates that, although marketing values that relate gaming to familiar media such as television and film are major factors in the push toward ever more realistic game characters, in essence an abstract representational mode in gaming is historically too hard to teach people, who want intuitive understanding of game rules. [2] A parallel problem applies to the abstract game as a feature in an artwork. But it is appealing to revive abstraction, as Push/Pull does, for its potential to self-reflexively focus interactants’ attention on their enactment of the work they are with, not on external narrative or discursive elements. The trade-off of provoking a perplexed response might be well worth it because of the immersion factor, shifting the response of feeling lost in the sense of cognitive confusion to getting lost in the sense of losing oneself in an experience. Although it would be possible to demo the mode of interaction at the most game-like panel of Push/Pull, for example through video shown in the exhibition space, that would foreground gaming as the core topic of the work, whereas the overall abstraction of light and sound interaction in Push/Pull is not secondary to the exchange with artificial agents.

Given the instantaneous call-response paradigm that dominates interactivity, it is useful to remember that one hallmark of an interesting artwork is that many more things are going on than are evident on the surface. Ever in reserve for plumbing these depths of the imaginary realm is psychoanalysis, rather sidelined in the culture world since its heyday in the 1980s but always satisfying for its ability to describe inner life. Subjectivity located in the interplay between one’s self and one’s otherness to oneself, and the allure of (mis)recognizing oneself in an image or representation, are game features that invoke a psychoanalytic reading. Focused on Lacan’s mirror stage and on Freud’s pleasure principle it could go thus:

… players pleasurably experiment with the surprising, often counterintuitive articulation between their manipulation of the interface and the avatar’s obedient response. If anything, such pleasures seem amplified by the uncanny difference between reality and reflection: an alterity enabling players both to embrace the avatar as an ideal and to reject it as an inferior other. … The avatar is not simply a means of access to desired outcomes, but an end in itself – a desired and resented lost object, existing in endless cycles of renunciation and reclamation. [3]

In the game, whatever the avatar (simple avatars are contained within all subsequent ones), the ego struggling ceaselessly to reconcile itself with the ideal and the abject can dominate the avatar as problematic other simply by controlling it. And the loss of this identification allows, in Lacanian terms, a momentary loss of ego, since ego is formed through - and as - identification with an externalized image of oneself. What feminist film theorist Kaja Silverman has described as “radical self-loss” can be experienced as pleasure in a gaming format, playing out the Freudian acquisition/loss repetition in a mode of sheer fun. [4]

Such repetition is certainly a dominant feature of Push/Pull. We imagined that the part of Push/Pull’s interaction overtly based on gaming would be most accessible to viewers, but the opposite turned out to be true: according to our observations of interactants’ unwillingness both to pursue the game action or to say much about it afterwards, they are reluctant to shift from perceiving light and sound as a play of abstraction to perceiving autonomous agents that are composed from the same elements. Complicating the gaming interaction is the fact that the artwork as a whole system seems to be an autonomous entity that knows its environment, even though this is not strongly modeled but operates as a secondary effect. While in gaming the interactant’s avatar has in a sense become the interface and the player has an assurance that it remains even when it goes off screen, in Push/Pull it is very evanescent, with multiple instantiations and also there and then gone. It is a problematized, misbehaving avatar.

Or maybe we are misreading what we have perceived as the interactant’s unwillingness to game play. Maybe it is just that the experience is very interior and personal, a play of power and desire that acts out in the order of the imaginary, difficult for the interactant to access or articulate even when they do become very immersed in it.

Social Understanding and Commonality of Experience

Some key aspects of reception of interactive artworks are not in smooth synch with the modalities of artworld exhibition. For example, how the interaction works should be very overt and transparent so that most people succeed at it, and in a short span of time. In some ways this is a frustrating syndrome (although clearly many artists resolve the dilemma very well). In research-based work there can be additional layers of intention and meaning that the artist would prefer to reveal to interactants, and extra-discursive modes of presentation like demos, drawings, texts, or live guides are all possible strategies. Some interactive artists perform their works to get around this limitation and tease out the layers of meaning in a work for an audience, whether one by one or in groups. There may be a threshold at which a work needs to be performed by a non-novice interactant for an audience, so as to be fully revealed – we have been experimenting with Push/Pull in this mode.

Moreover, an aesthetics of interactivity that can be communicated to the public at large needs to be developed, so as to foster a learning curve about several issues: the role of systems themselves in such works; the interplay of task and reward with more ineffable, qualitative features; and the relationship of interactants to passive observers. Interactivity requires an aware or awake user: the interactive work always speaks to the interactant in some way and launches an identificatory process that some would argue is even stronger than that of cinema. It is at the very least different from cinema – a new paradigm that is not well theorized or understood. The HCI (Human Computer Interaction) community has been making some inroads in this respect. [5] Artificial life art brings in questions about imputing intelligence and goals to computer systems, as well as an investigation into the mechanisms of assigning agency to non-living entities.

A direct parallel is obvious between social understanding and aesthetics: the deliciously impossible task of aesthetics is to describe experience in a language that can be common to as broad a group of people as possible, and the project of social understanding is to achieve among a group of people a shared interpretation, response, or even an agreement to disagree. Traditionally, the emphasis has been on the construction of the art object using conventionally held aesthetic principles that can assure a common ground for interpretation. This process already involves intensive feedback loops, because the reading of artworks by experts has over centuries constantly pushed the rules of how they are to be made and understood. A further loop with great potential for media art is the incorporation of knowledge about interactivity experience gathered from interactants, into a common social understanding of such experiences. The development of aesthetics for interactive media art absolutely requires user experience to be absorbed into both development and documentation of the work. [6]

The founding concept of the Lo-fi project was to take attention away from mimicry of human features and divert it to a different representational tension: that between uniqueness of experience (what is internal to the viewer or interactant, solitary) and commonality of experience. We propose that in Push/Pull, commonality is triggered by the sculpture’s recognition of the interactant, as noted above. This is enhanced by the interaction eliciting for the interactant the belief that the sculpture’s behaviours work in tandem with his or her own behaviour. That is, the participatory mode created in the interaction is such that the sculpture and the interactant clearly have unique but interdependent roles. These common understandings are complementary to the individualized process of interaction for each person. Our user interviews have shown that this balance of forces is clear to some interactants and not to others – the key interference in understanding the dynamic is the urge for more feedback, and precise feedback. Letting oneself go with the flow it is not the familiar mode for interactive art. We are interested in how an abstract, non-goal-oriented mode can inform public interpretation through art discourse, and loop back into artistic production. We caution that discourse doesn’t (and should never) override experience in our staging of the dynamics of social intelligence; an interactant’s personal, private experience with nothing more added is enough. There is no easy recipe for aesthetics: artists will continue to construct what they perceive to be the best qualities in an experience, and aim for awareness and understanding in the recipient of those qualities.

As an aspect of this aesthetics project, we are interested in contributing to guidelines for artists to develop user experience documentation. Preservation of new media artworks, media art histories and archives, and public understanding of these works are all bound up together in the concept of “user experience” and “usability studies” – for example in the work of The Variable Media Network and the Capturing Unstable Media project. There is space made in these structures for experiential documentation gathered by the media artist her or himself, but not yet methods for the artist to address how it can feed back into productions.


The author wishes to acknowledge the funding support of the Canada Council for the Arts/Natural Sciences and Engineering Research Council of Canada (New Media Initiative) and the Social Sciences and Humanities Research Council (Research/Creation Program).

References and Notes: 
  1. Dr. Baljko (York University, Toronto) and Dr. Sawchuk (Concordia University, Montreal) contributed to the development of this paper. See for information on the team and on “Lo-fi embodiment,” which Dr. Baljko and I developed to explore the threshold of agency attribution.
  2. Mark J.P. Wolf, “Abstraction in the Video Game” in The Video Game Theory Reader, eds. M.J.P. Wolf and B. Perron (New York-London: Routledge, 2003), 47-65.
  3. Bob Rehak, “Playing at Being: Psychoanalysis and the Avatar” in The Video Game Theory Reader, eds. M.J.P. Wolf and B. Perron (New York-London: Routledge, 2003), 107.
  4. Ibid., 110.
  5. See ACM TOCHI – Special Double Issue on the Aesthetics of Interaction 15, no. 3 (2008), which includes Baljko and Tenhaaf, “The Aesthetics of Emergence: Co-constructed Interactions.”
  6. Baljko, Sawchuk and Tenhaaf shot video and interviewed about a dozen interactants in December 2010 during thelivingeffect exhibition curated by Caroline Langill at the Ottawa Art Gallery (Canada). Baljko translated interactants’ movement in the videos into graphic images, two of which are reproduced here.