Robots as social actors: audience perception of agency, emotion and intentionality in robotic performers

This paper looks at the different ways audiences perceive and respond to anthropomorphic and bio-mimetic qualities in robotic characters, specifically their perceptions of agency, emotion and intentionality. The author argues that it is audience perception rather than the innate qualities of the robot that determines successful robot-audience interactions.


Analyzing Robotic Performance

This paper analyzes robots as performative entities that create themselves in the moment of their performance and also looks at how audiences perceive and interpret those performances through observation and interaction. Interactions between humans and robots take place in a variety of different contexts. Some of these contexts are explicitly performative or theatrical, including Honda’s ASIMO conducting the Detroit Symphony Orchestra, Hiroshi Ishiguro’s female android Geminoid-F acting in the Japanese play Sayonara and Louis-Philippe Demers’s robotic performers in Australian Dance Theatre’s (ADT) Devolution. These performances are all tightly scripted and rehearsed. Other human-robot interactions take place in more open environments, such as art galleries and museums where audiences can interact with robots in unscripted interactive encounters. Nevertheless, I would argue that there is a theatrical performative element to all public displays of robots. All robots are in essence performers: they are designed to act and interact in the world and are programmed (scripted) to perform in particular ways.

How then can we best analyze the performances of robots across both theatrical and non-theatrical environments? Moreover, how do audiences respond to these robotic performances? While there are a growing number of studies analyzing robots as performers, particularly from the domain of performance studies, [1] [2] [3] [4] it is the work of sociologist Erving Goffman that proves to be particularly useful in analyzing robotic performances and interactions with humans across both theatrical and non-theatrical contexts, such as art galleries and museums.

In The Presentation of Self in Everyday Life, Goffman views all human social interaction as a type of acting. We don’t have to be on a literal theatrical stage to act, we are all actors who craft and perform different versions of ourselves in our everyday lives depending on which social situations we are in and who we are interacting with. Goffman uses the metaphor of the theater to describe how we move between back stage and front stage arenas using various techniques of “impression management” such as selecting different modes of dress, speech and behavior to perform these different presentations of self to our different audiences. [5]

Using Goffman’s theatrical framework, we can analyze the physical appearance and behavior of the robot along with its staging and theatrical mise-en-scène to see how these all play a part in framing the robotic performance and how it is perceived and interpreted by audiences. The back stage preparation of the robot’s appearance and behavior includes its design, fabrication and assembly, as well as more conventional types of costuming and dressing up. How the robot is then presented to an audience, whether this is in a theater, gallery, museum or trade show, also contributes to the overall impression the robot will make.

We can break down these aspects as follows:

  • Appearance (robot morphology, for example machinic, biomorphic, zoomorphic, anthropomorphic, and costuming)
  • Behavior (the robot’s movement and actions including its interaction with its environment and with other actors)
  • Context (this includes the environment within which the performance takes place and aspects of theatrical mise-en-scène such as setting, props and lighting)

Goffman’s description of back stage and front stage arenas and the team efforts frequently involved in these everyday presentations of self marries itself very well to the production context of robotic performance, which typically includes the artist as well as literal teams of technologists, assistants and handlers who work behind the scenes in the presentation of the robotic artwork. In this team effort, the agency of the performance may be distributed in a variety of different ways between the members of the team and the robot itself. The robot may perform completely autonomously and have its own emergent agency and behaviors (albeit programmed by the artist/technical team) or it may be controlled in more direct ways through automated performance scripts or teleoperation.

Some Case Studies

Wade Marynowsky, The Discreet Charm Of The Bourgeoisie Robot (2008)

There is something of a camp aesthetics evident in Wade Marynowsky’s cross-dressing robot Boris in The Discreet Charm Of The Bourgeoisie Robot. Although Boris playfully references human attributes in his voice, clothing and behavior, he is still clearly a robot, he is not trying to pass as human. The robot is dressed in an old-fashioned Victorian black dress trimmed with lace but his glass-domed head with its camera eye clearly proclaims his identity as a robot — a robot playing dress-ups. As gallery visitors enter the space Boris whirls in circles and engages them in conversation. Marynowsky’s robot is reminiscent of the robot in Lost in Space, the Daleks in Doctor Who and Robbie the Robot in Forbidden Planet, but its historical lineage also includes the famous chess playing Turk, an automaton built by Wolfgang von Kempelen in the late 18th century. Von Kempelen’s automaton astounded its audiences with its uncanny chess playing ability until it was revealed that the Turk’s prowess was in fact attributable to unseen human operators hiding in the stand that housed its mechanism. Marynowsky’s robot is controlled by similar sleight of hand — in this case it is an unseen human operator (the artist) who remotely observes the actions of gallery participants and direct Boris’ movements and speech via the Internet.

The mise-en-scène of the performance — the lace-trimmed black dress and the old-fashioned gramophone horns lining the gallery walls — combined with the robot’s uncanny whirling when visitors enter his space evokes the feeling of a Victorian séance; especially combined with the spirit possession inherent in his channeling of his master’s voice through the Internet.

Simon Penny - Petit Mal (1989-2006)

There is nothing human-like in the appearance of Simon Penny’s Petit Mal. The robot is completely machinic in appearance. It sits on two bicycle wheels joined by an axis with an upright pole supporting three ultrasonic sensors and three pyroelectric (bodyheat) sensors in the front and a fourth ultrasonic at the back. However, although not ostensibly anthropomorphic or zoomorphic in appearance, the constellation of sensors nevertheless acts as a sort of ‘head.’ A colorful vinyl print covers some of the metal tubing which acts as a counterpoint to the utilitarian machinic appearance of the robot and gives it a more playful and frivolous appearance.

The robot moves around the gallery performance space generally avoiding walls but sometimes lightly glancing off them. It rocks back and forwards on its base as it pursues and reacts to people in its performance environment. It will approach audience members who are directly in front of it up to a distance of about 60cm and try to maintain this front-facing position and distance as its audience interactor moves. If the person comes closer than around 60cm, Petit Mal will retreat. However, the robot’s behavior can become confused if there are multiple people in the performance area or if it gets cornered. The appearance and gently erratic movement and behavior of the robot contribute to its playful demeanor. The robot’s name derives from a neurological term that describes a momentary loss of control or consciousness. The naming of the robot provides its behavior with a psychological frame. Is this robot out of control? Is it psychologically disturbed?

Petit Mal has appeared in many gallery performance environments, sometimes in an open gallery space and sometimes in specially constructed enclosures. The robot (when it was exhibited at Transmediale 2006 in Berlin) performs in a rectangular arena enclosed on all sides by hip-high white walls. This performance area is reminiscent of a zoo enclosure with the audience standing behind the wall to watch the actions of this strange creature. The robot is contained in this space with no other objects or props but audience members are able to enter the space to interact with the robot.

Audience perception of robotic performers

We can conduct a rigorous semiotic analysis of a robot’s appearance and behavior and the staging of its presentation as I have done above but this is only part of the equation. The key question remains: how do humans understand and interpret the performance of robots?

In his analysis of the everyday presentation of self, Goffman also places particular emphasis on the role of the audience in receiving and judging the performance. A successful performance is one where the audience views the actor as he or she wants to be viewed. We all test and judge each other’s performances. If robots successfully perform the behavioral signifiers of animacy, agency, emotion and intelligence, audiences will respond to those cues. However, the intention of the performer and the intended meaning of the performance is not necessarily what will be received by the audience. Both human and robotic performers are subject to performance mistakes and unintended behaviors. These gestures and behaviors (for example, the jerky movement of a robot or responses that are too fast or too slow) even if they are not an intentional part of the performance will be interpreted as meaningful by the audience and become part of the performance effect.

As Byron Reeves and Clifford Nass [6] have shown, human responses to computers and virtual characters are informed by deeply ingrained physiological and behavioral tendencies and habits. These instinctive physiological responses (such as reacting to facial expressions, body language and movement) and social responses (such as a tendency to be polite) are carried over from the physical world into our interaction with robots.

When robots display machinic, bio-mimetic or anthropomorphic characteristics, these performative signifiers (sign-systems) are measured against the audience’s own experience of other similar entities (human, animal, insect, machine, art) that they are familiar with. The robot’s movement and behavior are just as important, perhaps even more important, as its physical appearance in this regard. What the robot does, how it does it, and how it responds to its environment and other entities including audience members are key factors in how it is perceived.

Behaviors that look too controlled and automated can appear machinic and unexpressive. Unpredictable behaviors by the robot in response to its environment and to other objects/people in that environment give an appearance of agency, personality and even emotion. Hesitations, frailties and inconsistencies make the robot appear more like a living organism than a programmed machine. The active interpretive role of the audience is a key factor here. It is the audience's projection of their own meanings onto the performance that generates much of the expressiveness of the robotic performance. This, after all, is how audiences read and respond to the performances of human actors. We interpret each other's performances including perceived intentions and emotions through reference to our own experience and emotions.

In this scenario, whether the robotic performer is intelligent and has emotions or not is not the key issue, it is whether we can tell the difference or not. Human perception and emotional and cognitive responses are more important than epistemological ontologies when it comes to robotic performance. The successful performance of the robot, judged from the audience’s point of view, is determined by what the audience can directly perceive in the robot’s appearance and behavior rather than by the intrinsic qualities and abilities of the robot (for example, whether the robot is ‘truly’ aware, intelligent and socially responsive).

As Sherry Turkle comments in her book Alone Together, “Computers 'understand' as little as ever about human experience […] They do, however, perform understanding better than ever.” [7] Robots may not be truly alive, but according to Turkle, they are becoming “alive enough” for humans to have relationships with.

The intrinsic qualities of the robot including the sophistication of its manufacture, its sensing systems and Artificial Intelligence (AI) programming are only relevant to the audience to the extent that they impact on the robot’s observable behavior and performance. These factors may be highly relevant to scientific robotic research and robotic development but in terms of audience response, careful staging, programming and even trickery may be just as important factors in achieving an effective performance for the audience. Robotic performances may be completely autonomous or assisted by human operators. From the audience’s point of view, it may be difficult to tell the difference. Creative staging and showmanship along with elements of deception and trickery have a long history in machine performance, as in Von Kempelen’s chess-playing automaton. Wade Marynowsky’s Boris has automated sequences and is also teleoperated by the artist and other guest operators, making the robot appear to be much more intelligent and aware of its audience. This hi-tech puppetry and remote operation of robotic performers is also the case with Hiroshi Ishiguro’s teleoperated Geminoid robots, which are controlled by the humans operating them rather than acting as autonomous performers. In this process, agency and social intelligence is transferred and delegated from the artist/operator to the robot even though from the audience’s point of view, the intelligence and awareness appears to be coming from the robot performer itself.

Successful acting is all about simulation and making what is unreal appear real. For a robot, this is the ability to persuasively simulate or pass as human, or alive, or intelligent. Alan Turing’s famous test used to determine machine intelligence and social performance is essentially an acting test. It measures not whether a computer is intelligent or can think like a human, but whether it can perform as if it is human, or at least whether it can perform well enough to fool a human audience. Turing set out this test for machine intelligence in his influential 1950 essay Computing Machinery and Intelligence [8] where he describes the scenario for an ‘imitation game’ to test whether a computer can successfully imitate a human being. Turing based his test on an earlier game where an interrogator tries to guess the gender of two participants (one male and one female) by asking them questions and assessing their typewritten replies. In Turing’s version of the game, he replaces one of the human participants with a computer and suggests that if the interrogator cannot tell the difference between the human and the computer purely from their answers, then the computer can be said to be intelligent. In this way intelligence becomes a functional attribute achieved through persuasive simulation or ‘passing’ rather than an inherent attribute.

‘Passing’ or successful simulation means getting it ‘just right,’ but over-performance and under-performance are more common features of machine performance. Over-performance and under-performance may be perceived in a variety of different ways and can have both entertaining and unsettling effects on audiences. Exaggerated appearance and behavior, including over-emphasized facial features, expressions, gestures and movement are common features of cartoon animation and animated films, where these techniques are successfully used for comic effect and to enhance emotion and drama. More unsettling are the uncanny responses evoked by robots and digitally animated characters that are ‘almost but not quite’ human in their appearance and behaviour; these responses have been described by Japanese roboticist Masahiro Mori as the ‘uncanny valley’ phenomenon. [9] [10] These unsettling effects occur when the mimetic aspiration of the work falls just short of achieving a perfect simulation. While audiences generally find lifelike or human-like characteristics in a more abstracted form appealing and empathetic, when these characteristics become more realistic (but not quite right), audiences tend to focus more on the disparities and what is not working about the simulation. The human brain perceives these imperfect simulations as defective versions of the real thing.

As we have seen, audiences judge robotic performances in the same way as they judge any other type of performance interaction whether they occur in everyday social settings or in more staged theatrical environments. The success of the robotic performance depends on two key factors, the intended performance, the robot’s appearance and its ability to enact or simulate behavior, movement and interactive responses (to its environment and other entities/actors) and the perceived performance, the audience’s perception and interpretation of the robot’s appearance, behavior and interactive responses.

References and Notes: 

  1. Philip Auslander, “Humanoid Boogie: Reflections on Robotic Performance,” in Staging Philosophy, eds. D. Krasner and D. Saltz, 87–103 (Ann Arbor: University of Michigan Press, 2006).
  2. Louis-Philippe Demers, “Machine Performers: Neither Agentic nor Automatic” (paper presented at the 5th ACM/IEEE International Conference on Human-Robot Interaction, Osaka, March 2-5,  2010).
  3. Steve Dixon, Digital Performance (Cambridge, MA: The MIT Press, 2007).
  4. Yuji Sone, “Realism of the Unreal: The Japanese Robot and the Performance of Representation,” in Visual Communication 7, no. 3 (2008): 345–62.
  5. Erving Goffman, The Presentation of Self in Everyday Life (New York: Doubleday Publishing, 1959).
  6. Byron Reeves and Clifford Nass, The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places (New York: Cambridge University Press, 1996).
  7. Sherry Turkle, Alone Together: Why We Expect More from Technology and Less from Each Other (New York: Basic Books, 2010), 26.
  8. Alan Turing, “Computing Machinery and Intelligence,” in Mind 59, no. 236 (1950): 433–60.
  9. Masahiro Mori, “Bukimi No Tani [The Uncanny Valley],” in Energy 7, no. 4 (1970): 33–35.
  10. Kathy Cleland, “Not Quite Human: Traversing the Uncanny Valley,” in What was the Human?, eds. L.E. Semler, B. Hodge and P. Kelly (Melbourne: Australian Scholarly Publishing, forthcoming). Available at