Chameleon: A Study of Empathy
This paper will discuss Chameleon, an interdisciplinary video project exploring empathy and emotional contagion. Chameleon, shown at ISEA as part of the Australian Forum, is a six screen video installation, foregrounding emotional contagion and empathy. This paper will focus on the development of the cross cultural emotional expression video portrait database built for Chameleon.
My work has always explored aspects of the intimacies and vulnerabilities of being human. In the past I have explored the emotional signatures of our bodies, using pulse, sweat, prosody and movement as agency for moving image interactive works that highlight the nuances of emotions and its importance in our lives. 
Universal Emotions and Cross Cultural Nuances.
When we feel emotions, they tell us something important about ourselves, our relationship with the world and our relationship with each other. The ability to read emotions in both others and ourselves is central to empathy and social understanding.
Researchers suggest that over eighty percent of human communication is encoded in emotion facial expression and body movements. In the 1960’s Psychologist Paul Ekman traveled the world, from the USA to tribes in Papua New Guinea, showing pictures of facial emotion expressions to people. Following anthropologist Margaret Mead’s theories, he had set out to establish the differences in emotion expression. To his surprise, he found more similarities in the way people express and understand emotions than differences. He concluded that the facial expressions of some emotions are not culturally determined, but universal across human cultures and thus biological in origin. He established that the five universal emotions that all cultures can read and respond to are fear, sadness, happiness, anger, and disgust. This research lead Ekman to develop the Facial Action Coding System (FACS), a system for deciphering which of the 43 muscles in the face are working to express emotions at any given moment. Using FACS lead to a verifiable database of facial expressions representing emotional states. This research revolutionized the study of emotions and although these images and research is now nearly forty years old, this verified database is still prominently used in most emotion neuroscience research today.
With in our own culture, the understanding of emotional expressions happens automatically and without much conscious awareness. We are highly attuned to subtle and covert emotional signals and people automatically and continuously synchronize with the facial expressions, voices, postures, movements of others. Through the dance of unconscious mimicry, we become carriers, infecting each other with our emotions, forging a bond with each other long before we utter a word, blending into the tone of our emotional environment. The face informs the self, not just others. One of Ekman’s most fascinating findings is that if a person merely arranges his face into a certain expression, he will actually feel the corresponding emotion. This is the beginning of how we catch each other’s emotions. Scientists call the mimicry of social situations the ‘Chameleon Effect’.
However when we are taken out of our ‘in-group’, one discovers quickly that cultures do differ considerably in their use of emotional expression and cultural display rules vary about when, where, and how one should express emotions, how these emotions are experienced, the reactions they provoke and the way they are perceived. For example, research suggests it is Russians, closely followed by the Japanese and South Koreans, who most tightly control the display of their emotions, which may make Americans (who display the least control over their facial expressions) feel uncomfortable. In the West, we are more focused on the individual, with more attention to our own inner states and feelings, in the East one reads the emotions of a group with a lot more ease. As the complexity of our society broadens, when communicating, the overlay of cultural display rules can often get ‘lost in translation’, leading to a membrane of disconnects and negotiation.
Building Chameleon: An emotionally responsive video portrait project exploring empathy
With Chameleon, an aim was to investigate the social role played by the unspoken language of emotional expressions while exploring personal, technological and scientific biases and nuances.
To build Chameleon, we focused on three parts, the emotional code that triggered the video portraits was developed with neuroscientists Chris Frith, Hugo Critchley and Bruno Averbeck. The facial emotion reading technology that monitored the emotional expression of the audience was developed with affective computer scientist Rana El Kaliouby and Ros Picard at the MIT Media Lab. I directed the video database developed with everyday people sourced over the world. While developing Chameleon’s code, the video database and interaction, we worked with Nadia Berthouze to analyze user experience. Working with curator Helen Sloan, we decided to take a more experimental and intuitive approach to exhibition, developing the work in nine progressions, turning each exhibiting experience into a ‘lab’. We exhibited at a range of venues, including hospitals, museums and galleries.
Chameleon’s emotional video portrait database:
Most scientists agree that we respond differently when we look at an image of a chair, and an image of a face. Intuitively I know when I see a face, I can feel my body shift, an awareness takes over my body. Science tells us that with the perception of faces, major activations occur in the brain particularly in the fusiform gyri. I was interested in creating a work that exploited our biological predisposition towards faces. 
I was interested in creating a new, more dynamic video database for emotion studies, and I hoped this could be of use for science.  It was important to shoot the work cross culturally to explore the cultural, social and individually determined responses to the six universal emotions being explored in Chameleon. Faces are rich and varied, a brief glance of a face can provide us with knowledge to the individual’s gender, their origin, their emotional state, their familiarity to us, their personality, their attractiveness or interest in us which in turn, influences our emotional response. I wanted to pick up on these nuances.
Over a year and a half, I set off over the world, asking volunteers to be filmed expressing emotions. Shoots often took place over weeks. The subjects were shot in a private studio space with a neutral black background, with simple lighting.
Technical considerations for the shoot
As a video artist, it was important to create Chameleon’s video database using the medium of video. Although a 3D computer graphic rendered model of portraits would have been a much easier to create, easier to control, and more fluid, I was adamant that I wanted to document real people as part of the database. I was concerned about the ‘uncanny valley’ affect of 3D imagery. The theory holds that when renderings of people look and act almost like actual humans, they become overly "strange", thus will fail to evoke the empathic response I was looking for. At the beginning, I attempted to adhere to high production values using 3 HD cameras, large studios. I realized I needed to release this ideal and respond to the people I met everyday. I had my camera, a black sheet and made do with ad-hoc lighting so I could work with a range of people, in a range of spaces, in range of countries. I attempted a lot of continuity in post-production.
When creating the interactive design of Chameleon, it was integral to develop a novel interactive solution that matched both the conceptual and metaphoric content of emotional contagion. For a work about social emotions, it was an obvious choice to explore automatic facial emotion reading technology. It allowed for group interactivity, no training was needed with the audience and the monitoring of emotional state could happen from a distance, therefore a more fluid and seamless interaction loop could develop. Conceptually, interaction mode was delicate, provocative, and by analyzing the face, it was intimate. It also elicited awareness of these types of emotion recognitions technologies (both with fear and intrigue). We developed with the software with the MIT Media Lab (the software is called FaceSense). 
When designing the shoot, I was very aware that when the audience viewed the final video portraits, the emotional impact of the video portraits would need to trigger an unconscious emotional expression response in the viewer that the facial emotion expression technology needed to recognize.
Emotion Eliciting Techniques in the Studio
I elicited the emotional states using various techniques guided by my collaborators and others (psychologists, psycho-analysts, acting coaches, actors). For example, with disgust, I would ask the participants to view footage of people vomiting (that I had shot previously for a collaborative project about disgust). For surprise, I would talk calmly and then scream as loud as possible. For sadness and anger, I employed classical psychoanalytical techniques such as encouraging the volunteers to imagine different personal emotional scenarios from their past and to re-enact them as if in the present. I would often discuss personal emotional memories while shooting the participants facial emotion expression response to the stories. The studio time became a very intimate, vulnerable, trusting and often moving process for all involved. For a shoot to work, I learnt to listen attentively, subjects became friends.
Cross-cultural effects revealed themselves. One participant from Portugal displayed barely discernable emotions. North American participants were often the most vocally and facially expressive. Generally, participants from the UK had a harder time expressing anger. I wrote in my studio notes in March 2008 while working in Canada “...It’s taken a while to get comfortable asking people to evoke emotions. It’s been exhausting, because it feels so personal. It’s been a varied response, ranging from deep deep crying for half an hour to more laughter and very light expression..” Whereas my notes in April 2009, working in Paris “It’s harder to coax Parisians to reveal emotions… I need to spend more time getting to know participants, to develop a more trusting relationship. The studio time needs to be much longer than it was in Canada…”
On reflection, despite cultural differences, sadness was the easiest emotion to elicit. Most of the shoots resulted with the subjects in tears, sometimes deep crying lasting for up to an hour. I had to halt my intuitive response to step in front of the camera, to hug them, and make them feel better. I could only watch from behind the camera, and at most times, I ended up crying as well. After the shoot I asked the participants to reflect on the experience. “I felt messy and really really sorry for myself and very very lonely which made me feel even sorrier for myself… I felt like she (the artist) had been my therapist and that I owed her £50 for the session”. (email correspondence with artist August 2009)
I had developed a database of emotional expressions that were both felt and also acted. At first, I was on a ‘search’ for authentic, felt emotions, however, I soon realized that this was not important. In everyday we express emotions that are not ‘felt’, and their primary purpose may be some sort of manipulation . In everyday life, we seem to have an embodied awareness which drives a constant search of each others’ faces for truth. I also wanted this dynamic to evolve in Chameleon.
By the end of the project, I shot 30 participants, adding up to a 23-hour database of mostly non-verbal facial emotional expressions.
This material was treated carefully. Some footage was edited, as the footage would have been detrimental to the participants, as it was too private. Some footage was given back to the participants, to make their own works from. I edited the work by assigning in and outpoints for each emotion. I wrote in my blog in August, in Brighton 2009: “…I have built up compassion and attentiveness in the studio, I then have sat and analyzed and categorized that footage – more as looking at narrative and science of what an emotion is. This process has felt harsh – as if I am fragmenting a lovely relationship, objectifying it, making it into a production”.
Audience Response to Chameleon
The work was exhibited often, with most exhibition venues providing a chance to evaluate audience interaction. We tested a range of scenarios, with different types of screens (3D, low pixilation, High Definition, sculptural), displaying different portraits. We worked with the UCL Human Computers Interaction Center to evaluate the work.
Throughout the interviews of participants, feelings of intimacy came up repeatedly in the interviews although they were not explicitly part of the questions. In many cases the audience was affected by the emotions expressed by the characters, and the constant search for meaning and introduction of context generally followed this. “I was thinking of some sad things that happened to me, when [...the digital portrait] was sad for a while, it felt like a long time, and it reminded me of some things.” Also, “I didn't like it when he looked sad and I didn't know why.”
A goal of the work was to bring up reflective questions about our own emotional expressions. “I made a man start to scream, which was a little worrying - did I look like I needed to scream? Did I look frustrated? I then of course looked worried, which made someone else smile to make me feel better...” Another reports that the lack of interaction had made one questions there own facial expression “maybe I have got a tired face, umm… and sometimes when I am not smiling people say to me ‘oh cheer up, as if your … you know, maybe I do give that off instead of my feeling like I am emotionally upset or angry.”
Exciting Initial feedback suggests that the contagion of emotion leaks out of the gallery spaces. “I went out afterwards and felt like I was picking up the feelings of everyone I passed.” My hope was that Chameleon reminds as that our body as a clear-cut distinction with the rest of the world is dissolved, revealing that we are all interconnected.
The Art and Science collaboration
Chameleon successfully brought together a genuine and rare collaboration across the boundaries of arts and science, creating art installations, research papers, and novel, more dynamic models for scientific research. I was very aware, in choosing the title Chameleon. It related to the project, but it also reflected my artistic role. As I collaborated, looking at my work through new lenses, I was moderating my language accordingly, depending on whom I was working with. My work was becoming a synthesis, influenced by the qualities of multiple collaborators, attempting to reach a balance that would meet the needs of each collaborator. I felt like a Chameleon, shifting, adapting, a ‘changing self.’
The scientists involved in Chameleon reported feeling a sense of ‘freedom’: Chris Frith sees the cross disciplinary collaboration as liberating: “This project has developed far beyond what I would dare to do in the carefully controlled experiments that we are restricted to. But the end result will provide us with marvelous tools for doing new experiments.” Nadia Berthouze writes: “I see Chameleon as a source of ideas for the creation of digital environments conducive to patients becoming aware of their emotional states”. Rosalind Picard, who is mentored the building of the face reading technology, is interested in its use for people with autism: “As I watch people to learn from the interactions portrayed. These are scripts with naked emotion, uncovered, and whether ugly or beautiful, they are hard to turn the eyes from. Here is an engaging palate for helping people who don’t naturally understand emotional interactions, and who want to deepen their ability to do so”.
I would like to acknowledge the in kind support from the MIT Media Lab, Banff New Media Institute, SCAN, Fabrica and Institute of Neurology at UCL. The project is funded by the Wellcome Trust, Australian Network for Art and Technology Synapse Residency, Arts Council England, Lighthouse and the Australia Arts Council.
- I had had an interest in dynamics of portraiture for years. In earlier work such as Feel: Trace (2005), and Feel: Ferment (2006), I had began to explore the static facial emotion recognition databases most often used in emotion research such as the Ekman and Friesen Database, and the Karolinska database. In Feel: Insula (2007), I had begun the creation of a new moving image portraiture database that searched for a more authentic expression of emotion. Over a few months, I worked with clinical hypnotist, David Oakley, asking him to hypnotize me into different emotional states to create a more genuine representation of emotional expression. This became part of a video installation, and also the voice track for a short film, Feel: Melancholia (2007).
- A large percentage of current scientific experiments exploring facial emotion expression uses Paul Ekman’s 1970’s visual database of static facial expressions representing emotional states. Since I finished the database in 2010, I have had quite a few requests for the database to be implemented in science studies.
- Please visit http://www.tinagonsalves.com/chamcatalogue.html to download an interactive catalogue about Chameleon.