Tactile Video Love Letters: finding new modalities for non-verbal communication

This paper will discuss new research repurposing the mobile phone from a textual and voice device to a more multi-modal, synaesthetic, tactile, expressive, and gestural device. ‘Tactile Video Love Letters’ seeks new modes for individuals to express themselves intimately, visually and non-verbally––akin to remote ‘touch’, immediately and intuitively understood in a pre-conscious sense––direct and tactile route to interpersonal communication.


Tactile Video Love Letters: finding new modalities for non-verbal communication

Dr. Camille Baker, Lecturer 

School of Engineering and Design

Brunel University, London



This paper will discuss new research that repurposes the mobile phone moving it from a textual and voice device to a more multi-modal, synaesthetic, tactile, expressive, and gestural device. The project ‘Tactile Video Love Letters’ seeks to find new ways for individuals to express themselves, in an intimate, visual and non-verbal way––akin to sending remote ‘touch’ messages, immediately and intuitively understood in a pre-conscious sense––direct and tactile route to interpersonal communication.

Conceptual Approach 

Currently, we live in a world where we are increasingly more distant from our friends and families. Mobile video cameras can be repurposed to visually convey emotions and sensations, rather than merely a device for documentation of events, or as an entertainment gadget. As such, this new project ‘Tactile Video Love Letters’, aims to continue to bridge this emotional and physical divide between users.

Based on previous neurological research on human development [[i]], it is evident that humans need in-person, physical contact (as do all mammals) and interaction to establish and maintain intimate bonds, as well as to continue to develop strong neural pathways in the brain. According to psychiatrist neuroscientists Lewis et al (2001), this physical contact is essential for survival (and the only authentic, tangible, focused interaction). Therefore, communicating via written word, voice etc. is only part of this interaction.

Portable devices and increasingly ubiquitous forms of mobile media and communication devices have transformed the modalities in which we can communicate our affective and emotional “states,” however they are still predominantly via text and language.   Mobile images are often experienced as personal, intimate, and private expression. Once sent, the immediacy of these media images feels like a receiving a virtual kiss blown to you or of invisible pieces of them [[ii]] through the network. Therefore, if one thinks of a text message as a thought transfer, then an image or video is sensory or sight transfer, sending visual experiences, feelings, as well as one’s unique expression and perspective. Thus, people might wish to send their internal perceptions, instead of relying solely on words to communicate when using the mobile phone.  

Tactile Video Love Letters’ involves developing a novel method to repurpose the mobile videophone, using wearable technologies and smart textiles. This fresh approach hopes to replicate physical experience as much as possible.  The system will use both the mobile connection using a wearable interface for tactile interaction. This will create a reciprocal exchange with a way to reply to a mobile message sender in a non-verbal, embodied, multi-sensory, two-way dialogue. Thus, this media project studies to relay ‘felt’ experience or touch sensation ways through a sensitive interface that is a skin-like membrane or bio-material responds with a reply directly to the recipient's mobile phone application.

To address the problem of extended absences, global travel, and distance relationships, methods will be developed to enable the translatation of emotional content contained in the visual construction and patterns of mobile video messages into ‘touch’ messages. Visual interpretation software technology will be designed to transform and transmit the video messages recorded by mobile videophones into physical feedback and sensations. These emotional video messages will then be sent through to the tactile textile/membrane’s interface to pass the message to the person directly through the surface of their skin, by contracting, emitting heat, pulses or vibration and other such actuation outputs. Also under investigation are methods to relay the ‘felt experience’ or touch sensation back through to sensors, via the garment, to send a reply or notification response directly from the garment to the initial sender’s mobile phone. This is intended to better enable a person to have more realistic 'contact' or interaction with a friend or family member at a distance. 

CuteCircuit designed the first mobile connected sensory wearable garment with their HugShirt© in 2006. Others have since followed from their lead, with variations of mobile-to-garment projects. Tactile Video Love Letters builds on the innovation of CuteCircuit’s ground-breaking, interactive garment, the HugShirt© [[i]]. It aims for a two-way exchange and experience, with sensitive textiles, and mobile video, taking this interchange to a more complex level, with mobile connection to skin interaction. To achieve these aims involves a systematic approach in three key areas: video messaging lexicon development, sensor/textile interface design and mobile media application programming and development.

Tactile Video Love Letters asks users to draw upon the visual material in their environment and any imagery they think relevant or essential to their expression. The ‘tactile textile to video’ exploration will develop a customised video symbolic vocabulary or lexicon to express and construct visual ‘sentences’ or ‘utterances’, to then be translated into distance touch or embrace. The intention is to create a structured semiotic system for expression, especially for people with physical, verbal or linguistic challenges, to use for personal interaction and to have an embodied, tactile and visual means of messaging [[ii]].

Through a participatory design processes project will develop a method to use videophones and mobile imagery together as an alternative to the current textual or voice uses. It will utilise visual patterns and common approaches to the use of visual material in one’s environment, including users’ movement and gestures with the device. As the sensory experience/conceptual/interaction designer and facilitator, my role is to explore the physiological tactile experiences of emotion on the body and brain of the message receiver, as well as to develop (with the aid of the users) the video language used by the video message senders [[iii]].

The first step for this  ‘tactile video’ research, is the development of a symbolic video message lexicon to express with. This involves:

1) exploiting the possibilities and limitations of the mobile device and its video technology;

2) studying users’ comfort with the device;

3) understanding the limitations of available environmental visual material as an aid in communicating such messages;

4) testing/studying users’ ability to utilise the visual environmental material to represent their emotions, internal perceptions, sensations, or experiences;

5) studying visual content, meaning, patterns and representations to create a mapping of symbolic vocabulary to translate into touch sensations;

6) create structured activities for test users to be guided through, in order to find ways to express or record patterns and other visual elements as meaningful messages.

From this process, a vocabulary or syntax of representations of internal experience should emerge.

User interaction workshops for this process will be conducted starting before the end of 2011. This will begin with a designed set of structured improvisation activities to explore various types of expression as the impetus for developing a symbolic code for non-linguistic video expression. These workshops will be based on previous PhD research (Baker, 2010), where it was observed that the key aspects of mobile video-making process include these four features: 

1.     an innate performativity, movement, gestural qualities, afforded by the device size and its features;

2.     portability factors (i.e. ability to watch or shoot anywhere), which allow one to notice the mundane more acutely, and causing people capture all that catches their eyes;

3.     the phenomena that users start to view the world through the camera ‘vision’ of the mobile screen, rather than their eyes alone, adding novelty, and a re-engagement with the scenery;

4.     an innate intimacy of expressivity, fostered by ready access to the device, and the predilection of users to record close-ups – framing only that which is to be seen, allowing for abstraction.

In previous user recording workshops, users were observed capturing their experiences and exploring the immediate space of their bodies, as the main intrigue and focus of the mobile medium. This has become a common approach to mobile video capture, as other artists and researchers have found [[iv]]. It was found that the device inherently encourages movement, often resulting in a smearing effect making abstracted patterns through gesture. This instinctive approach to mobile video recording is facilitated or afforded by the device size itself (possibly not intended by the manufacturers). Thus, it appears that users’ immediate impulse is to wave the device around, as if it was part of their hand, blurring the images intentionally to experiment with the visual results. To exemplify the gestural and intimate aspects of mobile recording practice, artist Dean Terry states that the phone encourages one: 

[…] to project [the] very private space immediately surrounding the body into meta-space. Many of the videos show objects little more than a few centimetres beyond the tiny lens, often some body part, like hands or forearms that obscure an unknown, overexposed background space. Other pieces are gestural performances, recording the movements required when following a line, or when trying to create shapes by moving the camera in certain ways (2005, my emphasis).

One can only conclude that the videophone inspires a playful, gestural, or performative exploration resulting in a new video aesthetic unfolding merely of the qualities of the device.

A new modality for non-verbal communication

The primary dimension of this new project is in determining what non-verbal, visual communication can result when users are asked to express a certain types of inner experiences or perceptions using videophones.  The aim is to harness users’ own ingenious nuances of representing their inner sensations and emotions through the imagery. Thus, the focus of my work has shifted from facilitating non-verbal, internal, visual and synesthetic expression.

Non-verbal communication generally involves using eye-contact, gestures, body language, touch, tone of voice, and other physical indicators, to express certain information to others without words [[v]].  It can also be used to highlight a conversational or linguistic exchange, or to make a point more clear. Sometimes it is a mode of communicating the unspeakable or a way to interact prior to formal verbal exchanges or introductions being made. It is a means of initiating contact, or expressing something intimately without the need to verbalise the sentiment. We all use non-verbal modes of expression and learn them at a very young age, possibly before we are able to speak.

In his research on mobile phone use, Murtagh noticed that non-verbal behaviour:

[...] focus[es] on bodily gesture and eye contact emerged from the data when it was noticed that activities with and responses to mobile phone use were almost invariably non-vocal in nature […] Yet, so much of mobile phone use in public is organised through non-verbal action and interaction. It is suggested that these non-verbal aspects of phone use display the 'unwritten rules' of usage behavior in public. (2001: 82)

In my previous research, participants were asked to use videophone imagery to speak for them, to draw upon visual material within their immediate environment, as well as from their own sensations, perceptions, thoughts, and emotions, to share with others without words. Through gestures and movement it was evident they could represent their internal experiences via the external world visually and could have meaningful non-verbal expression. This modality falls outside of the usual physical or body language and is more akin to the cinematic language [[vi]] of video art, abstract cinema, audio/visual performance. This method of non-verbal expression does not require in-person presence, but could be transmitted in a visually with a mobile connection. This digital mode of remote non-verbal expression transmits different layer of emotional presence, beyond tone of voice or turn of phrase that is currently used. The embodied interaction through technology replaces the in-person physicality.

Also to be investigated here, is how a receiver of video messages might experience and interpret these messages, especially if the messages are abstract, emotional and visceral. A direct, literal approach to constructing the visual expression may be the key to meaningful exchange (or may not be). Thus, one option is to construct a semantic language or symbolic representation for video messaging. While it can be argued that, like a dream or an artwork­ – the receiver/viewer/dreamer should also be free to interpret the message any way they like, in some cases the representative imagery is clear enough in content that the message is evident and obvious [[vii]]). Yet, the way the communicator constructs the message can also help in its interpretation and how it is decoded, especially if they use recognisable signifiers and symbolism (see Image 2 below). As such, this research will develop activities to develop a visual lexicon, building on past research that can provide tools and a structure to guide users to create a video language of their own. The intention is to develop a coding system for direct personal, embodied, sensual, and meaningful visual messaging using synaesthetic possibilities of mobile expression. The goal is to enable the average, non-artistic user to communicate in playful, creative, intimate, and novel ways [[viii]].



This project hopes to help people to maintain close personal bonds when it is difficult to be phyically present or infrequent. Yet, most people likely do not seek a digital replacement for in-person contact with the spouses/partners, friends, family, or business associates. Ultimately, fleshy, corporeal, in-person interaction will always be the most sought-after and ‘real’ experience. We have evolved to a condition in which this the optimal form of clear human understanding, trust, survival, and connection between people. Global travel for business, pleasure, and trade necessitates that we develop better solutions to bridge the gaps between face-to-face interaction and immediacy to maintain relationship. The only reasonable justification for substituting fully embodied experiences of in-person interaction are to save money, resources, lives, and other life threatening activities that otherwise cause problems in our current socio-economic, political, and environmental state of the world. Instead of using 3D surround projection TV to bring us closer to a sense of real, live, in-person experience with others, this project seeks a more sensual, tactile, present, and cross-sensory approach.

Artists and scientists will continue to develop alternatives to in-person contact, ones that feel real and authentic, both emotionally and physically. This media art research will expand on previous work by continuing to seek new interpretations of everyday technology, to improve the quality of life through non-verbally express and transfer messages through various embodied modalities. As always, my overall goal is to devise deeply experiential interactions, in order to improve and expand communication options and traverse vast distances through creative, multi-sensory modalities.  

This project is only beginning so all suggestions are welcome.


References and Notes: 
Amerika, Mark (2009), Immobilité online film [Online] Available at http://www.immobilite.com/  (Accessed July 12 2010).

Baker, C.C. (December 2010) “MINDtouch– Ephemeral Transference: Liveness’ in Networked Performance with Mobile Devices”, PhD Thesis, published in University of East London Library.

Christensen, M. S. As We May Feel: Interpreting The Cultural of Emerging Personal Affective Mobile Media, PhD Dissertation, Denmark: Department of Innovative Communication at the IT University of Copenhagen, 2006.  

Lewis, T. MD, Amini, F. MD, Lannon, R. MD (2001), A General Theory of Love, New York: Vintage Books division of Random House.

Johansson, T.D. ‘The Tactile Image: A Veronican Approach to Pictoriality in Michel Serres’s Aesthetics of the Senses’. Sense and Senses in Aesthetics, Backstrom, P. and Johansson, T.D (eds.) NSU Press: Helsinki, Finland, 2003.

Murtagh, G.M. (2001), ‘Seeing the 'Rules': Preliminary Observations of Action, Interaction and Mobile Phone Use.’ In Wireless World. Social and Interactional Aspects of the Mobile Age, Barry Brown, Nicola Green and Richard Harper, (eds.), London and Berlin: Springer-Verlag UK, pgs 81-91.

Terry, Dean. mo.vid.1 (2005) video paintings, from a camera phone. [Online] Available at www.100lies.com (Accessed Dec 2008).