Flying Robotic Arts for HRI and Interface Research

The SAILS research program consists in developing geometric objects that can stabilize, move and rotate in the air, and that are able to develop behaviours and assemblages through emergent processes. This paper will focus on the qualitative results of this massively interdisciplinary project, which presents many examples of situations where questions and needs required by the art realm have led to technological premieres.



1 - Introduction

The use of machines to convey impressions and emotions is everything but new. They are reports of vapour-activated automata, designed to impress worshippers in ancient Egypt. Human-like or animal-like automata were a trend in XVIIIth century Europe.  As impressive as they might have been, these machines were seldom considered as art pieces at their time : they were more considered as pertaining to advanced crafts than to Beaux-Arts. Some were seen as amusement, or as demonstrators for their author's technological skills; others were scientific devices, meant to explain or understand biological phenomena.  It is only by the end of the XXth century that robots began to be created for solely artistic purposes. From then, the progressive development of efficient automated learning processes, supported by the availability of sensors of all kinds, quickly led a part of the art community to create robots whose only objective would be to generate artificial emotions and impressions through interactive processes.


The SAILS research-creation program was originally developed for such purposes. Developed for now more than five years in the NXI Gestatio Design Lab, directed by Nicolas Reeves, it aims at developing autonomous cubic flying robots for artistic performances and exhibitions. It brings together engineers, scientists and artists in a unique collaborative work process. [1] [2] The Tryphon is the latest aerobot developed within this program. It consists in a cubic polyurethane bladder surrounded by a cubic exoskeleton made from composite materials. Its overall size is 225cm. The Tryphons' predecessors were the prototype M180t (180 cm) and M170t  (170 cm) Mascarillon, whose structure was made of basswood, and the M160c Nestor.  This paper will discuss five performances with the flying cubes that occurred within the last years, focusing on the reactions that these objects triggered among various audiences.

2 - ROM<evo>

This performance took place at the Quebec Museum of Civilization in 2006. Three flying cubes, floating in a closed room, were programmed to avoid obstacles. The entrance of a visitor would trigger a series of events. First, one cube would stabilize in front of him.  Then an external, adaptive video projection system would map on two faces of the cube the eyes of a hidden actress, whose voice, coming from hidden speakers, started a discussion with the visitor. The actress could actually see and hear the visitor through a microphone, a headset and several video screens. The scenario of the performance was similar to a robotic version of a Türing test ; the actress was instructed to speak as if she was a machine, unable to understand the subtleties of a human language. For the robot, every word, every sentence was to be interpreted as a code : a system where each expression corresponds to one and only one meaning. It was unable to understand second-degrees, analogies, or metaphors. It began the performance with almost no knowledge about humans, and tried to accumulate information through its limited exchanges with the visitors.

A wealth of intriguing reactions intervened. Small babies were particularly attracted by the cubes. This was all the most surprising : apart from the gigantic projected eyes, no element of them could recall anything from human or animal beings that babies generally like. One hypothesis is that the slow oscillations and movements of the cube, which constantly repositions itself through its sensors and ducted fans, was seen by the toddlers as evoking a soft, gentle giant organism.

Another visitor spent a long time trying to teach a poem to the aerobot. The interesting point here is that the language of the cube is diametrically opposed, on a semiological level, to a poem : where the former tries to look like a code, a closed system where the meaning of each word is precisely defined, the latter corresponds to a form of text that is open to various meanings and significations. One can make the hypothesis that the visitor, knowing that the cube was eager to relate with humans, was trying to teach him a form of communication - art - that remains the privilege of mankind ; and to tell him implicitly that no real exchange would occur without a basic knowledge of the role of art on human communication.

Another curious event occurred when an old lady came into the exhibition room and began telling about herself to the cube. She came several times, and she talked about her feeling of loneliness ; she explained that her children never visited her ; and so on. Such a speech is not that surprising for an old woman ; what is more intriguing is the fact that she could talk so freely about intimate matters to a huge cubical machine. Here again, we can only make hypothesis about her behaviour; but she certainly anthropomorphized the flying object to a degree. The presence of the projected eyes may have helped in that respect, but this is all but sure, since other visitors were literally scared by them. It may be more fruitful to compare this situation to a psychotherapy session : she may have seen in the cube an empathic entity that, precisely because it is so different from a human being, would not put any judgement on what she was saying ; she could talk to it as long as she wanted without detecting on the cube symptoms of impatience or boredom that people frequently encounter when they repeat the same story several times.

This event also led to unexpected kinds of interactions. When a visitor was entering the room, the air currents created by his displacement and by the opening doors drawn more power from the cube’s ducted fans, whose noise became louder. This created a supplementary reaction of the aerobot to a human presence. The same thing happened when groups of several excited youngsters came to speak together with the cube. Since their displacements were triggering air currents, and since the sensors detected more agitation, the cube looked excited as well, through its own reactions to the air flows, and through the reactions of its motors, which were striving to restabilize it.

Another intriguing observation has been made : the answer of the question to know if a human being was hidden somewhere in the system depended closely on the formation and education of the visitor. The people who where the most successful at finding the real answer were coming from the art community. Most scientists where convinced that the whole system was robotic, and that it included advanced modules for language interpretation and generation. It is worth remembering that no such modules existed in 2006, and hardly exist today. Further discussions revealed that the elements that led some people to find the answer where actually linked to very subtle changes and modulations in the eyes and voice of the actresses ; these changes could never, for them, be generated by a machine : they could only be human. It is precisely this kind of observation that has the potential to open research tracks in psychology, HRI, neurosciences, and the like. This experiment led us to consider the impact on visitors of a similar aerobot whose vocal interactions would use a completely artificial system, based on a voice-recognition program coupled with an artificial locutor. This development has become one of our priority research axis.

3. The Summers of Dance in Paris

No direct interactions with people were planned for this event, which was seen only as an opportunity to test autonomous interactions in difficult circumstances, with large audiences. The aerobots would try to mimic human movements through their own translation/rotation vocabulary. All displacements were made through assisted, partially automatic remote control. 

In the summer of 2008, we were invited to fly three Tryphon cubes for the Summer of Dance in Paris, within the gigantic nave of the Grand Palais. In such a space, the air streams caused by convection and differential pressures generate an aggressive flight environment. All the mechatronics and stabilisation algorithms had to be completely redesigned prior to the event, in order to ensure proper a behaviour of the cubes over a dancing crowd, in such a wide space.

Calibration times were particularly long because of strong air streams, and the system only worked properly on the second performance. The first performance had to be made through semi-autonomous remote control : the movements of the aerobots were partly induced by direct commands from the pilots, coupled with self-stabilization and collision detection. Our degree of control on the robots allowed us to make them act like dancers in the air, or even on the dance floor. Their movements were determined mainly by the reactions of the pilots to the music, which were themselves influenced by the general energy of the crowd : they became the equivalent of proxies for the pilots, of whom they became a sort of cubic flying embodiment. The audience seemed to share the space with a different and new kind of dancer ; the other dancers had to adapt to the presence of these huge moving objects, and to learn how to predict their movement so as to give them sufficient space to evolve.

4. Nestor and Veronique

This event took place in the Montreal Center for Sciences. It was the first completely interactive performance involving an actress and an autonomous cubic aerobot. It was based on a simple scenario, in which the actress would tame a Nestor flying cube by interacting with it through her movements and displacements. Each interaction was fine-tuned to be as precise as possible; but there is no way to predict exactly the behaviour of an aerostatic object in every possible circumstances. An interesting evolution occurred during the event : since the actress had to adapt her movements to the slow pace and to the unexpected movements of the aerobot, the performance progressively developed into a two-ways relationship, just like if the particular behaviour of the aerobot had influenced the behaviour of the actress. After several shows, everything happened as if the unpredictable movements from the cube induced the actress to react on an improvised, almost choreographic manner, which greatly enriched the performance. Some other unplanned reactions were caused by small drifts of the cube, when it was caught by sudden air streams ; they actually added an anthropomorphic touch to the cube's behaviour, since small errors and imprecisions are more easily associated with humans than with machines. To our knowledge, this performance was the first experiment involving an interaction between a human being and an autonomous  blimp.

5. Geometric Butterflies

This performance took place in Moscow during the spring of 2009, as part of a Science-Art festival. It was the first long lasting, totally autonomous performance for the Tryphons. The cubes were flying in an area surrounded by blue spotlights. They were instructed to avoid obstacles and to run away from light. The spotlights sent them towards the center of the flight area, where they met randomly with each other ; their obstacle avoidance algorithms sent them back to the periphery, where they would detect again the light sources, and so on. This sequence created continuous, unpredictable orbits.  When they were getting too close to each other, their collision detectors reacted quickly, at times sending them towards and over the audience. The approach of these huge objects was impressive : the visitors would quickly extend their arms to stop them. This was sensed by the detectors, which sent the cubes back to the flying area, creating a first occurrence of interactions between the cubes and a whole crowd.

Other kinds of interactions happened. Even when the Tryphons were in an near-stable position, the visitors tried to interact with them and to make them react through the light of their cellphones. This desire to communicate with the aerobots and to influence their behaviour through individual or group interaction brought us to the conclusion that any future performance with the cubes, even theatrical, must consider the audience as an integral component of the environment, and should benefit from its presence to develop new ways of interacting.

6. The Floating Head Experiment

This HRI-artistic public experiment took place in Montreal in 2010. It was the result of a many-months collaboration between our lab and the team of Australian artist Stelarc. [3] A Tryphon aerobot was combined to Stelarc’s work The Prosthetic Head, which consists in a 5-meters-high projection of a 3D avatar of himself, represented through a synthetic image, and linked to a chatbot-like engine in order to discuss directly with the visitors. Stelarc’s artworks development is supported by the MARCS Auditory Laboratory in the University of Western Sydney. This team has transposed the Prosthetic Head into the Articulated Head, a new version that is embodied via a LCD screen attached to an industrial robotic arm ; this added a strong level of expressivity to the Prosthetic Head : all facial expressions could be correlated to spatial translations and rotations of the screen, thus approximating the movements of a human head when talking or discussing. Thanks to various sensing devices, the team managed to develop an attention model, called THAMBS (Thinking Head Attention Model and Behavioural System), to enhance the effect of presence triggered by the synthetic face.  

We quickly realized that the floating cubes of the SAILS project could play the same role as this screen, without the limitations imposed by a fixed stand. Through intensive international collaboration between our teams, we managed to realize a performance during which Stelarc’s synthetic head was projected onto a large floating cube, whose movements and displacements in the air conveyed the head’s emotions and impressions to the audience, while strongly enhancing the effect of presence of the virtual face.

 7. Conclusion

The visitors’ relations with the robots, as well as the actors’ relations with them, proved a valuable and meaningful source of information and knowledge for the design of HRI procedures. In each performance, the visitors were seeking a contact with the robot, and initiated interaction processes. They may have wanted to test their influence on what they saw as a large machine, or simply test its abilities ; but more fundamentally, what could be gathered from our discussions with them revealed that they were attracted by the strange appearance of these slowly moving cubic organisms to which they would attribute a personality and a kind of proto-consciousness ; and that these artificial creatures put the visitors in touch with the fundamental human desire to communicate, even with an entity whose all elements betray the artificial nature.

It has also been observed that a slow moving cube was creating more positive feelings : visitors who accidentally triggered brisk reactions from one aerobot were  generally impressed, and even afraid, when they realized that they were the cause of it. These reactions by neophytes proved quite informative in setting a preliminary common ground [4] for optimizing human-robot communications.

The development of interactions for performances followed a different path, since the performers had to get acquainted with the unpredictable reactions of the cubes that were caused by the parameters of the flight environment. This can be seen as a problem from an engineer’s point of view : the repeatability and reliability of interactions are almost always essential for any technological applications. Things are very different in art ; in this project, unpredictability may become a key ingredient in the definition of each cube’s personality. Interactions whose repeatability ranges from "nearly precise" to "moderately imprecise" add to the poetic dimension of the cube, facilitating their assimilation to living organisms. These last observations leads to one of the main conclusions of this paper : endowing robots with basic sensory and interaction aptitudes, and putting them in close contact with humans in a given context and an intentional frame, provides a wealth of observations and data that can be fruitfully exploited for the design of optimal human-to-robot interactions.

From their artistic origin, [5] and throughout all the artistic development that surrounded their technological evolution, the flying cubes demonstrate how several disciplines can contribute to the creation of objects that escape any attempt for classification, and hover over the undefined territories that stretch around the borders between arts, science and technology. We will borrow our conclusive sentence to the late Stephen Wilson : «Artists should be hungry to know what researchers are doing and thinking, and scientists and technologists should be zealous to know of artistic experimentation.» [6]

References and Notes: 
  1. N. Reeves, J. Nembrini, E. Poncet, A. Martinoli and A. Winfield, “Mascarillons: Flying Swarm Intelligence for Architectural Research,” Proceedings of the IEEE Swarm Intelligence Symposium (SIS 2005), Pasadena, CA, (June 2005). 
  2. D. St-Onge, N. Reeves and C. Gosselin, “A Modular Architecture for a Fast Parallel Develpment in an International Multidisciplinary Project,” Proceedings of the 15th Conference on Advanced Robotics (ICAR 2011), Tallinn, (June 2011). 
  3. D. St-Onge, N. Reeves, C. Gosselin, D. Herald, C. Kroos and Stelarc, The Floating Head Experiment, Proceedings of the 6th IEEE/ACM International Conference on Human-Robot Interaction (HRI 2011), Lausanne, (March 2001). 
  4.  K. W. Lee, J. H. Hwang and D. S. Kwon, A Comparative Study between Human-Human Interaction and Human-Robot Interaction, in New Challenges in Applied Intelligence Technologies, SCI 134, eds. Ngoc Thanh Nguyen and Radoslaw Katarzyniak, 2-12 (New York-Heidelberg: Springer- Verlag Berlin Heidelberg, 2008).
  5.  N. Reeves,  J. Nembrini, E. Poncet, A. Martinoli and A. Winfield, Voiles/Sails : Self-assembling Intelligent Lighter than Air Structures, Proceedings of the 10th Generative Arts Conference (GA2005), Milan, (December 2005).
  6. S. Wilson, Information Arts: Intersections of Art, Science and Technology (Cambridge, MA: The MIT Press, 2002).