The Metapiano: composing and improvising with sculpture

This paper concerns the implementation and control of interactive sculptural interfaces, designed to be used by the public as well as by musicians or dancers. The main device under consideration is called the Metapiano, an interface developed to trigger and modulate an array of musically expressive algorithms. The paper describes interactions with hardware and software systems and the nature of the music created.


1. Introduction

This paper concerns the design, implementation, demonstration and control of interactive sculptural interfaces, designed to be used by the public as well as by specialist performers, such as musicians or dancers. The main device under consideration is called the Metapiano, an interface developed to trigger and modulate an array of musically expressive algorithms using a (synthesised) piano as the primary sound source.

The paper describes the manner in which a viewer/performer interacts with hardware and software systems, examines the nature of the music created, and details how the two are related.  Of particular interest is the way in which the resulting music relates to both new and more traditional forms of composition, performance and improvisation.

2. Related Work

With the advent and growth of accessible methods of physical computing, generative composition and real-time synthesis, there has been a similar expansion of interest in more practical and physical aspects of music making.  That which was considered some years ago as a fringe activity for more experimental musicians and engineers (Theremin, Partch, Cage), has become increasingly mainstream and has enabled progressively more intimate collaborations between the arts.

This growth has been primarily focused on the musical interface - digitally through the graphical user interface, physically through the invention of new instruments and generatively through the development of algorithms to enhance, replicate, replace and analyse, human activity. [1, 6] Of course, analyses of human behaviour have demonstrated the complex relationships between the activities that can account for our behaviour – strategies, planning and construction – as well as our direct physical interference with the world. [10]

These developments in software have been reflected in movements in the cross-disciplinary field of physical interaction with software through hardware. The introduction of the Arduino in 2005 has been seminal in encouraging those from more creative disciplines to investigate issues of HCI and expression. The practically based work of Nic Collins, Perry Cook and O’Sullivan and Igoe is of particular importance here. [4, 5, 9]

The work of Alexander Calder and Ives Tinguely is clearly significant to the physical form of the metapiano, while its use as a musical interface adopts some ideas developed in Earle Brown’s ‘Calder Piece.’ [3]

3. Triggered, Gaggle and Wired

Triggered is a dance-music project that has been evolving since 2009. The core musical group of Tom Hall, Cheryl Frances-Hoad and Richard Hoadley was invited to collaborate with choreographer Jane Turner and her dancers in the creation of an interactive performance for the Human Computer Interaction conference held in at the Cambridge University Computer Laboratory and Microsoft Research. In an afternoon, a fifteen-minute dance-music performance was devised involving dance, physical interaction, algorithmic and human-controlled music generation and instrumental music performance. My contribution was the Gaggle ultrasonic interface. [8]

In June 2010, an extended production of Triggered took place at the Symposium on New Materialisms and Digital Culture at Anglia Ruskin University.  This performance involved two new sculptural interfaces: Wired and Gagglina. [8] Before and after each performance the public was encouraged to investigate the devices themselves and how they are used to generate musical material.

The 2011 Triggered production was further developed for performance at the Kings Place in London.  Both Gaggle and Wired were rebuilt using more usable, stable and dramatic forms and two new devices were built – a set of three canvas hangings painted with a variety of symbols and the metapiano, a metallic, Calder-esque mobile. 

In order to allow the dancers movement around the devices without obstruction, they were designed to operate wirelessly, hanging from the lighting rig of the venue.  

4. Hangings with Glyphs

Figure 1 shows the three hangings created by writer/poets and now artists Katy Price and Andrew Nightingale. The hangings are made of unfinished canvas and both ordinary and specially prepared conductive paint, which provides touch sensitive areas on the canvases. The panels include a variety of images referencing widely differing origins: cave paintings, modern computing symbols, antennae, etc.  As music interfaces they investigate links between sound, image and meaning as well as drama and expression in performance.

During both rehearsal and performance, the dancers spent time investigating the panels and indeed, when the configurations changed slightly between rehearsal occasions, complained that this hindered their learning ‘how to play’ the hangings.

In terms or capacitative touch, this was one of the most fragile and unpredictable devices. It was particularly difficult to ensure that touching actually provoked a response (and the reverse, making sure that a response didn’t happen constantly). I suggested a few times that from my position in front of the laptops, I could trigger some events occasionally. I was surprised by the vigour with which both dancers and choreographer rejected this on the grounds that it would be ‘cheating’ and, perhaps more seriously, that it would undermine the attempts of the dancers to learn how their instruments worked.

5. Metapiano

Of all the devices mentioned here the metapiano is the most complex, both in terms of hardware and software. It was conceived as an experimental multifunctioning object that could be played, or operated by a variety of types of performer, including members of the public, in a variety of settings. Reflecting this, there would be a variety of performance modes, including a standard music performance mode, where a musician would perform in a fairly traditional way, using the device as a performance interface. The mode utilised in Triggered is different: the dancers employ the power of movement and gesture as well as the algorithmic nature of the music produced to create a hybrid performance including music, movement, drama as well as, of course, dance. Another intended, but as yet unexplored mode, involves members of the public in gallery-like environments (although they could be anywhere appropriate) interacting with the device individually or in groups, and in doing so creating their own composition or compositions. This has happened quite successfully with the original Gaggle at events such as HCI 2009.

In these situations, what is a piece of music, or a performance, and how does it differ from sculpture, sonic or otherwise?  There are many examples of algorithmic and interactive art pieces that respond to movement, touch and pressure or that simply unfold over time. [6] One of the consequences has been the generation of a certain type of music: generally ambient or highly pattern driven or stylistic.  I was interested in reflecting the mood, style and intention of a traditionally structured piece of western art music.  One that struck me in particular was a composition that I was fortunate enough to encounter at this time, Lune Rouge by Alissa Firsova. [7]

This composition (and its performance by the composer) was about as far from what one might expect from a musically generative computer programme (it wasn’t to my knowledge!); this is what drew me to it.   Not only was it highly expressive, it was gentle, colourful, melismatic and above all structurally satisfying.  It was rich and original in its use of harmony and colour.

The metapiano, as with all devices mentioned, operates through a variety of sensors, the data from which are captured and transmitted by Arduino boards of varying configurations. The data are transmitted via Bluetooth modules to the SuperCollider audio environment, where they trigger or modulate a variety of high, medium or low-level algorithms.

One of the key sensors works through capacitative touch and is based around a 555 timer chip circuit. In addition, a force resistant sensor (FSR) was included on each of four of the leaves.  A performer could both touch and squeeze a leaf and by doing this generate data for immediate sonification. When touched, the capacitative sensors send a signal to one of the digital inputs on the Arduino microprocessor board. This in turn transmitted a ‘1’ when touched and a ‘0’ when isolated. The FSRs (and indeed the bend sensors when implemented) output between 0-5 volts; this is sampled and given a value between 0-1023.

The lowest level algorithm is simply a SuperCollider synth, a sampled piano, which plays a single ‘note’ event and allows for control over particular musical parameters.

This simple event provides the basis for all higher-level algorithms.  As an example, one of these generates arpeggios, imitating some of the most characteristic passages of the Firsova.  The function creates gestures made of arpeggios. An arpeggio is made of a number of sub-arpeggios, each containing a number of notes, usually three or four.

Through arguments, the number and value of these parameters can be modulated or randomized, as can a number of others such as tempo and starting note.  Amplitude is controlled algorithmically, or through another sensor such as an FSR or bend sensor.

As a message is transmitted by the Arduino board at a minimum of about 50 times a second, if an arpeggio were to be generated every time SuperCollider received a ‘1’ value from the metapiano, there would be many instances of the same algorithm occurring simultaneously, overloading the synthesis engine and swamping the musical texture. In order to control this, a counter is implemented and incremented with each ‘1’ message. Only when the counter has reached a certain value does the function trigger an arpeggio.  Even then, another counter is kept gauging the number of arpeggios playing at any given time and a further limit is placed on this. 

It is in the creation and manipulation of configurations such as these that much of the detailed composition occurs. The arrangement described above causes a sufficiently prompt response for the performer to feel physically responsible for the sonic event, while ensuring that neither the musical texture nor the computer’s CPU is swamped.

My original intent was that many interlocking algorithms triggered by different physical events should be configured so that they create a satisfying, varied and multi-layered ‘composition’ with a significant ‘improvised’ component when even an unskilled/unrehearsed performer, whether dancer or member of the public, played with the device. Due to the demands of performance and audience and lacking the luxury of many days of experimentation and rehearsal, I decided to supply a safety net in the form of a scheduled series of events – a seven-minute (approximately) algorithmic composition, although this can be lengthened, shortened or restructured easily. With this in place, whatever interactions the dancers instigated, there would be some thread to follow, some overarching structure.

So that the dancers should trigger appropriate events at appropriate times, the composition was structured using what I called ‘scenes.’ These were implemented as top-level functions directing the triggers from the real world towards particular sets of mid-level functions. During the first scene, a touch would be likely to generate a chord or small set of chords, a single arpeggio or small melisma. A subsequent scene would direct the gesture towards the generation of an arpeggio such as the one described above. Other scenes triggered rhythmic chords or microtonal swirls of notes. The use of scenes promises much in terms of a more ‘improvised’ and less scheduled musical structure. By allowing either the change of scene automatically, or through another as yet un-invented device, it should be possible in future to allow for more improvised experiences while maintaining feelings of structural dynamism and unity.

As a composer who was trained in the western art tradition, I find it a constant struggle to relinquish the layers of control so often necessary to the successful implementation of one of those earlier creations, and yet I am very drawn to the enabling freedom of expression allowed by technology. While I was unable to let go of the structures of western concert music sufficiently to risk the improvisation that was my initial aim, the structures created at least in principle provide a workable method to achieve this.

What is left are questions regarding the nature and importance of the interface itself: whether the nature of the devices and creations described above make a significant difference to the way people interact with them; or, indeed, the type of musical events or even structures they create.

A significant hint as to the answers to such questions may lie in the reaction of the choreographer and dancers to some aspects of the interaction with these devices during the small amount of rehearsal time available.  Since there were of the complications involved in the activities described above – not least the batteries running flat or Bluetooth disconnecting (for instance) – I made it clear that I could and was prepared to trigger events from my own laptop, so that ‘something’ would happen. The choreographer made it equally clear that she and the dancers did not want me to trigger anything. This would, in their opinion, be ‘cheating’ and would cloud their ability to understand what was really happening in their interactions with the devices. In other words, even if there was a problem with the interface, it would be better for this to be openly the case. Whether this purity of approach would have been so welcomed had there been serious problems with the performance will, fortunately, have to wait for another production as, in the event, there were no serious technical issues to contend with.

6. Conclusions and future developments

At least as far as the Metapiano is concerned, the Kings Place performance of Triggered represented a significant milestone in the development of this work, but not a zenith.

Suspending the devices proved an excellent idea, and was highly effective dramatically – this has opened up many options for future development. Any future use needs to consider the use of pulley systems to enable the devices to be easily raised and lowered. For venues where hanging objects from the ceiling or lighting rig is not feasible, boom stand systems would be a good idea.

Although the Bluetooth devices used for wireless communication worked well, I would prefer to use open, non-proprietary solutions where possible and so devices, such as the XBee, and systems making use of them, such as ‘Sense/Stage’, should be investigated. [2]

Less prosaically, the hanging panels revealed significant potential for the use of semantic links between image, paint and sound in the design of the interaction.

Of greatest significance is the development of these technologies for therapeutic uses in the ‘Touching Sound’ project, an international collaboration between musicians, therapists, programmers, psychologists, artists and others. In spite of many years of research into human computer interaction, there have consistently been issues with the explicit adoption of technology in therapeutic environments, not least because of the way in which clients and practitioners view such technologies: as unnecessary and intrusive. At the same time, there are many ways in which technology can be used productively, for instance in the spontaneous generation of aesthetically interesting artefacts as well as in the exploitation of kinaesthetic and multisensory elements, such as those described in this paper.

The Touching Sound project seeks to describe, implement and analyse work that investigates unique methods of articulating and implementing expressive gesture, synchronisation and entrainment, primarily through shared physical interaction with objects. This includes the implementation of custom-designed and built hardware and software using methods that are as close to invisible to clients and practitioners as possible while still maintaining the benefits of cutting-edge digital technologies. In particular, activities in music performance, dance and various physical and arts therapies are considered.

All of these activities emphasise one thing: the links between composing, improvising, performing and the nature of the physical interfaces used are of crucial significance in many forms of music and that we are approaching a point where a fuller and more open experimentation with such links is becoming conceivable.

References and Notes: 

  1. Sam Aaron, Alan Blackwell, Richard Hoadley and Tim Regan, “A principal approach to developing new languages for live coding,” Proceedings of NIME (Oslo, 2011).
  2. Marije Baalman et al. “Sense/Stage — low cost, open source wireless sensor and data sharing infrastructure,” Proceedings of the Linux Audio Conference (Utrecht, Netherlands, 2010).
  3. Earle Brown, “Calder Piece,” 1966, (accessed May 10 2011) .
  4. Nicolas Collins, Handmade electronic music (London: Routledge, 2009).
  5. Perry Cook, “Principles for Designing Computer Music Controllers,” Proceedings of NIME (2001).
  6. David Cope, Computer models of musical creativity (Cambridge, MA: The MIT Press, 2005).
  7. Alissa Firsova, Lune Rouge (Meladina Press, 2005).
  8. Richard Hoadley, “Implementation and Development of Sculptural Interfaces for Digital Performance of Music through Embodied Expression,” Proceedings of Electronic Visualisation and the Arts Conference (London: BCS, 2010).
  9. Dan O'Sullivan and Tom Igoe, Physical Computing (Boston: Thomson, 2004).
  10. Alvaro Pascual-Leone, "The brain that plays music and is changed by it," Ann N Y Acad Sci 930 (2001): 315-329.