Animated Graphic Notation

This paper discusses the authors’ use of animated graphic notation to encourage collaborative music making for a wide range of performers with different musical backgrounds. In terms of notational reform this essay will ask two important questions with reference to the author’s work: What are the problems with the old notation, both traditional and static graphic notation, and what is useful about new forms of animated notation. 

Author(s)

Origins and Motivation

As a music maker and music teacher the main aim of my work has been to create genuinely accessible musical experiences, which invite participation based not on musical experience or training but on a simple willingness to take part. This approach arises from the view that music is a fundamental human activity, which forms a part of all known human cultures past and present [1] and which not only satisfies our creative urges but plays an important role in our personal and social development just as it has in our evolution and survival [2] as a way of promoting cohesion through collaborative experience. In western art music the development of notation lead to a gulf between those who take part in music - performers and composers - and those who simply listen - their audience. This has been reinforced by a music education system which concentrates largely on music and notation of the past without giving due attention to the development of musical ideas in contemporary music and to notational reform. The use of graphic notation in education has been shown to encourage creative thinking, collaboration and ensemble performance, while also dealing with general musical concepts, which apply across many genres and styles of music. [3] From my own experience of using graphic notation in the classroom, and as a compositional device I began to develop a system of animated notation, which would further serve the idea of creative collaboration and accessible music making.

Notation – The Western Tradition

Conventional music notation represents a system of communication between composer and perfomer which has been the basis of musical creation in the western world for hundreds of years. The writings of Guido of Arezzo from the eleventh century reveal that notation was used to record and preserve music and to provide performers with a memory aid to assist performance. [4] In Micrologus (ca. 1025-28), which drew on the theoretical studies of the ninth century, Musica enchiriadis and Scolica enchiriadis, we also see the development of a fixed system of notation giving rise to the study of music as a theoretical language. With the refinement of this notation came the opportunity for composers to create polyphonic works for a number of musicians reading independent parts, replacing the single line melodies sung in unison, which had previously dominated. These developments allowed for the thoughtful organisation of music, which calculated the effect in advance of the performance and provided an exact timetable for the coordination of different parts. An in-depth knowledge of notation, and the musical theory needed to decipher it, became a pre-requisite for becoming a musician and certainly for becoming a composer. A hierarchical structure developed in western art music with the composer instructing the performer for the benefit of the listener, creating a situation where all of the musically uneducated can listen, few can perform and even fewer compose. If we consider listening to be the most vital and beneficial part of the musical experience then this system is justified. If we consider participation in music making to be the most vital aspect, then this system must be questioned.

Animated Graphic Notation

The evolution of graphic notation in the 1950s was, in part, due to the perceived failure of conventional notation to adequately represent the growing range of sounds and performance techniques being explored by some of the more innovative composers. In the 1920s work like Hyperism and Ionisation, by Edgar Varèse, liberated composition from melody, harmony, rhythm and regular pulse, ideas easily represented using conventional notation. The growing use of unpitched, non-western instruments and electronic sounds exposed the limitations of notation to accurately represent these imaginary sound worlds to be produced in live performance. Graphic notation allowed composers to create musical scores which would represent their ideas but which needed the collaboration of the musicians to produce the desired sounds. This implies a new composer/performer relationship, where the composer relinquishes some control over the resulting sound, alternatively trusting the musical instincts of the performer. The focus of the score now shifts from being an accurate timeline of sound events, which the musicians must produce, to a communication of a musical idea to be interpreted.

One of the great advantages of conventional notation is the control it offers the composer over durations within a piece. With the help of trained performers, composers can create rhythmically complex works with large ensembles, where every detail of the musical timeline is precisely arranged in advance. This is only achievable after establishing conventions of rhythmic notation as standard amongst composers and performers. With static graphic notation, the lack of objective rhythmic conventions often leads to both indefinite rhythm and unknown duration of many graphical pieces. Animate graphic notation allows for intuitive and clear cues for durations and for dividing sections of an ensemble using visual parameters, like colour and position on screen and although it is difficult to emulate the precise rhythmic nature of conventional notation, visual rhythms can be easily used to indicate pulse leaving the precise rhythmic interpretation up to the performer. Much of my work, which will be discussed later, chooses to leave the final decisions of rhythm, pitch and timbre to the performers with the notation designed to supply enough information to communicate a musical idea without specifying a musical sound. This type of animated notation allows each score to be accessible to a wide range of performers, with different levels of musical experience, and to be open to different interpretations from ensembles of all sizes.

Research, Performances and Installations

In April 2008, two experimental electronic artists, one cellist with a looped effects unit, a classical trombonist and clarinet player, a number of percussionists and around forty audience participants as vocalists, took part in my first experiment with animated graphic notation for a large ensemble in The Bernard Shaw venue in Dublin. None of the performers had seen the score before the performance so the ensemble was divided into three sections: the vocalists followed the white parts (mainly letters like ‘s’ and ‘t’, for texture and percussive sounds); the acoustic instrumentalists followed the blue yellow and orange sections: and the electronic performers interpreted the background video elements (Fig. 1). The notation for The Score in the Bernard Shaw resulted from a number of months of experimentation with simple animated shapes and video footage mostly concerned with changing scale and visual rhythms and the use of colour to divide up sections of an ensemble. The performance worked extremely well as an example of collaborative music making with an ensemble of performers who hadn’t rehearsed, many of whom were not familiar with graphic notation. The diverse range of performers working closely together as part of a spontaneous musical experience was the most encouraging part of this early experiment and much of the notation used here was adapted in the next stage of research, involving a more scientific approach.         

The next step was to create an installation piece which invited people to give a vocal interpretation of a short animated graphic score, each in isolation and without hearing the other participants’ interpretation. A mixture of participants with different levels of musical experience were chosen; from professional classical musicians to participants with no musical training. Their interpretations were recorded, analysed and used to create a layered backing track for a live multimedia performance in 2008 entitled Three Streams, a piece that used the same animated notation interpreted by trombone, cello and percussion. The score for the installation was divided into a series of animated shapes and symbols, which repeated three times. The first two were a line of ‘s’ and ‘t’ shapes moving across the screen used as an example of a socially learned symbols which are easily interpreted as vocal sounds. The symbols and shapes gradually became more abstract and with more complex movement in order to explore the instant oral response to the visual parameters of changing scale, movement, texture and position (in relation to screen position and to other symbols). These visual parameters were expected to influence the dynamics, duration, timbre and pitch of each interpretation.

Great lengths were taken to animate each part of the notation in a way which could be instantly interpreted by using changes in opacity (fading in and out) and screen position to mimic the forward reading that takes place when a musician sight-reads conventional notation. When sight-reading the brain does not simultaneously absorb individual pieces of visual information, process it and control the motor functions that make the sound. Instead, the reader takes in large amounts of information within the visual field in what are called fixations. [5] Each fixation involves looking forward but also looking backwards (regression or backwards fixation) which is represented by fade in and fade out in this system of this animated graphic notation. The analysis of the recordings, which included forty individuals, showed similarities not only in the interpretation of socially learned symbols like letters but also in the response to abstract shapes. Similarities in participant’s pitch, duration and even timbre suggest that certain visual parameters are objectively linked to sounds and that a common musical language exits between the participants, which may be socially learned but is not a product of musical training. A notable feature of these vocal interpretations was the lack of discrete pitches used throughout, with a complex combination of sliding pitches more common, making it difficult to analyse the exact pitches used (Fig. 2). The performance, which followed the installation, was a combination of the layered installation recordings with live interpretation of an extended score. This was projected for the musicians and the audience, many of whom had participated in the installation, providing an intimate insight into the performance and composition of the piece. 

Pulsing Shapes Colours

My 2009 piece Pulsing Shapes Colours was created to explore the use of animated graphic notation to create a spontaneous performance for orchestra introducing each section separately, with each following a different colour, but with a combined sense of pulse drawn from visual rhythms. No member of the orchestra had previously seen the score so the notation was designed to be easily sight-read using a small number of repetitive pulsing shapes, which fade in and out over eight minutes. Each one of these shapes contains both a simple pulsing rhythm and complex inner rhythms, which may be interpreted in different ways (Fig. 3). This was designed to keep the individual performers and each section linked into a common pulse while allowing them the freedom to explore different rhythms, from the visual rhythms on screen or from listening and mimicking each other. Each section of the orchestra was given a different part of the score to follow: The piece begins with brass and upper strings, followed by lower strings responding to any red parts of the score and woodwind following the red/orange section. The percussion were given free reign to pick up on any interesting rhythmic parts they saw on screen or heard from the rest of the orchestra. No performance instructions concerning pitch, performance techniques, rhythm,  timbre or dynamics were given. The members of the orchestra, who had been chosen through audition as the most highly accomplished musicians of their age in Ireland, seemed to relish the chance to perform freely without the level of instruction and direction they were normally accustomed to. Yet as soon as the lights went down and the animation began – which was projected onto a huge 20ft screen – they became immersed in a new musical experience where they were given the freedom and the combined responsibility for shaping the sound of the piece. The orchestra continued playing for ten minutes after the piece had finished, taking advantage of the opportunity to explore orchestral sounds in a spontaneous and improvised manner.

Three For Four

Three for Four was created for the Irish Composer Collective and performed in Sonic Arts Research Centre, Belfast and the National Concert Hall, Dublin. The piece uses just three simple shapes: lines, circles and triangles, in four colours moving in different ways. This piece was originally created for string quartet and was performed as part of a concert with a traditional audience/performer setting, with the exception that the performers were positioned side-on to the projection on stage so they and the audience could see the score clearly. Although in this type of concert the audience is not directly involved in the music in terms of performance, they are given an insight into the piece through the projected score. As was suggested in the installation piece for Three Streams a subjective connection between certain moving shapes and symbols exist so that the audience can understand the interpretations of the performers. Three for Four was also used to explore the versatility of animated graphic scores by recording a number of different interpretations from different groups including New Dublin Voices (a thirty-piece choir) and jazz pianist Johnny Taylor. Each section of the choir – soprano, alto, tenor and bass – followed a different color while Johnny Taylor recorded a separate layer for each colour, working within a loose harmonic progression for different sections and assigning certain themes and motifs to shapes and colours.  Each of the three interpretations were used in an interactive installation where the score was repeated on a loop and the passing audience members could use three faders to mix the recordings of the individual interpretations together. Although none of the groups heard the others interpretation and there was no discernable connections in choice of pitch in terms of traditional harmony, the subjective connections between moving shapes and sounds create a relative movement of pitch, dynamics and duration which compliment each other in the same way the orchestra found a common sense of movement. The choir recording also had a strong connection to the recordings taken as part of the Three Streams installation. Although New Dublin Voices are a highly trained and experienced group of singers they rarely rested on discrete notes with their undulating pitch and rhythm showing many similarities to the individual Three Streams recordings.

Conclusions

Over four years of performances and recordings have shown that animated graphic notation is an effective way of creating a shared musical experience through creative collaboration. Over 200 performers have taken part in these pieces ranging from experienced noise artists, jazz and classical musicians to amateur musicians and participants with little on no musical training. The connections between all of these performances indicate an understanding and ability to create musical sounds, which use dynamics, pitch, durations and rhythms, that may be socially learned but appear not to be based on musical training. It is the role of music education to encourage and nurture this shared musicality for the benefit of the individual, the group and of musical culture in general. For composers and musicians, new notation presents opportunities for working in more collaborative, accessible and versatile ways, which encourage participation in musical activities beyond the listening experience.  The advantages of using animated graphic notation to add structure, encourage creativity and provide a focus to ensemble performance makes it an important element of current notational reform.

References and Notes: 

  1. H. Cole, Sounds and Signs: Aspects of Musical Notation (London: Oxford University Press, 1974).
  2. H. Read, Icon and Idea (London:Faber and Faber, 1955), 32.
  3. T. Regelski, A Sound Approach to Sound Compositions, in Music Educators Journal 72, no. 9 (1986).
  4. D. J. Grout & C. V. Palisca, A History of Western Music (New York: W. W. Norton & Co Inc., 1996), 54.