AVOL - Towards an Integrated Audiovisual Expression

AVOL is an interactive audiovisual project for the Web by Video Jack (Nuno N. Correia and André Carrilho). Its main objective is to allow for an integrated musical and visual expression, in a way that is playful to use and engaging to experience. In this paper, the project is presented and contextualized with related works. Its design is then discussed. Finally, results of the project's evaluation are outlined, allowing for conclusions to be reached.


1. Introduction

AVOL (AudioVisual OnLine) is an interactive audiovisual project for the Web (http://www.videojackstudios.com/avol) by the author and André Carrilho, under the name Video Jack. The programming and music were developed by the author, while André Carrilho was responsible for the graphic design and animation. AVOL was released in December 2007, and further developed until 2010. It was one of the four winners of a call for projects by the Arts Department (DGA) of the Portuguese Ministry of Culture for their newly created net art portal. It follows up on previous audiovisual work by Video Jack such as Heat Seeker and Idiot Prince (both 2006). The project aims to allow for an integrated musical and visual expression in a way that is playful to use and engaging to experience.

AVOL was presented as an installation at several new media festivals in 2008: Cartes Flux (Espoo), Re-New (Copenhagen), Create (London) and Live Herring (Jyväskylä). In the same year, it was also presented as performance at Abertura Festival (Lisbon) and at Electro-Mechanica (St. Petersburg). In 2010, due to the redesign of Video Jack's website (http://www.videojackstudios.com), and in order to better showcase the possibilities of AVOL, new videos and music tracks based on the project were released online. At that time, three of the music loops in the project were updated.

AVOL can be contextualized with a wide range of works that have attempted to create integrated audiovisual experiences. The pursuit of correlations between music and visuals has a long history, from Ancient Greek philosophers such as Aristotle and Plato to renaissance artists, notably Arcimboldo. [1] The development of cinema allowed for further approaches in this field. Innovators such as Walther Ruttmann and Oskar Fischinger explored combinations of music with abstract animation. [2] A new generation of artists, notably John Whitney, pioneered the use of computers to create visual music films. More recently, artists such as Golan Levin and Toshio Iwai have taken advantage of developments in computer technology and human-computer interaction (HCI) to create playful interactive audiovisual experiences. Both Levin's Audiovisual Environment Suite (AVES), "an interactive software that allows for the creation and manipulation of simultaneous visuals and sound in real time”; [3] and Iwai's Electroplankton, a musical 'toy' for Nintendo DS consisting of an aquatic universe "filled with different species of plankton that can produce sound and light when you interact with them,” [4] were important influences for AVOL.

2. Project Design

In AVOL, the visuals and music, together with the graphical user interface (GUI) that controls them, are grouped in seven 'objects' entitled IAVOs (Interactive AudioVisual Objects). The visuals consist of abstract concentric vector animations that are audio-reactive. The reactivity to each sound is based on the scaling of the respective animation proportionally to the sound amplitude.

IAVOs contain four audiovisual options – four pairings of sound loops and animations. Therefore, there are a total of 28 interchangeable sound loops and animations in the project, allowing for numerous combinations. Each of the four content options has a differentiated character (in terms of color, shape, animation, and sound palette). In addition to buttons for triggering the four content options, each IAVO contains in its GUI buttons for: stop; solo; automatic motion; sound volume (which also affects the scale of the animation); and a draggable area.

Each IAVO has its own sonic nature (for example, 'bass drum' or 'guitar'). All sound loops have equal duration and tempo (16 seconds and 120 beats per minute respectively) and are synchronized. They were composed with the purpose of being coherent and harmonious independently of the active combination of sounds. The collision of two IAVOs triggers a custom sound.

2.1 Animation and visual design

André Carrilho tried to differentiate the four content options of each IAVO with different types of shapes, using modular compositions based on one or two simple elements. After finding the basic graphic elements, he would animate them with the movement suggested to him by their shapes. The shapes also inspired the choice of color. For example, the color palette of the second animation set (triggered by the second button in each IAVO) is white, red and orange, while the one in the third animation set is blue, blue-grey and white. There was an interpretative differentiation between animations according to the nature of the corresponding sound, mainly between rhythmical sounds (the first four, counting from the left) and melodic ones (the last three). For example, in the second animation set, the four 'rhythmical' animations are based on simple circles, while the three 'melodic' ones are based on a circular wave pattern. Despite these nuances, the animations were designed to be harmoniously interchangeable. Each animation contains two elements, an audio-reactive component and a non-reactive one, in order to convey current volume level and ensure that the animation is visible even when the sound amplitude is low.

The use of vector graphics is an important element in AVOL. It ensures that the animations are scalable and react fluidly to sound. It also guarantees a faster loading time of the website. The animations had to be as 'light' as possible, and not too complex in terms of shape or motion, due to computer performance restraints. But Video Jack believe that technical constraints may help to focus the creative process and lead to a coherent whole.

The animations, although abstract, suggest natural shapes of different scales with a concentric nature, such as atomic, cellular, floral or planetary imagery. The visuals in AVOL resemble John Whitney’s concentric animations. Quoting Gene Youngblood's description of one of his animations (which could apply to AVOL): “all colors move into the ring simultaneously from all sides, forming circles within circles all scintillating smoothly in a floral configuration.” [5] There are also similarities between the 'objects' in AVOL and the organisms in Electroplankton, even more apparent when collisions occur.

2.2 Interaction Design

The interaction design of AVOL aims to achieve an intuitive experience and to foster exploration by its users. As an example, the 'stop', 'auto' and 'solo' buttons within the IAVOs follow a traffic light metaphor. As Jakob Nielsen states, metaphors "can facilitate learning by allowing users to draw upon knowledge they already have about the reference system." [6] The visual design of the ring, with its rough edges, is meant to convey a 'click-and-drag' affordance. According to Donald Norman, affordances refer to “the perceived and actual properties of the thing", in particular those "that determine just how the thing could possibly be used.” [7]

An important element of AVOL is the aesthetic integration of the GUI with the animations. The design of the GUI with its petal aesthetics reinforces the floral visual undertone.

3. Evaluation

As designer and user of AVOL, the author considers that the project achieved its objectives in terms of playfulness and engagement, due both to the IAVO approach of integrating audio-reactive animations with the respective GUI elements, and to the harmony, coherence and mutual agreement of the audiovisual content. However, the author detected several shortcomings in AVOL that constrain its potential for creativity and expression. After the release of AVOL, Video Jack started designing a new project that would address these issues. In 2010 the new project, entitled AV Clash (http://www.avclash.com) was released. Following this release, the author planned a questionnaire to users of Video Jack projects, including a section comparing AV Clash and AVOL. This section would allow him to evaluate if his initial conclusions regarding AVOL would be confirmed by users.

3.1 AV Clash – follow-up project and benchmark

AV Clash intends to solve some of the insufficiencies detected by the author in AVOL, mainly its scarce audio manipulation options and its limited amount of sounds and visuals. In order to access a larger amount of sounds than AVOL, AV Clash connects to Freesound.org, an online sound database. Approximately 240 sounds from Freesound.org are used in AV Clash. The project also contains a larger number of animations (96) than AVOL. Audio manipulation capabilities include audio effects ('echo' and 'filter') and sound trimming. [8]

The sound character of AV Clash is substantially different than AVOL: the sounds are not synchronized; they have different durations; and they have a much more diverse nature, ranging from field recordings of nature and voice recordings to synthesizer sounds. Some usability improvements where introduced, such as color-coding of IAVOs and usage of 'info tips'. Nevertheless, there are several common elements between AV Clash and AVOL: a similar visual style (abstract concentric vector animations); a similar IAVO approach of integrating sound visualization with GUI (although the number of IAVOs was reduced to four due to performance issues); and the same audio reactivity behavior, based on animation scaling.

3.2 Questionnaire and results

In the first half of 2011, an online questionnaire was setup to evaluate AV Clash. The questionnaire contained closed and open-ended questions, and was answered by 22 anonymous respondents. It included questions comparing AV Clash with AVOL. From the answers to these questions, important insights can be reached regarding AVOL.

AV Clash seems to have been more successful than AVOL in terms of engagement. Approximately three quarters of the test users (16) have spent more time interacting with AV Clash than with AVOL. Half of the users (11) attributed this to the additional manipulation options. Approximately two thirds (68%) of the respondents consider the possibility of accessing a larger amount of content in AV Clash to be appealing, against 27% who do not. One of the users is indifferent. Half of the test users consider the additional audio manipulation options in AV Clash to be interesting, against 18% who do not. Nearly one quarter of the respondents are indifferent. These results show that users value the additional content and functionalities of AV Clash.

When asked which project had a more enjoyable sonic and musical approach, 41% of the test users chose AV Clash, against 32% who prefer AVOL, with 27% enjoying equally both. The test users who preferred the sounds in AVOL mentioned the synchronization of the loops, the curation of sounds, and the inclusion of percussive elements as factors for their choice. Regarding ease of use, 36% of respondents consider AV Clash to be more intuitive, whereas an equal percentage of users reply that AVOL is easier to use. 27% consider that they are on the same level regarding intuitiveness. When asked the reasons behind their choices, six out of the eight users who considered AVOL more intuitive mention the simpler interface and fewer options as the reasons for their choice. 

The majority (59%) of the respondents considered that AV Clash gives a higher feeling of creativity than AVOL, whereas only two users chose the latter. Nearly one quarter of the test users answer that both projects are on the same level regarding this issue, and two users do not get a feeling of creativity from either. From the 13 respondents who consider that AV Clash gives a greater feeling of creativity, seven mention more manipulation options, and another one more variety in sound, as reasons for their preference. One of the test users who consider AVOL to offer a more creative experience mentions that the sounds "fit together nicely", adding that switching between them created interesting results. From the respondents who were indifferent to or dissatisfied with the level of creativity allowed by these projects, one user considers that both projects shape the sound and visuals too much to allow for his/her own creativity, and another mentions that the projects are "too structured". 

4. Conclusions

Despite the low number of respondents to the questionnaire (22), some useful insights can be reached regarding AVOL and interactive audiovisual projects in general. The results from the sections of the questionnaire comparing AVOL and AV Clash confirm the insufficiencies detected by the author in the former: the limited content and the few manipulation functionalities result in less creative options and a more restricted expressiveness. The majority of users consider that AV Clash gives a higher feeling of creativity. It was successful in achieving a greater engagement than its predecessor because of its additional content and media manipulation capabilities.

However, extra manipulation options and added content come at a cost in terms of usability: AVOL achieves similar results to AV Clash in terms of ease of use, despite the attempted improvements in interaction design. Moreover, AVOL comes close to AV Clash in terms of sonic enjoyment. Although more users (41%) manifest preference for the audio in AV Clash, a significant amount of users (32%) consider the synchronicity, coherence and harmony of sounds in AVOL to be more pleasant. Therefore, even though the majority of respondents favor AV Clash, there seems to be a significant number of users who prefer a simpler project such as AVOL, with fewer options and content, but with curated and more harmonious sounds.

Some of the users are dissatisfied with the creative and expressive potential of both projects, considering that they are "too structured" and shape the end result too much. Further developments in this line of projects could address these limitations, expanding the manipulation options, content, and customization possibilities. This path would concentrate on expanding the creative potential of interactive audiovisual projects as tools. Another possible path for AVOL would be to concentrate more on playfulness and intuitiveness, targeting users who prefer a simpler and more curated approach. 

In future developments, the author considers that emphasis should be given to recording and sharing capabilities. Net art audiovisual projects such as AVOL should take advantage of their online presence to facilitate the recording and distribution of user-generated content, namely by leveraging social media. The author also believes that multi-touch mobile devices are attractive for future interactive audiovisual projects, since they allow for a more direct and flexible manipulation of visuals than a pointing device interface.

In the author's opinion, there is a large potential for creativity and engagement in AVOL's path towards an integrated audiovisual expression, where “composer, performer and audience converge in the playing subject.” [9] The results achieved so far show that this path might develop into separate branches, exploring different balances between expressiveness and playfulness.

References and Notes: 
  1. Cretien van Campen, The Hidden Sense: Synesthesia in Art and Science (Cambridge: The MIT Press, 2007), 45-46.
  2. William Moritz, “The Dream of Color Music, And Machines That Made it Possible,” Animation World Magazine, April (1997), http://www.awn.com/mag/issue2.1/articles/moritz2.1.html (accessed August 22, 2011).
  3. Christiane Paul, Digital Art (London: Thames & Hudson, 2003), 133.
  4. Ryan Davis, “Electroplankton Review,” Gamespot, 2006, http://www.gamespot.com/ds/puzzle/electroplankton/ (accessed August 22, 2011).
  5. Gene Youngblood, Expanded Cinema (New York: P. Dutton & Co., 1970), 220, http://www.vasulka.org/Kitchen/PDF_ExpandedCinema/ExpandedCinema.html (accessed August 22, 2011).
  6. Jakob Nielsen, Designing Web Usability (Indianapolis: New Riders, 2000), 180.
  7. Donald Norman, The Design of Everyday Things (New York: Basic Books, 2002), 9.
  8. Nuno N. Correia, “AV Clash – Online Tool for Mixing and Visualizing Audio Retrieved from Freesound.org Database,” in Proceedings of Sound and Music Computing Conference 2010 (Barcelona, 2010), 220-226, http://smcnetwork.org/node/1365 (accessed August 22, 2011).
  9. Alex Stockburger, “An Audience of One – Sound Games as a Specific Form of Visual Music,” in Audio.Visual, ed. Cornelia Lund and Holger Lund, 122 (Stuttgart: Arnoldsche, 2009).