audiovisual performance

f(x) - an audiovisual performance environment

 

f(x) is an audiovisual performance environment to enable exploration of 3-dimensional continuous cellular automata. The automata provide the basis for sound synthesis and computer graphics parameter mapping. During the performance the behavior of the automata world is affected in real time in an attempt to reveal the complex and organic behavioral patterns in three dimensions and modifying the mapping space in response to them. 

 

Author(s)

 

Introduction.

Digital technology has provided an incredible variety of opportunities for artistic exploration and has fostered a new perspective on human culture and society. It has forced scientific methods and concepts into the working process and aesthetic framework of an artist due to its very nature. The beginnings of the digital computer are inseparably connected to research into the biology of self-replication and the possibility of artificial life. The early work of most remarkably Alan Turing and John von Neumann was essential to the way the modern world operates and the fundamental concepts based on the spectacle of biological evolution and natural selection have been integrated into every piece of digital technology with which we have surrounded ourselves. f(x) is a performance environment created to reveal some of the aspects and principles of digital technology. It is based on a concept that has only been made possible with the advent of computers - cellular automata – and was born out of research into artificial self-reproduction. It is designed as a live audiovisual experiment in which the performer interacts with a world of 3-dimensional spatial functions defined – analogously with the principles of cellular automata – in terms of each other. The performance is seeking to reveal complex patterns of behavior, generated by relatively simple instructions and rules that would uncover some of the elusive characteristics of digital media surrounding us now in almost every situation.

Cellular automata constitute an area of research belonging to the interdisciplinary field of complex systems science. They have been used as computer models mainly in computability theory, mathematics, theoretical biology and physics, but also have found many applications in generative art and computer music. The beginnings of the research in complexity can be considered as coinciding with the advent of the field of biology in the beginning of 19th century. In the middle of the twentieth century two independent lines of research were started that have greatly shaped our understanding of complexity. In molecular biology, it became evident from the research that led to the discovery of the chemical structure of DNA, that every living system is highly organized and this organization is coupled to a complex molecular apparatus, which functions as a stored information code for regeneration. There are two complementary modes of existence embedded in each complex living system: the internal physical-chemical workings of a cell and the informational mode where information is selected, stored, and interpreted by the physical actions. In early computer science research an investigation was launched into the nature of complexity in general in a computational or mathematical sense. John von Neumann initiated the study of self-replicating automata in 1940ies. He recognized the dual functioning of information that self-replication requires. The two modes are present in any cellular automata system as well: the dynamics of an automata system and the rules that evolve it.

 

Cellular automata.

Cellular automata are deterministic dynamical systems, which are discrete in space and time, operate on a uniform lattice and are characterized by simple local interactions. An automata world consists of a number of cells on a grid and evolves a number of discrete time steps according to a set of rules based on the states of neighboring cells. The behavior of every cellular automata system depends on four basic features: (1) the size along each dimension of the grid, (2) the number of distinct states, (3) neighborhood over which cells affect each other, and (4) the initial state of each cell at the outset. The most elementary family of cellular automata rules is defined as a one-dimensional row of cells in which each cell can only be either 0 or 1. The automata world evolves by iterating through each of the individual cells at each time step to determine the next state according to the established rule. The resultant patterns of behavior can be characterized as 'emergent', meaning the local simple interactions can produce unpredictable and complex behavior on the global scale of the automata world. The main attraction of such a system is the demonstration of the dynamics of how complex structures can emerge from simple interactions without premeditated design or intention in analogy to the most basic processes in nature.

 

Automata in f(x).

The automata principles utilized in the f(x) environment are slightly modified from the standard discrete valued approach. It is also possible to contemplate systems where the values are not discrete, but continuous within a specified range. In such systems, the value of each cell can be defined by calculating the average state of the defined neighborhood. In this scenario, it becomes necessary to either multiply the average of the neighborhood of each cell by a fraction greater than 1 – e.g. 3/2 - or add a value typically between 0 and 1 and then only use the fractional part of the result for the system to exhibit complex behavior similar to the elementary automata discussed previously. Otherwise the averaging process will quickly settle the lattice into a uniform state all across. The multiplier or the added value becomes the determining factor in the behavior of the automata on the global level, with smaller values producing an overall smoother transition effect.

The behavior of the continuous-valued world can be further affected by assigning weights to each of the cell's neighbors, making variable the amount of influence each cell has in the neighborhood. In this case, weighted mean value is calculated for each cell when updating the states. Weighted automata rules increase in significance as the automata world is extended into higher dimensions as the geometrical properties of the system become more pronounced. In the f(x) environment, the automata idea is expanded into a three-dimensional world of regions, in which there are significantly more complex geometrical phenomena that emerge in the shape of patterns created. One can imagine a cube-shaped world of three-dimensional space sectioned into smaller cube-shaped regions, each of which has an associated state value and is defined in terms of surrounding space. The shape of the surrounding space depends on the neighborhood definition and is further molded by the weights assigned to each neighboring region. These spatial fluctuations can be viewed as continuous functions permeating a region of imaginary space while being defined in terms of each other. This imaginary three-dimensional world constitutes the foundation of the performance environment.

The environment enables a real-time exploration of the evolving spatial patterns by modifying a number of parameters, including the global addition value and the shape of the neighborhood by selecting different patterns of neighborhood weights and enables the user to manage predefined computer graphics algorithms and audio synthesis definitions in response to the automata behavior. The current automata world consists of 4096 regions or cells – 16 along each of the three dimensions. This choice is rather arbitrary having more to do with limitations on real time computing power than any logical or aesthetic considerations and can be easily altered. Currently, only two cell neighborhood configurations are used as the weighting provides further options for different neighborhood configurations. The three-dimensional equivalent of what is known in cellular automata literature as 'Moore neighborhood' considers 26 nearest neighbors, including those diagonal from the center cell on the lattice. Figure 1 illustrates the Moore neighborhood in 2D, with the neighborhood surrounding the center cell shaded darker, and the equivalent in 3D with the center cell represented as a black cube and the neighboring 26 cells as wireframe cubes.

 

System overview.

The system has been implemented in two separate parts. The automata world and graphics functions are part of a separate custom-written application, while the control center, audio synthesis and activation routines, and user interfaces for external control have all been implemented in the SuperCollider programming environment. The communication between the applications is managed through the Open Sound Control (OSC) protocol. Cell state values are used to generate audiovisual content during the performance. Due to the large size of the world and the high standard message rate (24 cycles per second), only a selection of individual cell values is polled by the SuperCollider application for audio synthesis mappings. This selection can also be specified and modified during the performance and it usually coincides with the selective visual representation of the cells. In addition, a global mean cell state and standard deviation for the entire world as well as mean state values for 8 sub-regions are calculated and sent out through the OSC client. The sub-regions are created by splitting the cube-shaped world in half along each of the 3 axes, thus creating 8 smaller cubes of 83 or 512 cells each.

Another dynamic feature of the automata world is linear interpolation of cell state values, which allows “time-stretching” the automata behavior by delaying the update function by a desired number of cycles. This feature allows slowing down the propagation of fast patterns when the addition value is set relatively high. Drawing functions are implemented as encapsulated patches, each with a specific characteristic. These patches can be easily added and modified during development and constitute a basic unit of the graphics application. Each patch has a number of parameters for transparency, color mapping and other values that can be controlled remotely. Besides the individual patch controls, global transparency, screen background color, zoom along each of the 3 axes, speed and direction of rotation, and relative rotation angle around each of the 3 axes can be controlled in real time. The added functions are expected to contain mapping logic from state values to audio synthesis parameters and can also be used to control certain parameters of the graphics application, creating parameter feedback loops during the performance. Audio synthesis definitions and time-structuring routines can be added dynamically during the performance either from predefined functions or specified in a live-coding just-in-time programming manner.

 

Audiovisual composition.

The audiovisual composition relies on two elemental concepts: audiovisual macro compositional entities called zones and the separation of audio and visual content within these entities both physically and conceptually. The concept of a zone emerged in the process of developing this project as a means to have a meaningful segmentation of considerably different audiovisual material and allows introducing new approaches to audiovisual mapping without restructuring the existing organization. Each of these zones can be activated in any sequence during a performance and, for better or worse, have provided a general form for the entire composition. Each zone is characterized by unique and specific mapping procedures resulting in a distinct audiovisual entity.

The disparate networked structure of the environment keeps the conceptual and operational schism between the auditory and the visual. However, it is intended that neither is subservient to the other medium and both have an independent and equally important role in the composition. The two worlds are explored for their inherent characteristics and there is always a hint of a connection, at times seemingly synchronized and at others not easily identified. The 3D world is represented in computer graphics by mapping selected cell state values to low-level vertex drawing functions. Most commonly the cell state or inverse of the state is mapped to the drawn object’s grayscale color, transparency and size. The variation of these three parameters already yields a significant palette of mapping options, considering that what really makes an impact in terms of the visual content are the intricate patterns that emerge even from the simplest of representations.

Audio synthesis parameter mapping is driven by the messages received from the automata world and handled through 3 different types of OSC responder functions as described in the previous section. A commonly utilized technique is to activate a number of parallel synthesis processes, identifying a certain number of modifiable parameters and mapping the incoming cell state values to appropriate mapping ranges for each parameter. Each synthesis process can have its custom mapping range for every selected parameter or synthesis processes can be grouped and the same mapping ranges used for equivalent parameters. The mapping ranges and curves themselves can also be modified during the performance, which provides even more powerful expressive tools for the performer. An example of the described mapping technique from the ‘zone 1’ section of the environment defines first a prototype synthesis definition for time stretching existing audio buffers with 8 modifiable parameters: overall amplitude, start read position in the buffer, end read position in the buffer, amplitude of the signal sent to auxiliary effects bus, pitch-shifting ratio, sound field rotation angles around the x-, y-, and z-axes (in terms of Ambisonics spatialization).

When the 8 synthesis processes are activated in parallel, each with 8 modifiable parameters, a cell states value responder function is added to the responder function evaluation queue containing instructions to map the incoming cell values to the specified ranges. Each time the SuperCollider language application receives a message marked as ‘cell states’, the function is instructed to clump the 64 incoming values into groups of 8, and then map each of 8 cell state values in a group according to the 8 parameter ranges and update the running synthesis processes with the obtained parameter values. Similar technique is used throughout the different zones, however, this is not the only method of mapping employed in the environment. Synthesis processes of shorter duration are also triggered according to received messages, for instance, and there are ranges of cell state values specified within which a certain process is active and is automatically switched off if a specific cell state goes out of that range. There are an endless number of alternative mapping strategies and as the environment is developed further, hopefully new approaches will keep surfacing.

 

Conclusion.

f(x) is an experimental environment, continually under development, exploring possible strategies for audiovisual laptop performance, in which the generative computer art paradigm is brought into the context of live performance. Every performance on the system has so far produced distinctly differing outcomes, even though the introduction of the concept of 'zones' has channelled the performances into previously explored territories and encouraged the performer to favor certain system behaviors over others. In general, the environment fosters the idea of the composer/performer as an explorer or a discoverer, rather than a spontaneous creator. The design of the system tries to balance the amount of control and gestural character of the human performer against the distinctly non-human behavior of the automata world, simultaneously exploring the conceptual differences between the auditory and visual composition. 

 

References and Notes: 

 

Alo Allik, Tehis: a cellular automata programming environment for computer music composition. (M.A. Thesis, Institute of Sonology, the Royal Conservatoire, The Hague, the Netherlands, 2005).

 

Peter Beyls, “Cellular automata mapping procedures.” Proceedings of the International Computer Music Conference, Miami, Florida. (2004)

 

Open Graphics Library: http://opengl.org

 

Open Sound Control: http://opensoundcontrol.org/

 

Norman Packard and Stephen Wolfram, “Two-dimensional cellular automata.” Journal of Statistical Physics 38, Nos. 5/6, (1985): 901-946.

 

SuperCollider: http://supercollider.sourceforge.net

 

Alan M. Turing, “Computing machinery and intelligence.” Mind 59, (1950): 433-460.

 

John von Neumann, The theory of self-reproducing automata. (University of Illinois Press,1966).

 

Stephen Wolfram, A New Kind Of Science. (Wolfram Media Inc., 2002). 

 

  

The Ephemeral in Audiovisual Realtime Practices: An Analysis into the Possibilities for its Documentation

Realtime audiovisual performance is an art-moment defined as a unique narrative. By establishing a theoretical structure, grounded on a permanent process of becoming, the subject of documentation will come to the surface. The text aims at presenting references and considerations for the study of documentation of the ephemeral project towards the collective construction of practice’s memory.

Author(s)

Situated between the moment and its memory, a document is evidence of an action. Ephemerality is a key feature of contemporary artistic practices happening at the point of convergence of digital technology and mobile communications through interdisciplinary discourses.  The performative moment has a unique narrative that occurs in the present, which is no longer past, neither is future yet. The moment combines all that has preceded and re-arranges elements to constitute possible futures.

The interdisciplinary nature of realtime audiovisual performative practices permits related reflective thinking and theoretical discourse to be diffused and, therefore, associated and located within other disciplines . [1] For this reason, a discourse that is particular to audiovisual performance as a specific field of research is minute. In order to create the necessary theoretical arena to deal with the specificities of the practices in contemporary terms, descriptive documents are necessary to be identified.

In the generic landscape of digital ephemeral practices, the few projects related to documentation usually take empirical approaches, mainly aiming at the recovery and archiving of the past. Current reflection on the possibilities for documentation attempt to deal with the conflict between the temporal, immaterial and interactive essence of work developed within contemporary technological landscape by redefining the fixity that defines a document (with specific features, capable of constituting evidence for preservation). Within realtime audiovisual performance, this same conflict takes specific features as it deals with process, improvisation and identity, and has a unique potential through conversion of the tools of practice into the tools for crafting documentation.

This paper presents a set of references from where to think documentation towards its further empirical application.

Although specific within digital ephemeral practices, realtime audiovisual performance has under its umbrella several expressions differentiated by processes (more or less multidisciplinary, such as VJ/DJing), by context (in closer association with cinema: Live Cinema), by history (establishing a connection with musical composition: Visual Music). Moving away from nuances, differences and divergences, we define realtime audiovisual performance within the combination of two dialogical components: audio and moving image, in a unique, multisensory experience, centered around an audience. We establish a specific interest in the developments made by collectives, with emphasis on process.

We at first established a theoretical structure, through relationships rather than hierarchies, from where to look at realtime audiovisual performance. At its basis, in order for the relationship between process and event to be established, having in mind we are studying a time-based art form, the practice is divided into three moments: creative process, performative moment and community gathering moment.

We propose the performative moment, in itself the art expression, as a stable state within an interconnection of processes that encompass not only its own development but also the performer’s body of work, as well as the practice itself as a whole (in its historical and contemporary dimensions). In this interconnection of processes, the time that directly precedes the occurrence of a performative moment will be its process of becoming, what we called the creative process, in terms of Gilbert Simondon’s process of individuation . [2] Because there is no process of individuation without the individual, there is no creative process without performative moment. This process occurs in a chronological order. The first event of this order is the principle of individuation itself and the second are it in practice, in a process that results in individuation. At the third and final event is located the individual. Simondon´s individuation is “primordial, for it is this process that at once brings the individual into being and determines all the distinguishing characteristics of its development, organization and modalities.” [3]  By looking at the relationship between the elements (technology, methodologies, knowledge, and other) that constitute the creative process, we establish connections with Simondon’s system of  historical-cultural evolutionary complexity between humans and technology where man has the role of organizer and interpreter of the ensemble of open machines as expressed in On the Mode of Existence of  Technical Objects. The audiovisual technical set-up, which allows artists to play live, is an ensemble or a technical object of genetic complexity that establishes a relationship between artist and technology in two ways. Firstly through the capacity to change and make specific, sometimes unique, technical set-ups from a combination of elements. Whether proprietary or customized, mutable technical setups for live performance are developed having in mind a specific performer or performance. Secondly, through the improvisational nature in the relationship between artist and technology, as expressed in a poetical way by Mark Amerika in the description of the persona of the VJ. [4] It is from the point of the performative moment that we propose a look at its process and therefore, to look at the documentation.

To document the process of individuation is to make visible concepts and plans that enable the performative moment with its uniqueness. We propose a look into Process Art to understand ways to deal with documentation of the process as well as to documentation and presentation of Software Art. Possibilities for documents may take the form of technical drawing, scores and registration of dialogues between artists. These will provide ways to describe and understand the performance during its process and its connections with technological developments.

The creative process is what makes each performance unique, even when  the elements that constitute it are the same. Returning to Simondon, the process of individuation does not end with the pre individual but continues within a metastable regime, born and maintained by the individual who carries its inheritance of the pre individual environment. The same way, the performative moment will give way to another process and another performance. Following this line of comparison, the preindividual is a source for other metastable states to occur and other individuations to take place. This results in a complex affection between performer, performance and practice as a whole. The process of individuation is here considered being part of an ontogenic process of a larger entity.

The performative moment is constituted by connections between the artists and the projected image, diffused sound, between artists and between artists and audience. If audio and video outcomes and photographic evidence of the environment, resulting from the performance, constitute the most common form of documentation, other aspects are less likely to be looked at. We propose two other elements to be considered of relevance when documenting the performative moment: experience and fruition. Experience is resultant from the relationship established between the performer and work and dialogue with other performers. Fruition is the relationship established by the audience with the work, with each other and the performers. A look into methodos Fluxus artists recurred to in the documentation of their actions, acts, happening and performances, will provide indications of the impact on the ephemeral work that documented descriptions of the audience may have. As an example from the contemporary audiovisual context, the festival Cimatics in its 2007 edition, had a team dedicated to collecting experiences from both spectators and artists. From the edited footage videos were made available online. These videos provide glimpses into the performances and an insight into firsthand experience.

Performance is the moment from where to look into the future. The third moment is the community gathering moment that occurs in physical spaces. Community is a nucleus of those who actively participate in the creation, in critical and theoretical reflection, and experience and fruition of the audiovisual performance. It is composed by individuals from diverse locations and cultural backgrounds. As expressed by Manuel Castells, gatherings allow the community to strengthen the bonds by connecting individuals that come together in a physical location. As nodes of the community, collaborative projects are points of intersection which envisage the network of connections. Collaboration can be identified as the joint effort of two or more people in a dialogical process, grounded on experimenting with concepts, data and technology. To document the community gathering moment is, therefore, of major relevance to understand the practice as a whole as it encompasses all the previously mentioned moments as well as the new one, which is each community gathering moment.

The concept of what is a document has changed, compelled by the shift from art-object to art-moment. Contemporary documentation, moving away from a formula to preserve and exhibit, has a dynamic meaning, which describes a tool to reinstall an installation, or to re-enact a performance. Performances, as well as ephemeral installations, exist only within the context they are presented into the public. In storage (warehouse or database) they are mere materiality. Defined by its uniqueness, the moment is an artistic, ephemeral manifestation that documentation should not replace. From the thirty three case studies that constitute the research project Inside Installations (www.insideinstallations.org) one is relevant to highlight as example. A series of performances with liquid crystals that took place between 1965–66, by the artist Gustav Metzger, were exhibited at the Tate in 2005. For its original format, a team was necessary to manage material and 12 projectors. For the exhibition, the artist, together with the museum’s technical team, developed an installation version based on the original concept. A remix of the material was presented recurring to recent technologies. The installation exhibited is also documentation of the performance. The replacement of the realtime and the performative by an object creates a situation that is worth analysing. We take a different and more challenging path with a proposal to approach documentation in a way that not only makes possible to preserve the practice’s features but reinforce them.

Which criteria should describe this documentation?

Radical actions, throughout avant-garde movements of the past century, gave way to possibilities of new concepts of making and experiencing art. It is within a conceptual frame that Fluxus can be understood as a community and as a philosophy rather than simply a historical movement [5]. It is through an interest in publication of documents that a concept emerged, but it was through a performance festival that Fluxus started as a network. The emphasis on the playful and ephemeral is visible in published objects related to Fluxus (Flux Boxes), but the opposite can be said about the performances, if we refer to their printed scores (Fluxus Cookbook). This affection of ephemeral on object and vice-versa is also true for individual artists. Allan Kaprow’s writings referenced directly to his work. In fact, they constitute the part that is left of most of his work. Kaprow wrote scores, collected written recollections of himself and of members of the audience of his performances and published about his work under a pseudonym [6]. The intermedia, performative and participatory character of Fluxus took many shapes, for example in events, publications and films. It is the relationship between ephemeral and document in the work of the community as well as of the individuals, that is relevant. Fluxus is a great inspiration for looking at the possible shapes documentation can take in relationship to realtime audiovisual performance from the point of view of the practitioner, from the collective and from the community.

Without defining documenting as a set of rules, but drawing a trajectory for the possibilities of documenting as complementary to practice, an example can be presented. Being interested in the creative process, the collective Aether9 (http://aether9.org) explores the possibilities of realtime manipulation and transmission of audio and video. Geographically located in different points of the globe, the collective’s members maintain communication and perform exclusively via the Internet. Documentation of their Skype meetings that happen during preparation and during the performance took the shape of (so far) two books published by Greyscale Press (http://greyscalepress.com). These books can be purchased through online print-on-demand Lulu (http://www.lulu.com). This way, the readers can be located in any place of the globe with Internet and postal access. Through this process of documentation, Aether9 provides non-descriptive layers of the performances which help understand them beyond their results, based on the interaction between artists during the process of development and final presentation.

This example contrast with another one provided by the DVD Immersive Works by Granular Synthesis. The DVD presents recordings of performances between 1991 and 2001. A distinctive feature of this work, when compared with general attempts to document audiovisual events, is its descriptive nature (as opposition to the promotional short version very common especially in social networks). It is for this feature that we consider as an example of a document that provide evidence of the group’s work. While Aether9’s books detach the reader from the actual outcomes and presents creative process and performative moment as an ongoing process, in the DVD by Granular Synthesis the focus is on the audiovisual experience of the performative moment. Both examples use individual, objectified documents, and none of them can be perceived as replacement of the performative moment. 

To establish a theoretic structure, process oriented, from where to construct instruments to analyse and reflect on the contemporary audiovisual performative practice, is a proposal that comprehend both its ephameral and multidisciplinary nature. Parallel to the institutionalized procedures, documentation of the contemporary practice is a subject of the community’s concern. From within, documentation can be considered as registration of a series of relationships and interconnected processes that constitute means to retain and re-experience (individually and collective) the art-moment. This registration can be developed recurring to tools, technologies and knowledge from within the practice. The documents, developed by the community, are a primary source for further study, research and memory construction. In sum, to document is also to contribute actively to the construction of identity and context. In the future, this focus on the community and the possibilities of its actions will likely provoke changes not only in the practice itself but also in the way it is perceived externally.

References and Notes: 

  1. Dieter Daniels and Sandra Naumann, See This Sound. Audiovisuology Compendium (Vienna: Ludwig Boltzmann Institute, 2010), 15.
  2. Gilbert Simondon, “The Genesis of the Individual”, in Incorporations, ed. Jonathan Crary and Sanford Kwinter, 300 (New York: Zone).
  3. Ibidem, 300.
  4. Mark Amerika, Meta/Data: A Digital Poetics (Cambridge, MA: The MIT Press, 2007).
  5. Owen Smith, “ Fluxus Praxis: An Exploration of Connections, Creativity, and Community,” in At a Distance: Percursors to Art and Activism on the Internet, ed. Annmarie Chandler and Norie Neumark (Cambridge, MA: The MIT Press, 2006).
  6. Alex Potts, “Writing the Happening: The Aesthetics of Noart” in Allan Kaprow – Art as Life, ed. Stephanie Rosenthal et al, 20 – 31 (London: Thames and Hudson, 2008). 
  

s0: an audiovisual performance

The author is developing detailed sonic models whose possibilities are explored in an audiovisual live setup. Drawn from literatures of nonlinear dynamical systems, the work seeks to address idiomatic approaches to digital media. Besides the sonic reproduction, the generated digital signals are transcoded into binary patterns and are visualized using a custom-made program.
Dates: 
Thursday, 15 September, 2011 - 17:30 - 18:00
s0 version installation
s0 version installation
s0 version installation
s1 version performance
s0 version performance
Author(s): 
yota morimoto

The author is developing detailed sonic models whose possibilities are explored in an audiovisual live setup. Drawn from literatures of nonlinear dynamical systems, the work seeks to address idiomatic approaches to digital media. Besides the sonic reproduction, the generated digital signals are transcoded into binary patterns and are visualized using a custom-made program. The project fetishizes the digital media. It opens up a possibility of a digital aesthetics by using custom-built sonic and visual models which unveil the underlying binary dynamics in electronic media.

Syndicate content