f(x) - an audiovisual performance environment

 

f(x) is an audiovisual performance environment to enable exploration of 3-dimensional continuous cellular automata. The automata provide the basis for sound synthesis and computer graphics parameter mapping. During the performance the behavior of the automata world is affected in real time in an attempt to reveal the complex and organic behavioral patterns in three dimensions and modifying the mapping space in response to them. 

 

Author(s)

 

Introduction.

Digital technology has provided an incredible variety of opportunities for artistic exploration and has fostered a new perspective on human culture and society. It has forced scientific methods and concepts into the working process and aesthetic framework of an artist due to its very nature. The beginnings of the digital computer are inseparably connected to research into the biology of self-replication and the possibility of artificial life. The early work of most remarkably Alan Turing and John von Neumann was essential to the way the modern world operates and the fundamental concepts based on the spectacle of biological evolution and natural selection have been integrated into every piece of digital technology with which we have surrounded ourselves. f(x) is a performance environment created to reveal some of the aspects and principles of digital technology. It is based on a concept that has only been made possible with the advent of computers - cellular automata – and was born out of research into artificial self-reproduction. It is designed as a live audiovisual experiment in which the performer interacts with a world of 3-dimensional spatial functions defined – analogously with the principles of cellular automata – in terms of each other. The performance is seeking to reveal complex patterns of behavior, generated by relatively simple instructions and rules that would uncover some of the elusive characteristics of digital media surrounding us now in almost every situation.

Cellular automata constitute an area of research belonging to the interdisciplinary field of complex systems science. They have been used as computer models mainly in computability theory, mathematics, theoretical biology and physics, but also have found many applications in generative art and computer music. The beginnings of the research in complexity can be considered as coinciding with the advent of the field of biology in the beginning of 19th century. In the middle of the twentieth century two independent lines of research were started that have greatly shaped our understanding of complexity. In molecular biology, it became evident from the research that led to the discovery of the chemical structure of DNA, that every living system is highly organized and this organization is coupled to a complex molecular apparatus, which functions as a stored information code for regeneration. There are two complementary modes of existence embedded in each complex living system: the internal physical-chemical workings of a cell and the informational mode where information is selected, stored, and interpreted by the physical actions. In early computer science research an investigation was launched into the nature of complexity in general in a computational or mathematical sense. John von Neumann initiated the study of self-replicating automata in 1940ies. He recognized the dual functioning of information that self-replication requires. The two modes are present in any cellular automata system as well: the dynamics of an automata system and the rules that evolve it.

 

Cellular automata.

Cellular automata are deterministic dynamical systems, which are discrete in space and time, operate on a uniform lattice and are characterized by simple local interactions. An automata world consists of a number of cells on a grid and evolves a number of discrete time steps according to a set of rules based on the states of neighboring cells. The behavior of every cellular automata system depends on four basic features: (1) the size along each dimension of the grid, (2) the number of distinct states, (3) neighborhood over which cells affect each other, and (4) the initial state of each cell at the outset. The most elementary family of cellular automata rules is defined as a one-dimensional row of cells in which each cell can only be either 0 or 1. The automata world evolves by iterating through each of the individual cells at each time step to determine the next state according to the established rule. The resultant patterns of behavior can be characterized as 'emergent', meaning the local simple interactions can produce unpredictable and complex behavior on the global scale of the automata world. The main attraction of such a system is the demonstration of the dynamics of how complex structures can emerge from simple interactions without premeditated design or intention in analogy to the most basic processes in nature.

 

Automata in f(x).

The automata principles utilized in the f(x) environment are slightly modified from the standard discrete valued approach. It is also possible to contemplate systems where the values are not discrete, but continuous within a specified range. In such systems, the value of each cell can be defined by calculating the average state of the defined neighborhood. In this scenario, it becomes necessary to either multiply the average of the neighborhood of each cell by a fraction greater than 1 – e.g. 3/2 - or add a value typically between 0 and 1 and then only use the fractional part of the result for the system to exhibit complex behavior similar to the elementary automata discussed previously. Otherwise the averaging process will quickly settle the lattice into a uniform state all across. The multiplier or the added value becomes the determining factor in the behavior of the automata on the global level, with smaller values producing an overall smoother transition effect.

The behavior of the continuous-valued world can be further affected by assigning weights to each of the cell's neighbors, making variable the amount of influence each cell has in the neighborhood. In this case, weighted mean value is calculated for each cell when updating the states. Weighted automata rules increase in significance as the automata world is extended into higher dimensions as the geometrical properties of the system become more pronounced. In the f(x) environment, the automata idea is expanded into a three-dimensional world of regions, in which there are significantly more complex geometrical phenomena that emerge in the shape of patterns created. One can imagine a cube-shaped world of three-dimensional space sectioned into smaller cube-shaped regions, each of which has an associated state value and is defined in terms of surrounding space. The shape of the surrounding space depends on the neighborhood definition and is further molded by the weights assigned to each neighboring region. These spatial fluctuations can be viewed as continuous functions permeating a region of imaginary space while being defined in terms of each other. This imaginary three-dimensional world constitutes the foundation of the performance environment.

The environment enables a real-time exploration of the evolving spatial patterns by modifying a number of parameters, including the global addition value and the shape of the neighborhood by selecting different patterns of neighborhood weights and enables the user to manage predefined computer graphics algorithms and audio synthesis definitions in response to the automata behavior. The current automata world consists of 4096 regions or cells – 16 along each of the three dimensions. This choice is rather arbitrary having more to do with limitations on real time computing power than any logical or aesthetic considerations and can be easily altered. Currently, only two cell neighborhood configurations are used as the weighting provides further options for different neighborhood configurations. The three-dimensional equivalent of what is known in cellular automata literature as 'Moore neighborhood' considers 26 nearest neighbors, including those diagonal from the center cell on the lattice. Figure 1 illustrates the Moore neighborhood in 2D, with the neighborhood surrounding the center cell shaded darker, and the equivalent in 3D with the center cell represented as a black cube and the neighboring 26 cells as wireframe cubes.

 

System overview.

The system has been implemented in two separate parts. The automata world and graphics functions are part of a separate custom-written application, while the control center, audio synthesis and activation routines, and user interfaces for external control have all been implemented in the SuperCollider programming environment. The communication between the applications is managed through the Open Sound Control (OSC) protocol. Cell state values are used to generate audiovisual content during the performance. Due to the large size of the world and the high standard message rate (24 cycles per second), only a selection of individual cell values is polled by the SuperCollider application for audio synthesis mappings. This selection can also be specified and modified during the performance and it usually coincides with the selective visual representation of the cells. In addition, a global mean cell state and standard deviation for the entire world as well as mean state values for 8 sub-regions are calculated and sent out through the OSC client. The sub-regions are created by splitting the cube-shaped world in half along each of the 3 axes, thus creating 8 smaller cubes of 83 or 512 cells each.

Another dynamic feature of the automata world is linear interpolation of cell state values, which allows “time-stretching” the automata behavior by delaying the update function by a desired number of cycles. This feature allows slowing down the propagation of fast patterns when the addition value is set relatively high. Drawing functions are implemented as encapsulated patches, each with a specific characteristic. These patches can be easily added and modified during development and constitute a basic unit of the graphics application. Each patch has a number of parameters for transparency, color mapping and other values that can be controlled remotely. Besides the individual patch controls, global transparency, screen background color, zoom along each of the 3 axes, speed and direction of rotation, and relative rotation angle around each of the 3 axes can be controlled in real time. The added functions are expected to contain mapping logic from state values to audio synthesis parameters and can also be used to control certain parameters of the graphics application, creating parameter feedback loops during the performance. Audio synthesis definitions and time-structuring routines can be added dynamically during the performance either from predefined functions or specified in a live-coding just-in-time programming manner.

 

Audiovisual composition.

The audiovisual composition relies on two elemental concepts: audiovisual macro compositional entities called zones and the separation of audio and visual content within these entities both physically and conceptually. The concept of a zone emerged in the process of developing this project as a means to have a meaningful segmentation of considerably different audiovisual material and allows introducing new approaches to audiovisual mapping without restructuring the existing organization. Each of these zones can be activated in any sequence during a performance and, for better or worse, have provided a general form for the entire composition. Each zone is characterized by unique and specific mapping procedures resulting in a distinct audiovisual entity.

The disparate networked structure of the environment keeps the conceptual and operational schism between the auditory and the visual. However, it is intended that neither is subservient to the other medium and both have an independent and equally important role in the composition. The two worlds are explored for their inherent characteristics and there is always a hint of a connection, at times seemingly synchronized and at others not easily identified. The 3D world is represented in computer graphics by mapping selected cell state values to low-level vertex drawing functions. Most commonly the cell state or inverse of the state is mapped to the drawn object’s grayscale color, transparency and size. The variation of these three parameters already yields a significant palette of mapping options, considering that what really makes an impact in terms of the visual content are the intricate patterns that emerge even from the simplest of representations.

Audio synthesis parameter mapping is driven by the messages received from the automata world and handled through 3 different types of OSC responder functions as described in the previous section. A commonly utilized technique is to activate a number of parallel synthesis processes, identifying a certain number of modifiable parameters and mapping the incoming cell state values to appropriate mapping ranges for each parameter. Each synthesis process can have its custom mapping range for every selected parameter or synthesis processes can be grouped and the same mapping ranges used for equivalent parameters. The mapping ranges and curves themselves can also be modified during the performance, which provides even more powerful expressive tools for the performer. An example of the described mapping technique from the ‘zone 1’ section of the environment defines first a prototype synthesis definition for time stretching existing audio buffers with 8 modifiable parameters: overall amplitude, start read position in the buffer, end read position in the buffer, amplitude of the signal sent to auxiliary effects bus, pitch-shifting ratio, sound field rotation angles around the x-, y-, and z-axes (in terms of Ambisonics spatialization).

When the 8 synthesis processes are activated in parallel, each with 8 modifiable parameters, a cell states value responder function is added to the responder function evaluation queue containing instructions to map the incoming cell values to the specified ranges. Each time the SuperCollider language application receives a message marked as ‘cell states’, the function is instructed to clump the 64 incoming values into groups of 8, and then map each of 8 cell state values in a group according to the 8 parameter ranges and update the running synthesis processes with the obtained parameter values. Similar technique is used throughout the different zones, however, this is not the only method of mapping employed in the environment. Synthesis processes of shorter duration are also triggered according to received messages, for instance, and there are ranges of cell state values specified within which a certain process is active and is automatically switched off if a specific cell state goes out of that range. There are an endless number of alternative mapping strategies and as the environment is developed further, hopefully new approaches will keep surfacing.

 

Conclusion.

f(x) is an experimental environment, continually under development, exploring possible strategies for audiovisual laptop performance, in which the generative computer art paradigm is brought into the context of live performance. Every performance on the system has so far produced distinctly differing outcomes, even though the introduction of the concept of 'zones' has channelled the performances into previously explored territories and encouraged the performer to favor certain system behaviors over others. In general, the environment fosters the idea of the composer/performer as an explorer or a discoverer, rather than a spontaneous creator. The design of the system tries to balance the amount of control and gestural character of the human performer against the distinctly non-human behavior of the automata world, simultaneously exploring the conceptual differences between the auditory and visual composition. 

 

References and Notes: 

 

Alo Allik, Tehis: a cellular automata programming environment for computer music composition. (M.A. Thesis, Institute of Sonology, the Royal Conservatoire, The Hague, the Netherlands, 2005).

 

Peter Beyls, “Cellular automata mapping procedures.” Proceedings of the International Computer Music Conference, Miami, Florida. (2004)

 

Open Graphics Library: http://opengl.org

 

Open Sound Control: http://opensoundcontrol.org/

 

Norman Packard and Stephen Wolfram, “Two-dimensional cellular automata.” Journal of Statistical Physics 38, Nos. 5/6, (1985): 901-946.

 

SuperCollider: http://supercollider.sourceforge.net

 

Alan M. Turing, “Computing machinery and intelligence.” Mind 59, (1950): 433-460.

 

John von Neumann, The theory of self-reproducing automata. (University of Illinois Press,1966).

 

Stephen Wolfram, A New Kind Of Science. (Wolfram Media Inc., 2002).