Context Machines: A series of autonomous self-organizing site-specific artworks.

‘Context Machines’ are a family of site-specific, conceptual and generative artworks that capture photographic images from their environment in the construction of creative compositions. They are produced in an art-as-research practise at the intersection of generative arts, and cognitive theories of creativity and dreaming. They invite us to reconsider what is essentially human, and reflect on our constructed conceptions of ourselves.

Author(s)

1 Introduction

Context Machines (CMs) are generative artworks (GAs) inspired by models of memory and creativity drawn from cognitive sciences. A central motivation is the creation of systems whose output is, to some degree, surprising to the artist. The CMs' creative behaviour is manifest in the generative representation presented to the audience. CMs are image-makers, and the process by which they generate images is of equal or greater significance than the images themselves. Cohen [1] describes the relevance of cognitive processes in image-making:

An image is a reference to some aspect of the world which contains within its own structure and in terms of its own structure a reference to the act of cognition which generated it. It must say, not that the world is like this, but that it was
recognized to have been like this by the image-maker, who leaves behind this record: not of the world, but of the act.

The CMs share a number of core features: they all involve a computer controlled camera, used to collect images of their visual context, and use computational methods to generate novel representations. ‘Resurfacing’ [2] is a precursor to the cognitively inspired CMs, and is discussed to illustrate the transition between the overtly interactive artworks produced before 2006 − where the viewer's behaviour is integral to the work − and the emphasis on autonomy that informs the cognitively oriented CMs. ‘Memory Association Machine’ (MAM) [3] is an explicit application of the Self-Organizing Map (SOM) [4] and Gabora's theory of creativity [5]. The integration of these processes results in associative sequences of images captured from the immediate environment. This is the central contribution of this work and is further developed in the ‘Dreaming Machines’: ‘Dreaming Machine #1’ (DM1) and ‘Dreaming Machine #2’ (DM2) refine the method that generates associative sequences and frames them as machine dreams. During the day, associations are initiated by images in the world, while at night they are randomly activated and implicitly reference Hobson's [6] conception of dreaming. ‘Self-Organized Landscapes’ (SOLs) are high resolution print collages that reflect the SOM organization of thousands of pre-recorded images.

2 Background

CMs are characterized by features consistent with conceptual, site-specific and generative art practises. In conceptual art, the idea is of equal or greater importance than the object. Both conceptual art and GA have a strong emphasis on process over object. Conceptual art includes ‘instruction’ works where the artist provides a recipe for the construction of an artwork. These works are highly analogous to GAs, where the artistic concept is encoded in software instructions and executed by the computer. Site-specific art locates the meaning of an artwork in a specific social, historical or physical environment. For Kwon, a site-specific artwork gives “…itself up to its environmental context, being formally determined or directed by it” [7]. The CMs automate this task by literally capturing images of the environment, and using them as raw material from which to generate their own representations. GA is a niche within the broader context of electronic media art, a contemporary art practise at the intersection of technology and cultural production. For Whitelaw [8], “[n]ew media art self-consciously reworks technology into culture, and rereads technology as culture.”

Gabora's conception of human creativity [5] enables ‘Memory Association Machine’ and ‘Dreaming Machines’. The theory focuses on the generation of creative ideas rather than their evaluation. Gabora considers creative thinking a form of highly controlled association between memory components. A chain of many small, and perhaps obvious, associations can lead to surprising and creative results. The CMs' ability to organize diverse visual images is enabled by the SOM Kohonen (2001), which models a topological and content-addressable memory field analogous to the ‘conceptual space’ in which Gabora's creative associations occur. The SOM is an unsupervised AI technique where many simple units organize input patterns by similarity. The details of the SOM, as it is implemented in MAM, is discussed by Bogart [3].

3 Resurfacing

‘Resurfacing’ [2] integrates generative and interactive components. The artwork autonomously explores its visual context and collects images that are stored in a navigable structure. The installation is composed of two screens housed in an architectural facade, and a computer-controllable video camera mounted to collect images from outside the gallery. The system is initiated with twenty manually selected camera positions. The frames resulting from these positions are ‘moments’ indexed by the pan, tilt and zoom of the camera. Over the course of the installation, the camera continuously captures images as it cycles through these moments. The right screen shows a live video feed from the camera, while the left screen presents a collage of moments. The camera position (pan/tilt/zoom) is mapped to on-screen parameters (x/y/scale), resulting in an image that approximates, due to lens distortion and a lack of precision, the spatial relations between moments in the physical context. As the camera position changes, the collage translates and scales to match.

Sustained touch on the right screen results in a hole opening, at the contact point, that reveals corresponding images from earlier in time. As the viewer runs her fingers over the display, up to five layers of images, from the increasing past, are shown. Each moment is annotated with a ‘value’, calculated during each touch event, that reflects the relative number of contact events that occur while the moment is on screen. Each time a moment appears, its value is compared with a threshold. If the value is below the threshold, then a new random camera position will take its place during the next cycle. The value system ranks moments in order to replace low value moments with new and potentially interesting ones.

‘Resurfacing’ aims to facilitate the viewer's examination of aspects of the world to which she may be habituated. The machine's gaze is strikingly different than a human's. It tends to focus on visual items that are often ignored, providing a representational surface through which to encourage curiosity and exploration of the world.

4 Memory Association Machine

‘Memory Association Machine’ [9] (MAM) consists of three screens and a computer-controllable video camera. The left screen is a live video feed from the camera, and corresponds to the current stimulus. The middle screen (Figure 1) presents the system's memory field − the memory field that results from the SOM. The right screen presents MAM's associative sequence through collected images. Each screen presents one of the three processes that define MAM's behaviour:

(1) ‘Perception’ captures images from the visual context. The camera's gaze is driven by random pan/tilt values. For each associative sequence, the camera moves to a random position and one image is captured. Each image is sub-sampled to 40×30 pixels and fed to the ‘integration’ process as a vector of RGB values.

(2) ‘Integration’ organizes captured images into the memory field, as enabled by the SOM. The middle screen shows the memory field where each node is represented by its corresponding image (Figure 1). To emphasize the content of the images − and de-emphasize their arrangement − Gaussianoid alpha channels are used. The SOM is continuously training in its attempt to learn the structure of the world. Due to the finite number of memory locations, and the complexity of the world, the SOM will never converge at a stable topological representation that perfectly reflects the structure of the world.

(3) ‘Association’ sequences images from memory and is enabled by an independent network of units that mirror the arrangement of units in the SOM, such that each unit is linked to a corresponding image in the memory field. When a new input stimulus is presented to the SOM, the most similar image from memory is activated (presented on the right screen) and becomes the basis of a new associative sequence. The activation of an association unit results in the propagation of that activation to its neighbours to a lesser degree and after a random delay.

Each image is presented on screen with an opacity, and for a duration, proportional to the degree of activation. The sequence is complete when the degree of activation falls below a threshold. The camera chooses a new random direction and a new  image initiates another associative sequence. The length of these sequences is an emergent result of the interaction between the current image and the memory field. Reactivation is restricted by an inhibitory model that prevents already activated memories from being selected. At night, MAM ceases to capture images, and association units are randomly activated. This corresponds to the random PGO activation of brain regions that result in dream imagery, according to Hobson's AIM model [6].

‘Memory Association Machine’ uses a novel combination of a SOM and Gabora's theory of creativity to generate associative sequences of images. These images are collected from the visual context and represent the sum of the system's experience. MAM's random night-time associations inspire ‘Dreaming Machines’, in which sequences are framed as machine dreams.

5 Dreaming Machines #1 and #2

‘Dreaming Machine #1’ (DM1) and ‘Dreaming Machine #2’ (DM2) (Figure 2) refine the associative process initiated in MAM. DM1 is a prototype and uses the same video camera as in MAM installations. In DM2, the video camera is replaced with a digital still camera on a computer-controllable pan/tilt mount. Both ‘Dreaming Machines’ use a single screen that presents a fusion of the memory field and the associative sequence. DM1 and DM2 manifest the same process and only differ in hardware and installation details.

In one installation of DM2, for the Elektra festival, the camera is mounted on the second floor and looks over the street below. The associative sequence is projected on a large display in the lobby. The display shows the current activated memory in the centre, surrounded by its eight immediate neighbours, all masked with a Gaussianoid alpha-channels and overlapping fifty percent.

Whereas the camera in MAM was driven by random pan/tilt positions, the DMs use a random walk to trace the camera over the visual field. In the DMs, images are not sub-sampled and fed directly to the SOM, but are abstracted into RGB histograms. As demonstrated in the ‘Self-Organizing Landscapes’ (Section 6) the histogram is sufficient when used on unconstrained real-world images.

In MAM, memory activation is similar to dropping a pebble in a pond − energy is propagated in every direction. This results in an extremely dense and complex network of associations. In the DMs, an activated memory propagates only to its most similar neighbour. The strength of the activation decays proportionally to the degree of similarity between memories. Similar memories are visible for shorter periods, while dissimilar memories are shown for longer periods. The temporal inhibition, used in MAM, is replaced with memory specific inhibition where a memory will only be activated if its reference is not in a ring-buffer that stores previously activated memories. These refinements result in sequences that progress smoothly through individual associations [10].

An aesthetic weakness in MAM, DM1 and DM2 is that the SOMs never achieve a topological representation of the world. In ‘Self-Organized Landscapes’ the SOM is applied to a finite number of images where memory fields properly reflect the topology of the input images.

6 Self-Organized Landscapes

Due to their topological arrangements, the ‘Self-Organized Landscapes’ (SOLs) are the most faithful application of the SOM among the CMs. The bulk of SOLs are constructed from video frames captured on a hand-held HDV camera. Figure 3 shows an example SOL comprised of approximately 10,000 video frames, arranged in a euclidean lattice, overlapping fifty percent, where each is masked with a Gaussianoid. The SOLs directly apply knowledge attained through the development of previous installations.

7 Future Work and Conclusion

Current research is focused on ‘Dreaming Machine #3’ where production will move away from notions of creativity in order to explore implications and qualities of explicitly implemented cognitive models of perception, memory and dreaming. SOLs are large, and topologically correct, representations and are ideal ‘memory fields’ for the associative process used in DM2, resulting in ‘Dreams of Self-Organized Landscapes’. Another project would construct a SOL from images collected live from the context of installation.

CMs are artworks whose generative representational processes are inspired by images captured from their contexts of installation. Little research explicitly implements cognitive models of memory and creativity in artworks that learn from the world. These works encourage us to see the world anew through a reconsideration of art, perception, memory, creativity and dreams. The artwork is meant to be a public discursive interface for questions such as: What are crucial aspects of creativity and dreaming and can they extend to animals and machines? What aspects of humanity are not represented in AI systems and cognitive models? What is lost if we accept strict scientific conceptions of mind? A machine that creates and dreams is a reflection of our, perhaps misguided, conceptions of ourselves.

8 Acknowledgements

The authors thank the Social Science and Humanities Research Council for supporting the research that lead to MAM, and future work on DM3. DM1, DM2, the initial SOLs and ‘Resurfacing’ were produced thanks to support from the Canada Council for the Arts. ‘Resurfacing’ was produced in collaboration with Donna Marie Vakalis.

References and Notes: 

References and Notes

  1. Cohen, H. (1979). What is an Image? In Proceedings of the International Joint Conference Artificial Intelligence.

  2. Bogart, B. and Vakalis, D. M. (2006). Resurfacing. In Responsive Architectures, pp. 182–183. Subtle Technologies: Riverside Architectural Press.

  3. Bogart, B. D. R. (2008). Memory Association Machine: An Account of the Realization and Interpretation of an Autonomous Responsive Site-Specific Artwork. Master’s thesis, School of Interactive Art and Technology, Simon Fraser University.

  4. Kohonen, T. (2001). Self-Organizing Maps. Springer.

  5. Gabora, L. (2002). Creative evolutionary systems, Chapter The Beer Can Theory of Creativity, pp. 147−162. Morgan Kaufmann Pub.

  6. Hobson, J. A. (2009). REM sleep and dreaming: to-wards a theory of proto-consciousness. Nature Reviews Neuroscience 10(11), 803−813.

  7. Kwon, M. (2004). One Place After Another: Site-Specific Art and Locational Identity. MIT Press.

  8. Whitelaw, M. (2004). Metacreation: art and artificial life. The MIT Press.

  9. Bogart, B. D. R. (2007). Self-Other Organizing Structure 1: Seizures, Blindness & Short-Term Memory. In A. Brouse (Ed.), Proceedings of the Second International Pure Data Conference, Montréal; August 21−26, 2007. Pure Data Conference 2007.

  10. A selection of dream videos.