Algorithms as Structural Metaphors: Reflections on the Digital-Cultural Feedback Loop
Through a series of illustrated examples this article purports to show how the application of digital — algorithmic — paradigms to analog media may illuminate values and perceptions inherent in the digital models themselves.
The following excerpt from the full-length article focuses on three examples: an R.E.M. music video and two art projects co-authored by the writer. An extended discussion of metaphor, cognitive paradigms and perception, which establishes a theoretical background to the given examples, has been omitted for lack of space, as have most of the footnotes and images. The text has been edited for continuity.
In 2001, the American rock band R.E.M. produced an intriguingly “economical” music video for their song Imitation of Life : a four-minute clip of an elaborate and densely populated pool party, pieced together from a single, stationary, twenty-second long shot. A virtual camera wanders through the scene, zooming in and out to focus on different vignettes while the source clip loops continuously back and forth. 
The short duration of the single shot and its looped unfurling and recoiling divorce it from its linear cinematic substrate and transform it into a pictorial, a-temporal, medium perhaps more akin to a kinetic picture, i.e. a picture with moving parts. Were we to view the source clip alone running back and forth unedited, we would be unlikely to experience it as passively as we would do, a cinematic narrative. Rather, we would actively explore its pictorial-scape, tracing our own paths through the crowded scene, choosing the vignettes we wished to focus on and the duration of our engagement with them. We might begin with a close-up of a woman bidding a fond farewell to her older partner only to turn and run into the arms of another, zoom out and refocus our eyes on a man set ablaze by a barbecue fire, then shift our spatial orientation, perspective, and attention to a group seated around a table presided over by a woman whose monologue is cut off only as another empties a glass of water in her face, and so on.
The cinematic experience of the R.E.M. video is rendered, not through the narrative of the twenty-second shot, but rather through the (meta-) narrative imposed by the sequential stringing together of selected vignettes. The edited video offers one possible narrative path, simulating the wandering eye of a viewer exploring the a-temporal scene.
The iterative spatial and temporal reframing of segments of the video clip might be described as a perpetual resampling of the video “database,” analogous to the process of random access whereby computers retrieve arbitrary data directly, without the need to sequence through prior locations. If the random access metaphor seems apt, it is because we recognize in it a freedom that we associate not only with the nature of human perception, but also with cognition itself. The eye’s capacity to wander the visual landscape of R.E.M.’s pool party is intimately connected with our sense of agency and freedom of choice.
The dynamic and multifarious nature of perception has long been embodied in traditional East Asian painting. Unhampered by the convention of single-point perspective, Asian paintings often depict scenes that combine multiple perspectives within one composition. The viewer’s eye is made to wander in, out, and across the picture plane, alternating between grasping the composition as a whole and focusing in on details. Transitioning through scenes presented from multiple vantage points, the viewer constructs a multi-dimensional mental image of the depicted world. A peasant trailing a caravan of donkeys viewed from afar and above, for example, may appear at the foot of mountains which themselves are painted at eye level, and which may, in turn, be seen in the foreground of great cliffs viewed up close and from below. Mimicking human perception itself, the multiple scenes, each assigned an idiosyncratic perspective, appear to exist in a single harmonious space.
Similarly, as it wanders through the looped visual time capsule of the R.E.M. video, the virtual camera offers reframed vignettes for our mind’s eye to assemble into a coherent whole. The video thus offers a representational model that evokes, at one and the same time, the manifold perspectives that harmoniously coalesce in the traditional Oriental painting, the mechanism of random access data retrieval, and the perpetually shifting focal points and fields of vision that characterize the free dynamics of human perception.
French phenomenologist Maurice Merleau-Ponty died in 1961, almost thirty years before the dawn of Photoshop. ”When the data of the world is subjected to coherent deformation,” earlier that same year he wrote, “meaning is always present.” 
Merleau-Ponty’s characterization of experience as malleable data to be acted upon rings surprisingly contemporary, as it conjures up a world that can be accessed by means of algorithmic computation. Merleau-Ponty’s view corresponds to western ideals of artistic expression that have, since the early Renaissance, exalted the creative vision of the individual artist. Indeed, Modernism strongly endorses the notion that what constitutes the particular creative vision of an artist or artistic movement is expressed through their idiosyncratic, yet coherent, representations of reality. In other words, it is the coherent de- and re-formation of reality that we commonly refer to as the expressive quality of a given work identifying in it the artist’s conceptions, emotions, and ways of seeing, or in short, its meaning.
Present day technology forces us to reexamine Merleau-Ponty’s analysis of the provenance of “meaning.” We need only consider the multitude of preset filters packaged with standard media editing tools, such as Photoshop and After Effects, to realize that these programs provide their users with a host of automated procedures for fabricating “coherent deformations.” However, does the rote application of a graphic filter necessarily produce meaningful expression in the altered image?
It might be argued that the meaning inherent in the application of a standard filter is the effect of the distortion itself. Thus, the meaning of a Gaussian Blur filter, for example, would be the blurriness of the affected image. However, we are only able to attribute meaning to the Blur filter due to a shared convention about what “blurriness” denotes. It is not the automated distortion; that imbues the image with meaning, but rather our perception of it.
Whether our perceptions are cognitive models grounded in convention (such as when we interpret the meaning of standard image filters), or formed in response to an artist’s singular idiosyncratic vision, they acquire meaning by virtue of their relationship to their object of contemplation--that is, by virtue of intentionality without, intentionality meaning would remain indeterminate.
According to the Sapir-Whorf hypothesis, the cognitive structures fashioning the meanings we assign to a given image or utterance in everyday discourse are generally indiscernible as they constitute a seamless part of our perception. We become cognizant of these structures when ruptures occur in our cognitive “fields of view.” These ruptures may be caused by internal sources, such as a sense of dissonance that rises to the level of awareness, or by external sources, such as an alternative paradigm that challenges our perceptions.
Art practice is a distinct arena where such challenges take place by design. “Dissonances” are introduced into artworks through “intentional coherent deformations” that are meant to draw attention to the structures of expression as much as to the content. In the words of Lev Manovich: “In art, the connection between content and [form] is motivated,”  and “just as modern thinkers, from Wharf to Derrida, insisted on the [opacity] of code idea, modern artists assumed that content and form cannot be separated.”  Consequently, the critical focus of modernism has been no less about the forms of expression than about the subject matter itself, often collapsing one into the other.
If the formal characteristics of an artwork shape its content, it can be said to render visible its structural code. While some elements of the code may be idiosyncratic, reflecting the singular vision of the creator, others are necessarily normative, relying on established convention. In either case, the code is made visible through the intentional analysis of its semantic value. Modern art is, therefore, a unique investigative space for exploring the structural determinants that underlie our conventions of representation and the semantic readings they dictate.
Viewed from this perspective, R.E.M.’s Imitation of Life video can be said to assign meaning to the navigational structure of the piece, i.e., to its non-linear “coherent deformation” of time and space–a reading which reveals characteristics of the code that otherwise might have remained opaque. Below are two examples of art projects that introduce digitally-inspired structural paradigms into physical interfaces, thereby capitalizing on the metaphoric potential of intentional coherent deformation.
The Garden Library
The Garden Library is an open-air structure situated in the heart of a public park in the center of Tel Aviv. Established in 2009 to serve the community of refugees and migrant workers who congregate in the park on weekends, the library has no walls or door and is comprised solely of two bookcases supported by the walls of a public shelter. It contains approximately 3,500 books in 16 languages.
ARTEAM, the artists’ collective that initiated and produced the library sought to break away from traditional categories of classification and to realize a sorting and indexing system that would playfully manifest the values of an open society. Accordingly, the books are not catalogued according to genre or author name, but dynamically, according to reader input.
On the inside back, cover of each book is a sticker that asks, “How would you describe the book?” and offers seven emotional responses the book may evoke: amusing, boring, bizarre, depressing, exciting, inspiring, sentimental.
When returning a book, the reader is asked to choose the fitting emotional descriptor, and the color-coded judgment is added to the history of responses on the spine of the book. The book is then placed on the shelves according to its latest emotional classification. In other words, the placement of the book is not decided by popular vote, but by the last reader, using a dynamic system that everyone can impact and in which every participant’s input counts. The cataloguing system constantly restructures the layout of the book collection, creating at any given point in time a transient “wandering map” that reflects the readers’ opinions and preferences.
ARTEAM thus sought to apply the non-linear algorithmic logic of digital technologies to the physical holdings of the library, transforming the book collection itself into a database that is continuously restructured on the basis of user input. The cross-disciplinary application of the algorithmic procedure to the library’s physical collection directs attention to the structure of the cataloguing system itself. The system transforms the library into a small, parallel world where the books wander between the shelves as their readers wander the world, carrying with them their emotional history.
Hall of Memory – Ghetto Fighters’ House, Israel
The Ghetto Fighters' House Museum commemorates Jewish resistance during World War II. Founded in 1949 by a community of Holocaust survivors and former members of the Jewish underground and partisan units, it was the world’s first Holocaust museum.
In 2007, the museum inaugurated a new Hall of Memory, designed to allow visitors direct access to archival material. Glass walls form the rear panel of the archive drawers, and visitors are invited to choose the drawers they wish to illuminate by touching blue light indicators on the glass panes. Touch-activated interfaces located behind the glass panes provide access to information about each item.
The designers sought to make available to the public the memories contained within the artifacts, enabling free access to the physical legacy of the country, its people, and its history. As in many traditional archives, these “semantic building blocks” of the historical narrative had previously been guarded as national treasures, open only to researchers and curators.
The multiple paths, which visitors can trace through the archives, echo the multi-thread non-linear structure of parallel computing. Indeed, the digital model was not only the initial inspiration for the design of the archive experience but is also integral to its symbolic reading. By transferring responsibility from the institution to the individual, the open archive democratizes the historical narrative. Each visitor becomes as it were a curator, entrusted with the task of determining their own path through the physical “database” of historical memorabilia.
The archive occupies two adjoining sides of the hall, with a large-scale generative installation on a third wall. The installation cycles through over 4500 names of Jewish communities that existed before the war. The names are formed from letters that float up from a rubble-like base, pausing momentarily to assemble into a name, and then immediately breaking apart again.
The fragmentary and individual sampling of the database of names formally echoes the visitors’ experience of navigating the museum archive. At the same time, the installation symbolically complements the archive by reflecting the notion that the viewer is responsible for sustaining historical memory. As each name falls apart, its memory lingers on only in the mind of the beholder.
George Lakoff and Mark Johnson’s Cognitive Theory of Metaphor has had a seminal impact on the perception of metaphor in contemporary semantics. Once looked upon simply as a linguistic or literary device, metaphor is now regarded as a conceptual mechanism. Lakoff conceptualizes metaphor as a cross-domain mapping,  namely, “a cognitive mechanism whereby one experiential domain is partially ‘mapped’, i.e., projected, onto a different experiential domain so that the second domain is partially understood in terms of the first one.” [9,10] Any yet, while the projected domain is intended to elucidate the target domain, metaphor is self-reflexive: it cannot help but draw attention to those characteristics of the original domain we “intuitively” perceive to be relevant.
In this sense, the R.E.M. video, which metaphorically applies the logic of a random access database, employs a representational paradigm that structures our interpretation of the video while simultaneously providing a context for a semantic reading of random access itself. Similarly, the cataloguing system of the Garden Library and the open archive of the Ghetto Fighters’ Museum are symbolic systems that imbue the projects with meaning while acting as interpretive models, which they borrow, from the digital algorithmic structures.
Such cross-disciplinary mapping from the digital to the physical sheds an intriguing light on aspects of the feedback loop between culture and the technologies it engenders. When they appear in a work of art, these metaphorical structures illuminate the cognitive constructs and values that digital technologies are introducing into our lives, perhaps representing a means by which the non-transparency of the digital code may become a little less opaque.
This paper was supported by the World Class University program funded by the Korean Ministry of Education, Science and Technology through the National Research Foundation of Korea (R32-20067).
- R.E.M., “R.E.M. - Imitation Of Life (Official Video),” YouTube, http://www.youtube.com/watch?v=0vqgdSsfqPs&ob=av3n (accessed March 26, 2012).
- Murmurs.com, “Garth Jennings Talks Life,” http://www.murmurs.com/topic/82227-garth-jennings-talks-life (accessed March 26, 2012).
- M. Merleau-Ponty, Das Auge und der Geist, trans. Hans Werner Arndt (Hamburg: Felix Meiner Verlag, 1967), 84.
- Lev Manovich, The Language of New Media (Cambridge, MA: MIT Press, 2001), 67.
- Ibid., 66.
- The Garden Library is a project by ARTEAM Interdisciplinary Arts (Romy Achituv, Marit Benisrael, Yoav Meiri, Hadas Ophrat, Tali Tamir), and Mesila - Aid and Information Center for the Foreign Workers and Refugees of Tel-Aviv-Jaffa.
- Hall of Memory Project credits: Bina Sela-Tzur (Museum Director) and Romy Achituv, Kowalski-Efrat Architects (Design).
- George Lakoff, “The Contemporary Theory of Metaphor,” in Metaphor and Thought, ed. A. Ortony, 203 (Cambridge: Cambridge University Press, 1993).
- Antonio Barcelona “The Cognitive Theory of Metaphor and Metonymy,” in Metaphor and Metonymy at the Crossroads, ed. Antonio Barcelona, 3 (Berlin: Mouton de Gruyter, 2000).
- Cristina Soriano, “Some Anger Metaphors In Spanish And English. A Contrastive Review,” in IJES 3, no. 2 (2003): 107.