art & technologies

Mobile Tagging as Tools for Augmented Reality


The objective of this paper is to describe the potentialities of Mobile Tagging (2D barcodes like QRcodes) as a tool for increasing and spreading the effects of Mixed Realities in Art. In this sense, we will start introducing the main concepts and some examples of Mixed Realities followed by the concepts and examples of Mobile Tagging, showing that they are connected and benefit each other. 





Mobile Tagging as Tools for Augmented Reality

Martha Gabriel
Universidade de São Paulo


The objective of this paper is to describe the potentialities of Mobile Tagging as a tool for increasing and spreading the effects of Mixed Realities, including in the field of Arts. In this sense, we will start introducing the main concepts and some examples of Mixed Realities followed by the concepts and examples of Mobile Tagging, showing that they are connected and benefit each other.


Mixed Reality (or MR) refers to the fusion of the physical and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. On the other hand, mobile tagging is the process of reading a 2D barcode using a mobile device camera. Allowing the encryption of URLs in the barcodes, the mobile tagging can add a digital and/or online layer to any physical object, providing so several levels of mixed realities related to that object.


The uses of these levels of mixed realities have applications in several areas going from medicine and engineering to arts. This paper/presentation will use some artworks as examples to illustrate the functionality of the mobile tagging for creating mixed reality.


Mixed Reality


According to the Virtuality Continuum concept (MILGRAM, 1994), the mixed reality is anywhere between the Virtual Environment and the Real Environment, comprising stages of reality, augmented reality, augmented virtuality and virtuality.


Examples of Virtual Reality are the immersive caves, where the interactor dives into the virtual environment. Some examples of mixed realities applications are:


 MINI Cabrio – car advertisement (Youtube, 2009-2). 


SPOILER - game (Youtube, 2009-1). 

BMW - engine maintenance (Youtube, 2009). 


Arcane Technologies  - educational and military applications (Arcane, 2009).

Several kinds of devices and technologies can be used as tools for mixed realities, such as glasses, gloves, monitors, computers, cameras and mobile devices (PDAa and cell phones). Due the pervasive nature of the mobile devices, their potentiality for increasing the dissemination of mixed realities is enormous and can be leveraged by mobile tagging as described next.


Mobile Tagging


Mobile tags are 2D-barcodes that can be scanned by mobile devices in order to decode the information kept in the barcode.


There are many types of 2D-barcode (tag) and it is possible to encrypt many kinds of data into them. However, regarding mobile tagging, the most common encrypted information is URLs. The process of mobile tagging consists of scanning the tag with a mobile device camera using a mobile tag reader, which decodes the tag, opening the decrypted information on the device screen.

The most used patterns of 2D-barcodes for Mobile Tagging are QR Code (Quick Response Code) and Datamatrix. While conventional bar codes are capable of storing a maximum of approximately 20 digits, a QR Code, for example, is capable of handling up to thousand characters and all types of data, such as numeric and alphabetic characters, Kanji, Kana, Hiragana, symbols, binary, and control codes (Denso-Wave, 2009).


According to (Denso-Wave, 2009), the capacity of storage of a QR code is:


QR Code Maximum Data capacity 

Numeric only - 7,089 characters 

Alphanumeric - 4,296 characters 

Binary (8 bits) - 2,953 bytes 

Kanji, full-width Kana - 1,817 characters 


Nowadays, although mobile tags are still a novelty for most of the countries, they are starting to spread as the 3G mobile technology get available around the world. The QR codes use is very common  already in Japan and Datamatrix is more used in Europe, especially in UK. Most of the new models of mobile devices come already with the mobile tags (QR code and Datamatrix) reader. Older versions of devices can install a QR code reader, such as i-nigma (, becoming so able to scan them.


A very interesting use of mobile tagging as mixed reality is the ( that stimulates the use of in physical places that are present in the Wikipedia, mapping them. In this sense, each place is provided with a new layer of dynamic information coming from the digital online world, increasing so their use.


Mobile Tagging in Art


There are several interesting examples of exploring mobile tags in Art. We will present three artworks using QR codes that range from traditional arts to electronic interactive arts. The first example is the work STILL by Frabrice de Nola (2011) that uses 2D and liner barcodes to build the image and links.

We can mention as one of the first interactive digital artwork the SENSITIVE ROSE (Gabriel, 2008) artwork, which builds an interactive compass rose formed by QRcodes that navigates into people’s desires (figure 1).  The work is a big projection (3 meters x 3 meters) and all the interactions happen through this projection by scanning the dynamic QRcodes for participating. The work was launched in November of 2008 and has received already more than 800 interactions (May.2009).


Figure 1 – Screenshot of the artwork Sensitive Rose (GABRIEL, 2008)


Another interesting artwork that uses QRcodes is the “Suite 4 Mobile Tags” (Beiguelman, 2009) which proposes an exercise of random and anonymous collective musical composition. By pointing a phone w/ QR-reader to a display, participants play a ringtone. The result is a sudden and temporary suite that plays with hi and low tech, the portability, the confusion between public and private, music and noise.




Since the mobile tags are simple tags that can be placed in virtually any physical object or person, added to the fact that the cell phones with camera have become a very inexpensive and pervasive device, the mobile tagging process can be said as one of the easiest and simplest way of creating mixed realities.


The use of mobile tagging can range from expanding the information on packages, bus stop routes, museum objects, to art.


Mobile Tags work like physical links to the web, allowing so that virtually anything can be part of an expanded mixed reality environment.


References and Notes: 

Arcane Technologies (2009)

Beiguelman, Giselle & Fleury, Mauricio (2009). Suite 4 Mobile Tags. []

Gabriel, Martha C. C. (2008). SENSITIVE ROSE. []  

Milgram, P. & Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information Systems. Vol. E77-D, No. 12. Dec.1994.

Nola, Fabrice (2011). Still (Mirror Edit). []

Youtube (2009). BMW augmented reality.

Youtube (2009-1). Cube Speedrun *SPOILER*

Youtube (2009-2). MINI Cabrio: AugmenAugmented Reality Ad.

Wikipedia (2009). Mixed Reality. [] 



The Legacy of Experiments in Art and Technology (E.A.T.) : an Environmental Aesthetics

E.A.T.'s legacy rest on the early development of an environmental aesthetics. This aesthetics, however, does not focus on the idea of nature (as the prevalent notion of environment has it) but rather on the built and, particularly, the technological environment. This environmental aesthetics problematizes the nature/culture dichotomy in a manner that is of particular relevance to contexts that are increasingly infiltrated by technology.


Experiments in Art and Technology (E.A.T.) is a well-known example of interdisciplinarity at the intersection of art, science and technology. It was founded by Billy Klüver, Fred Waldhauer, Robert Whitman and Robert Rauschenberg in order to facilitate collaboration between artist, engineer and industry. As Branden W. Joseph wrote in Artforum in 2004 : “Klüver bequeathed to us a set of questions and contradictions involving art, industry, technology, and corporate sponsorship that--amid the glitz of new technologies and the renewal of foreign wars--deserves a place at the forefront of our historical consciousness.” [1] Indeed, E.A.T. sought to recognize the role of technology in society and especially the new responsibility of artists and engineers in complex industrial societies.

The organization has been examined in part by curators, art historians and researchers who focus mainly on the 9 Evenings: Theatre and Engineering festival (1966) and, to a lesser degree, on the Pepsi-Cola Pavilion at “Expo '70”, at the Osaka World Fair. The Pavilion can be seen, as Fred Waldhauer says, as “a culmination of the experiment during 9 Evenings.” E.A.T. and its projects are often interpreted either in terms of their success, by defenders of new media art, or their failure, by contemporary art critics in the main.

From a different perspective, however, the Pavilion can be considered a turning point, and some experimental works and projects resulting from collaborations initiated by E.A.T. meaningful attempts to renew fundamental aesthetic questions. Closer examination of the statements associated with E.A.T. projects pre- or postdating the Pavilion, or even projects that remained unrealized (which are numerous and merit attention), reveals the omnipresence of the concept of “environment.” Beyond the development of devices as tools or instruments, that would be available to other artists, this notion, investigated in Oracle and in the Pavilion, can be seen as a key concept in seeking to understand the switch by E.A.T from an art to a non-art context. E.A.T.'s legacy can be said then to rest on the early development of an environmental aesthetics, which does not focus on the idea of nature (as the prevalent notion of “environment” has it [2]) but rather on the built and, particularly, the technological environment.


Following Allan Kaprow's Assemblage, Environment and Happenings or the exhibition Environments, Situations, Spaces, it is tempting to consider an environment as a new art form. The Oxford English Dictionary defines an environment in art as “a large structure designed to be experienced and enjoyed as a work of art with all (or most) of one's senses while surrounded by it, rather than from outside.” The definition is precisely illustrated by a quote by Robert Whitman from The New Yorker on the Pepsi Pavilion. [3] But the environment (“that which environs”) is a much more difficult notion, according to Frank Popper or Peter Sloterdijk. Contrary to the artistic concept of landscape – which implies the idea of a frame – the concept of environment seems to be used in art to question our understanding of our “surroundings”, and how it affects us and can be affected by us. As a result, certain of Tinguely, Rauschenberg and Cage's works can be seen as different attempts to focus on the perception of our shifting urban environment, in this case New York City.

Rauschenberg's Oracle is an essential work that deals primarily with this issue. Deeply affected by Jean Tinguely's self-destructive sculpture Homage to New York (1960), Rauschenberg subsequently tried to realize, with the help of Klüver, an “interactive environment where temperature, sound, smell, lights, etc., could be affected by the person who moved through it.” [4] This resulted in Oracle. First exhibited at Leo Castelli Gallery in 1965, this work consists of a console with an aluminium staircase housing AM radios and electronic control equipment, to which Rauschenberg added four other sculptural elements described by him as “gifts from the streets”: a round industrial duct in the form of a funnel; a window frame with duct; a car door mounted on a rolling typewriter table, with a large piece of crumpled metal behind it, and a basin combined with an air-conditioning duct through which water circulates. All of these assemblages were on wheels so that the artist could move them freely. As it is no longer possible to interact with the work, one has to revert to earlier descriptions of the experience, in the 1960s, to understand how this work problematises the perception of environment. Klüver specified that, in this installation at the Leo Castelli Gallery, “the viewer could freely walk among them and operate the controls on the staircase.” Visitors were able to move control the volume and the speed of the dial scan. “In full operation,” Klüver wrote, “Oracle becomes an animated cityscape.” As an active participant in the work, the visitor took part in this urban environment and was responsible for what he did. Oracle was also described by the art critic William Berkson as “a funhouse, a torture-chamber or a laboratory for testing perception.” This latter metaphor corresponds exactly to Andrew Forge's description:

To stand by the console is to be aware of a continual coming and going of sound, predominantly spatial. […] The very sensation of hearing, it seems, has become a kind of looking. One doesn't know which way to point one's eyes, so strikingly does the sound reframe the appearance of the machines, giving them a kind of speed -despite the fact that they just stand there dinning- or if not speed, a kind of flashing nowhereness like parked cars seen peripherally from a speeding scooter. And as soon as you move away from the console, among the pieces, you find your movements, your familiar physical measuring of close distances becoming a matter of urgency to be set alongside this new space that you are hearing inside your head. [5]

Forge's description of his experience focused on the plasticity of sound and the mutual interdependence between seeing and hearing which Rauschenberg would explore further during the 9 Evenings with his performance Open Score. This festival took place at the 66th Regiment Armory in New York, in 1966, as a result of several months of collaboration between artists and engineers from Bell Laboratories. According to Clarisse Bardiot, the Theater Electronic Environmental Module, known as the TEEM, was the “major achievement of 9 Evenings and its most important message for the art experimentation that would follow.” [6] Designed to fulfil the function of an on-stage environmental electronic system, it enabled the performers to reconfigure the space in which the action took place by using a remote-control system for the lights, speakers, microphones, cameras, film, motors, etc. John Cage's statement for his performance Variations VII was to “use sounds available at the time of the performance” picked up indifferently inside or outside of the Armory. 10 telephone lines were installed in the Armory, open in different places in New York city including a restaurant, an electric power station, the New York Times press room and Merce Cunningham’s studio. In addition, there were contact microphones on the performing platform itself and on domestic appliances (a fan, a juicer, etc); there were also 20 radio bands, 2 television bands, and 2 Geiger counters. 30 photocells and lights set up around the performance area activated the different sound sources as the performers moved around. Classical musical composition and traditional instruments are here replaced by a protocol which welcomes, like Oracle, selected sounds “in the air” and challenges profoundly the perception of inner and outer exhibition space frontiers as well as the perception of distance.

McLuhan's theories – that John Cage always praised – were controversially discussed during the 9 Evenings. The recognition of technology by Klüver and Rauschenberg as a “natural environment” resembled in a certain way the naturalization of technology as emphasized by the media theorist and essayist. The variable environments built in the Armory can also be interpreted as a means to employ “multiple models for exploration”, McLuhan's so-called “method of our time”, to make people aware of technology's effects on perception. Indeed, McLuhan described the role of the artist, with regard to the technological environment, as follows :

Environments are not passive wrappings, but are, rather, active processes which are invisible. The ground rules, pervasive structure, and over-all patterns of environments elude easy perception. Anti-environments, or counter-situations made by artists, provide means of direct attention and enable us to see and understand more clearly. [7]

Marcelyn Gow rightly established that Klüver's concept of the environment differs from McLuhan's understanding of a pervasive and ineluctable process. According to Gow, Klüver thought it “in relation to human interaction with technology or what could be called programming, in order to produce specific effects”, i.e., where “feedback mechanisms” enable the effects of technology to be actively reshaped. [8] We can add that the core opposition, aesthetically, is based on the quantity and the quality of the feedback(s), that is to say between a conception of art as anti-environment (i.e as a reactive art form), and art as open-ended “experience, environment, process” (Barbara Rose), illustrated by the PepsiCola Pavilion for Osaka '70.

The Pavilion was a unique opportunity to work with industrial support, in the manner expressed in the E.A.T manifesto. While not deeply involved in the project, Rauschenberg suggested that artists working on the project “shift their approach to include elements that appealed to all the senses rather than just the visual, that is, elements that would create what people would feel as an 'invisible' environment.” [9] This “invisible environment” was quite different from McLuhan's determinist understanding of such an environment, since the visitors would be encouraged to participate and then to “create their own experience” in what they called a “living responsible environment.” Despite the relationship with Pepsi breaking down, the design and development of artificial fog and of an indoor programmable environment, responding respectively to local weather conditions and to visitors, can be considered meaningful artistic research. Outside the pavilion, Fujiko Nakaya's designed a Fog Sculpture in collaboration with the physicist Thomas Mee. 2,250 special fog nozzles were developed to create an artificial fog made of pure water at the request of the artist. Pumps were programmed to respond to different weather conditions: an automatic control system for programming was designed with real-time feedback of local meteorological data; namely, wind direction, velocity, and wet/dry bulb temperatures transmitted from the sensors at 6-minute intervals. The artist described her work as a “negative sculpture” because atmospheric conditions sculpted the fog in a concrete sense. She thus abandoned so-called “artistic control” in the shaping of this hybrid nature-culture artefact. Here again, however, she defined a protocol. Inside the Pavilion, a hemispherical mirror made of aluminised Mylar produced striking optical effects. One of these was an effect known in physics as 'real image', consisting of an upside down or inverted image that exists suspended in the 'real' space inside the dome, rather than in the 'virtual' image created by an ordinary mirror. (i.e. images produced in ordinary mirrors exist in a 'virtual' space behind the mirror itself). Above all, this interior space, consisting of the mirror plus lighting and sound systems, was also designed as an “instrument” to be used “by individuals from different professions who [would] come to the Pavilion to implement their program ideas, and through this participation be able to adjust, expand and extend their ideas in response to the situation and opportunities they [would] find there.” [10]

The Pavilion is a turning point not only because E.A.T. assumed an environmental approach  to its activity but also because the team moved to a non-art context; in large part they weren't even sure what they were doing was art (they came to recognize it as such by the end of the project). The move to a non-art context – which had already been achieved by 1969 – may disturb the world of art but it makes sense from an environmental aesthetics approach.  Thinking indeed that “the main influence of art and technology together will come in the area of the environment,” the aim of E.A.T. was redefined in 1969 by Klüver, to “encourage the artist-engineer collaboration to fulfil its potential as a revolutionary force in shaping the hardware and software of our technological environment.” [11] It is not a utopian definition since the nozzles developed for the Pavilion's Fog are currently used by Mee Industries Inc. in agriculture and industry, proving that successful transfer of an innovative technology, developed in collaboration with an artist, to industry is possible.

The word 'art', however, tended to be neglected in 'non-art' projects. Nevertheless, one can still recognize Klüver and Rauschenberg's aesthetic statements in an unfunded proposal for ten exhibitions. E.A.T. proposed a series of exhibitions for which the overall theme would be “Technology for the Individual: Recognition and Choices.” [12] Indeed, the subjects chosen represent “areas of technological change where the unresolved issues will affect the direction of technological development in advanced as well as developing societies.” The aim of these exhibitions was “to promote a recognition of the options presented by the new technology for the individual.” The exhibitions were planned for October 1969, with the opening of Automation House in New York City, established by Theodore Kheel for “people to adjust in a rapidly changing world of automation and helping the individual to have a sense of participation in the society in which she or he lives.” These exhibitions would have been designed by contemporary artists in collaboration with experts in the appropriate fields. The working titles for the ten exhibitions speak for themselves: “Variations of the Body: Genetics” by Allen Ginsberg; “Variations of the Body: Renovation, Transformation and Extension” by Steve Paxton; “Interactive Technology for the Three-year-old: Environments Designed by Teenagers” by Olga Klüver and Robert Rauschenberg; “Woman: Her Technological Environment” by Jean Dupuy; “Sports Equipment: Individual and Nature” by Claes Oldenburg; “Secrecy, Privacy and Snaring: Effect of the new communication and information technology” by William Burroughs; “Automation: Involvement or Alienation?” by Jean Tinguely; “Technology and the Environment: an Interactive, Computer-Simulated Ecosystem” by John Cage; “Atomic Energy: the Cloud and the Clear Sky” by Öyvind Fahlstrom; and “Shaping the Environment: Participation by the Individual” by Robert Whitman. This list comprehensively reveals the different preoccupations of the artists by this time, the extent to which they were concerned by the shifting environment they lived in and their desire to analyse the mix of technology, human and nature subsumed within this concept. Rauschenberg and Whitman's propositions in particular suggest an insistence on personal involvement as opposed to a more analytic approach.

Projects Outside Art (1969-1972) notably illustrated how E.A.T. paid attention to the specificity of a given environment, in contrast to McLuhan's position. Presented as “an exhibition of realizable projects in the environment”, interdisciplinary teams were asked to propose a project dealing with education, health, housing, concern for the natural environment, climate control, transportation, energy, etc., using the most innovative technology. Participants were asked “to recognize, in particular, the scale adequate for the problem undertaken, social and ecological effects, organizational methods necessary for realizing the projects” and for their project to “apply to specific geographical environments.” In Children and Communication, two groups of children from remote parts of the city of New York (considered as a rich and a poor district) were placed in two connected environments built by Robert Whitman. They were invited to experiment with available communication technology through the use of telex, fax, etc., a situation that Hans Ulrich Obrist compared to “a sketch for connected schools ages before the emergence of the Internet.” Another selected project, City Agriculture, aimed at creating closed-environment systems that would make it feasible to undertake city agriculture on a large scale. This also sounds extremely contemporary. Crossing different cultures or sociological contexts – city/countryside, rich/poor – was considered as a means of discovering solutions to contemporary problems while developing creativity.

Part of the multi-dimensional scaling projects or studies realized in collaboration with psychologists at Bell Laboratories, A Scaling Project Facing the Nation precisely dealt with the perception of social problems. The project aimed to correlate 22 economic, technological and social problems (i.e. unemployment, inadequate healthcare, pollution of the environment, racism, over-population, war, misuses of technology, etc.) with technical and scientific resources applicable to these areas. Individuals were given questionnaires and were asked to evaluate the relation between these problems and different contexts. The data was then processed by statistical analysis algorithms – the INDSCAL program developed by Douglas Carroll and Myron Wish at Bell Telephone Laboratories – and “the results were such that the axes in a three-dimensional space could be interpreted as local political--national political; technological--non technological; and moral individual--large scale organizational.” The subjects and the techniques involved (information visualization) are at the forefront of our contemporary preoccupations.

This move “outside art” makes sense as an attempt to escape the reification of art by the cultural industries and by the art world itself and to promote, as Klüver has it, variety and choice against repetition and uniformity. Nevertheless this move was misleadingly interpreted as a departure from aesthetics. However, the importance of aesthetic decisions and aesthetic conflicts in collaborative situations had already been noted and an aesthetics symposium was scheduled as part of the Projects Outside Art in 1970 to analyse these problems as well as to question the relevance of interaction between artists and engineers and of artists participating in non-art projects.

Through the concept of environment, the works and projects described stress the notion of artist or engineer control in art, technology and sophisticated industrial societies, and subsequently the role the artist can play therein. There was a belief in the possibility of improving the world quite different from postmodern cynicism. This belief placed the emphasis on the artists themselves – and in a larger context, on individual responsibility in a high-industrial context. E.A.T.'s environmental aesthetics does not consist of an aesthetic appreciation of natural, human environments or indistinct everyday activities. Rather, it is a question of examining art through the concept of environment and the environment through the practice of art; recognising a specific artistic expertise for environmental issues in art or non-art projects which require collaboration between artist and engineer. Moreover, this aesthetics problematises the nature/culture dichotomy in a manner that is of particular relevance to contemporary contexts increasingly infiltrated by technology. As a result, it can be brought to bear, fruitfully, on discussions of contemporary strategies in art and design, ecology and technology.

References and Notes: 
  1. Branden W. Joseph, “Engineering marvel: Branden W. Joseph on Billy Kluver,” in Artforum (March 2004).
  2. Allen Carlson, Nature and landscape : an introduction to environmental aesthetics (New York: Columbia University Press, 2009).
  3. J.A. Simpson and E.S.C. Weiner, The Oxford English Dictionary Vol. 5 (Oxford: Clarendon Press, 1989): 315.
  4. Billy Klüver, “Working with Rauschenberg,” in Robert Rauschenberg, A Retrospective, ed. Walter Hops, 310-327 (New York: Guggenheim Museum, 1997).
  5. Andrew Forge, Rauschenberg (New York: Harry N. Abrams, 1969), 230.
  6. Clarisse Bardiot, “9 evenings: theatre and engineering,” Fondation Daniel Langlois, 2006, (accessed September 4, 2011).
  7. Marshall McLuhan and Quentin Fiore, The medium is the massage : an inventory of effects (London: Penguin, 2008), 68-69.
  8. Marcelyn Gow, “Soft- and Hardware: E.A.T.’s Environmental Feedback,” Site Magazine (March 1, 2004): 10-11.
  9. Billy Klüver, Julie Martin, and Barbara Rose, ed., Pavilion (New York: Dutton, 1972), 20.
  10. Pepsi Pavilion: Expo 70 : live programming, June 13, 1969.
  11. Techne Vol.1, no. 1 (April 14, 1969): 1.
  12. E.A.T. proposal for ten exhibitions at Automation House, New York: Experiments in Art and Technology, March 17, 1969.

Labster / Text Collecting Workshop

Labster project aims at creating an online, interactive map, in order to offer personal narratives related to the worldwide Art and Technology DIY culture. In the first phase of the project, narrative texts are collected without a standard data input process to catch site of a wide range of terms, notions, definitions of lab’s practices, ecosystems and lab-life.
Saturday, 17 September, 2011 - 10:00 - 15:00
Attila Nemes
Catherine Lenoble

Workshop Leader: Attila Nemes, anemes [at]
2nd Leader: Catherine Lenoble, catherine [at]

Labster project aims at creating an online, interactive map, in order to offer personal narratives related to the worldwide Art and Technology DIY culture. In the first phase of the project, narrative texts are collected without a standard data input process to catch site of a wide range of terms, notions, definitions of lab’s practices, ecosystems and lab-life.

Syndicate content