Towards a Taxonomy of Interactivity

In this paper I take a critical look at the notion of interactivity; an inadequately defined term. It has very deep roots within the world, both as it self-organises and as we construct it. I offer an analysis of interaction based on the notion of “relations” as a general term for the interconnections through which all interactions occur. I examine the degrees of relations that operate so that information flow between entities is enabled.


When one looks critically at a great deal of the contemporary new media art that is described as interactive, one finds a huge variance in its formal manifestations such as

  1. the form of the artwork and its technical constructs; i.e., whether it is an installation or a performance, a sculpture or a software application
  2. the location and accessibility of the artwork; whether it is standalone or networked, situated in public space or gallery space
  3. the kinds of interactors; whether human or machine, audience or individual, or even computer to computer.

These kinds of categories constitute the taxonomy being developed by Katja Kwastek and visualised by Evelyn Munster for Ars Electronica [1] and no doubt others. However this kind of taxonomy does not investigate the actual processes of interaction itself, i.e., it does not explore what happens in terms of the flows of information and signs between entities, human or machinic, when engaged in these kinds of interactions. It is the intention of this paper to develop, in a preliminary form, a taxonomy of the kinds of relations through which information and signs flow in the actual activity of interaction, where relations constitute the connections or linkages between entities.

Interaction – and its co-relative, participation – have a very wide range of structures. Interactive artworks occupy a wide range of levels between the potential fully conversational robot and the video replay that simply switches on when the spectator enters the gallery. These run the gamut from the 'interaction' in the mind of a viewer in their active mental interpretation with an entirely passive artwork (a painting or sculpture) through to detailed and creative conversation between individual people and possibly between people and machines. It is the notion of the conversation that for me constitutes the full concept of an interaction.

In order to come to grips with an understanding of interaction in contemporary art and its range from the entirely one-sided to the fully conversational, I want to assemble an understanding that is as general as possible, so that it is not restricted in its application simply to human-machine interaction, but is operational over the full range of processes that can be described as interactions, be these the exchange of chemical products between micro-biological entities or the possibility of having a truly conversational relationship with a robot in the way that you and I might interact when we are having a conversation. This generality should then allow us to go even further, to the point where we might engineer truly conversational relationships between machines. I use the term 'conversational' because it entails a notion of inventiveness, which we might think of as the capacity to generate new behavioural repertoire and by which we might be able to produce a true artificial or machine intelligence, i.e., a creative machine, one that can not only trick us into believing that it has passed the Turing Test, but can truly generate new and creative ideas.

Interaction implies reciprocal actions or influences of two (or more) entities upon each other, where an entity is some kind of organised object of multiple components that has some degree of autonomy and agency. Thus, interaction takes place between entities that possess the capacity to act for themselves. It also requires that these actions alter the internal (cognitive) structure of each. At the most basic level interaction is fundamental to life, since it is the means by which an organism deals with and adapts to its environment. [2]

While this paper examines the interaction between an artwork and its spectators, in general the entities that engage in this reciprocal behaviour may be biological, social or machinic. Of the biological, at the lowest possible level of entity are single-celled organisms, at the highest possible level are whole societies or even whole ecosystems, though I will discuss human organisms for the most part. Machinic entities are artificial or constructed, e.g., technical and computer driven installation art, robots, and other forms of potential artificial intelligence. These devices must be, in some sense, adaptive, i.e., able to change their state to accommodate changes in their environment. This is a necessary first condition that has to be possessed by any entity that will undergo interaction.

There seem to be two general terms that apply. One, participation, though not usually thought of as involving relations with some kind of machine or constructed object, may be characterised as one-to-many, and involves engagement with a group of others in an event of some sort. To participate is to place oneself in the context of some process and to engage with whatever it offers that allows some kind of entry into the overall event. One becomes part of some larger thing or event that is the participation, per se. These can be happenings, theatrical events, and events in which the spectator has to supply at the very least their presence so as to complete the work. There will be multiple processes of connection developing over time producing a wide spectrum of activities.

The other, interaction, involves engagement with, usually, a single other entity (person or machine), and is commonly one-to-one. Again the work is not complete without the interaction, but here the focus is on reciprocal relationships and their development over time. To interact is usually considered to involve engaging with devices of varying kinds through the exercise of controls or sensors or other data gathering attachments that provide information as to changes in local conditions, and thus permitting the spectator, as user, to participate in the process of some 'thing' so that some kind of reciprocal relationship develops with it.

A distinction is often drawn between participation and interaction. This has an historical basis in that the word participation applied to early (1960s) forms of happenings and other event based art, whereas an interaction is usually thought of as being between a computer-driven or other machinic device and a person. In English usage, one 'participates in' but 'interacts with' some thing. However, in both situations it is the coherence of some larger process – a product of all the entities involved and greater than each entity when each is seen as an individual – that one experiences. As Pask has noted:

an observer who comes to know the system must be a participant in the system. The boundaries of the system, far from being pre-fabricated, are created by the activities of the system. This is a prescient notion of autopoiesis, or organizational closure.” [3, 353]

The use of the term 'participation' arose in the period of the happenings of Allan Kaprow or, for example, Nam June Paik's Participation TV (c.1963). Apart from the obvious person-to-person interaction required in a happening, I suspect that this distinction has lately been drawn through a need for a formal distinction between works produced through the use of analogue systems and works in which the computer is the locus of the choice-point selection process that is seen as interactivity in much recent contemporary art. Thus, participation and interaction can be shown to have a very similar set of characteristics whatever the technological means.

If participation was analogue and interaction is digital then given that they are both means by which one develops some kind of relationship with an other; be they animal, human, significant, analogue or digital machine (computer), environment or any combination of these, then are they not two words for the same thing? The key is that some sort of communication transpires; a reciprocal exchange of (generally 'meaningful') information that endures because of that meaningfulness and its reciprocation. Thus I argue that the separation between participation and interaction is meaningless, artificial, and misdirecting.

But what exactly are the processes of interaction? What are its characteristics? Firstly, whether the process is direct – through the exchange of chemistry (e.g., in biology), or mediated – through language or any of the extensions of our capacities that are embodied as analogue or digital technologies; a body, or some material functional object the states of which are alterable, must be involved. Thus, interaction must be embodied. Interactivity, being medium independent, needs some sort of physical channel through which information transfer can occur. These channels function as relations to the other and could as easily be speech as they could be a camera sensing people walking into the gallery. Ultimately what counts is what is recognised in the sensing and that, like it or not, is analogue.

Interaction is the relational dynamics that occurs between an entity – an organism or device – and its environment.

An organism is any coherent biological entity that metabolises energy in order to maintain that coherence (its organisation) within an environment, to gather and process information about its environment, and to permit its reproduction. At the machinic level are devices that are in some sense adaptive, i.e., that can change their state to accommodate changes in their environment. The need to be adaptive is a necessary first condition that has to be possessed by any entity that will undergo interaction.

An environment is all other organisms and the physical, social, cultural and machinic context that constitutes the experiential space of an organism for any duration. Every thing that is in some sense other to (i.e., not) an organism is its environment. Only the most sterile of environments are entirely passive or neutral; thus interaction and its corollary, adaptability, are necessary for any entity that has to survive in an environment.

To any organism its environment is 'active' when other organisms interact with it by competing with it for resources or by generating outputs (signals) into the environment which may or may not be useful to it. Its capacity to adapt to changes in its environment is essential to its continued coherence and its reproduction. This adaptive capacity is tested by its capacity to use the resources in its DNA, or program, and its stored experience to handle day-to-day changes. But to 'know', in any sense, about those changes it must be able to sense its environment and effect internal changes that accommodate those sensed external changes. These processes are structurally fundamental to interaction, and they constitute the primary level of the process of communication. [2] Further, they require the two orthogonally related conditions of 'autonomy' and 'agency'.

Autonomy implies that an entity can stand alone in some sense, making decisions based on its own knowledge of its situation. Its etymology is from the Greek auto for 'self' and nomos for 'law', i.e., self-driven or self-governing and, thus, self-regulating. Based on this we might think of something like a static autonomy, for example an object such as a painting or a sculpture that is complete in itself, through to an active mobile autonomy best represented by a living organism that is capable of moving, feeding, sensing and, overall, making decisions for itself. The notion that a static object, something that just sits there and does nothing is autonomous seems trivial but it stands as the lower bounding case of autonomy. We normally think of autonomy as applying to an entity that is in some sense self-sustaining; that has the capacity to sense its environment, operate on it, and thereby make decisions for itself, and thus we start now to see a merging with the notion of agency.

Agency is that property of an autonomous entity that is its capacity to act in or upon the world. That is, having made a decision it has the capacity to carry out (or execute) that decision.

The kinds of entities that have both autonomy and agency will be both biological (living) and artificial (constructed), e.g., robots and other attempts at artificial intelligence, and our chief interest here, installation artworks.

Adaptation by an entity to its environment both requires and supports its autonomy, allowing the organism to behave independently of other organisms, survive on its own and enact its own decisions. An organism's autonomy requires internal feedback relations in which aspects of its internal system can enact the regulation of its local environment in intentional ways. When these relations spread beyond the organism's boundaries you get social environments in which organisms communicate, sense and have intentionality and from this comes interaction. [4]

There are degrees of relations that may develop when a spectator encounters an artwork. These may manifest in several possible ways.

Degree 0: The artwork may be entirely passive and the only interaction is that process of the interpretation of an artwork that the viewer has to make to be able to see it and render it meaningful to them. Such action takes place entirely within the viewer, and although it is dynamic it has no impact on the artwork, which is itself entirely passive.

Degree 1: It may be triggered to start some playback of a pre-programmed sequence. Obviously interpretation on the part of the spectator is involved, but now the work becomes, in a very limited way, active. However there is no further impact on the artwork beyond the commencement of its pre-programmed trajectory.

Degree 2: It may respond with an action of some sort which will in turn draw further behaviour from the spectator. The artwork can now be said to be interactive. This is the common 'interaction' that occurs when the actions of a spectator elicit some sound, movement, visual or other event from the work that, crucially, causes the spectator to make further moves that are sensible to the artwork, thereby eliciting different sound or visual events from it. It is the kind of interaction that a musician has with an instrument, and the spectator may in fact be able to develop some skill with the artwork so as to be able to play it like an instrument. However here, the artwork's responses are all preprogrammed in the sense that a particular movement or action will elicit one particular response from the object, or may force a selection from several possible responses depending on, say, a contingent branch in the program flow.

Degree 3: If further movements of similar type are produced, a changing range of responses (e.g., new sounds) may be produced, since, having made one response the machine may then 'know' to make a different, albeit preprogrammed, response when given a similar stimulus.

The above classification is not dissimilar to that of Cornock and Edmonds. [5] For them interactions could be:

  1. static: allowing no opportunity for interaction
  2. dynamic-passive: change in response to environment, but not influenced by users
  3. dynamic-interactive: “generate outputs that correspond directly to input from audiences.”

and more recently Edmonds, Turner & Candy's addition of the class: dynamic-interactive (varying):distinguish[es] articles that change over time, either through automated learning or through updates from the artist.” [6]

However they do not go far enough, thus:

Degree 4: By any measure the peak class of interaction is conversation – an ongoing, inherently stable, multi-sided, adaptive process of information transfer, that consists in alternating, reciprocal production and transmission of information and response to that information, through consideration, recognition (of signs), understanding (of their meanings), development or extension of 'ideas' embodied in the messages and the production of further transmissions. Conversation must involve understanding which is a function of a mutually agreed, or learned, set of signs (language) that convey the meaning.

Conversation necessarily involves feedback; the closing of the loop through the response by the second party, which is in turn considered and responded to by the first. Thus a continuing cycle of feedback undergoes temporal development as the conversation continues, and each party is, at least, able to utilise its existing repertoire of behaviours – ranging from language and gesture to the demonstration of objects and processes or the operation of machines. This cybernetic feedback relation, though largely neglected in contemporary art over recent decades, provides a framework of immense value in understanding how interactive systems can work, and it is the circular feedback system that renders the conversation something greater than what exists within each party, such that its coherence gives it a mutually embodied autopoietic presence.

More interestingly, in any intelligent entity (living or artificial) the learning of a set of signs to convey meaning will require the development of new behavioural repertoire through a process of adaptation. The interactive context (the environment) will make demands on each entity and place constraints as to the effectiveness of any behaviours, moulding the development of any new repertoire.

In sum, while none of this is strictly 'new,' little of it has been spelt out and incorporated into contemporary art practice. Pask's work would be the only counter example. Regarding conversation, and paraphrasing him [3, 358–360]:

  1. Conversation between individuals occurs over time and alters the mental contents of each individual over that time.
  2. Conversations have a start and a finish and unfold over time, although they may run in parallel, supposing more than two individuals are engaged.
  3. The conversation is in the the union of the minds of the individuals engaged. That is, it exists as a superstructure that is not contained exclusively in either mind but necessarily is a function of the activity of both.
  4. There is a process of feedback that gives conversation its unifying character.
  5. There is a “transfer of tokens” (language, signs) between each individual within the conversation.
  6. These 'tokens' must be mutually understandable. However, the interpretations of the conversation are nevertheless a function of each individual mind.

While many interactive artworks demonstrate some level of “reciprocal production” I know of none that have achieved a truly conversational level of interaction. This I suppose is due to the intractable problems of building a true artificial intelligence. One of the nearest approaches to this status is in Stelarc's work, the Prosthetic Head, [7] which has also been the locus of a great deal of work intended to produce various aspects of this capability under the framework of the Thinking Head project. [8]

Finally, I list the component sequences (i.e., the dynamic relations) of the process. What interaction needs is

  • A potentially dynamic system in some environment.
  • The entry of an interlocutor and a stimulus generated by that interlocutor.
  • A response to that first stimulus, functioning itself as a first return stimulus.
  • A further response to that first return stimulus followed by a further response on the part of the second party.

This must develop into an ongoing loop of stimulus-response sequences. Ideally it should follow a coherent line of development, and it might stimulate the production of new behavioural repertoire. Thus begins the process of developing a creative machine, in the way that we are creative.

References and Notes: 
  1. K. Kwastek and E. Munster, “Theme Landscape: Interactive Art,” (accessed September 2011).
  2. S. Jones, “Interaction Theory and the Artwork,” in Proceedings of Interaction: Systems, Practice and Theory Conference, ed. E. Edmonds and R. Gibson, 283–303 (Sydney: UTS, 16–19 November 2004).
  3. G. Pask, “Heinz von Foerster's Self Organization, the Progenitor of Conversation and Interaction Theories,” in Systems Research vol.13, no.3 (1996): 351–364.
  4. Stephen Jones, "Sensing, Communication and Intentionality in Artificial Life," in Proceedings of the 5th International Symposium on Artificial Life and Robotics, ed. M. Sugisaka and H. Tanaka (Oita, Japan: AROB, 2000), 26–29.
  5. S. Cornock and E. Edmonds, “The Creative Process where the Artist is Amplified or Superseded by the Computer,” Leonardo 6, no.1 (1973): 11–16.
  6. E. Edmonds, G. Turner and L. Candy, “Approaches to interactive art systems,” Graphite '04, Proc. 2nd international conference on Computer graphics and Interactive techniques in Australasia and South East Asia (New York: ACM, 2004), 113–117.
  7. M. Smith, Stelarc: The Monograph (Cambridge, MA: The MIT Press, 2005), 68–79.
  8. C. Kroos, D. C. Herath and Stelarc, "The Articulated Head Pays Attention," 5th ACM/IEEE International Conference on Human-Robot Interaction (Osaka: ACM/IEEE, 2010).