Aesthetic Agents: Experiments in Swarm Painting

We introduce a swarm-based multi-agent system that produces expressive imagery through the use of multiple digital images. At birth, agents in our system are assigned a digital image that represents their aesthetic ideal. When groups of agents with different aesthetic ideals occupy the same canvas, a new image emerges through the convergence of their conflicting aesthetic goals.



Our system uses autonomous agents to model swarm intelligence for the purpose of non-photorealistic rendering – a category of research we will refer to as Swarm Painting.

Non-photorealistic Rendering (NPR)

Where traditional computer graphics has focused on photorealism, NPR looks to artistic styles such as painting, drawing, animated cartoons, and technical illustration as inspiration. In addition to its expressive qualities, NPR can offer more effective means of communication than photorealism by adopting techniques long used by artists e.g. emphasizing important details and omitting extraneous ones. [1]  

Autonomous Agents

An agent can be defined as “anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors.” [2] An autonomous agent is an agent that can operate independently and is capable of controlling its actions and internal state. Agents can be grouped into two general categories: cognitive agents and reactive agents.
Cognitive agents have an explicit symbolic understanding of their environment and can be seen as an extension of symbolic AI techniques. An example of a cognitive or intentional model is BDI-architecture. In a BDI-based model the beliefs, desires, and intentions of an agent forms the basis of their reasoning process. [3]
Reactive agents are specified by their behaviour i.e. how they react to perceived stimuli in their environment. In a reactive agent model, rules map perceived input to effectual output that is generally executed immediately. Purely reactive agents have no internal history or long term plans, but choose their next action solely upon the current perceived situation.
Each model has its advantages: cognitive models provide more powerful and general methods for problem solving; reactive models are faster and capable of producing complex emergent behaviour from simple sets of rules. [4]

Swarm Intelligence

Individually, social insects such as ants and termites appear to behave in a simple, almost random fashion. However, when a colony’s collective behaviour is examined complex and seemingly intelligent global behaviours emerge. Initially, it was assumed that the insects were either communicating in an undiscovered fashion or that each individual had some kind of internal representation of a global plan. However, research in the biological sciences has determined that the behaviour is in fact the result of individuals working autonomously with only local information. [5]
One way that collective intelligence can emerge is through stigmeric interaction. Stigmergic interaction refers to spontaneous, indirect coordination between individuals that occurs when the effect of an individual on the environment can influence the actions of others. [6] An example of this is the pheromone trail that an ant creates on the way back to the nest after it has found food. The pheromone trail attracts other ants who reinforce the trail with their own pheromones. Pheromones fade over time so once a food source is exhausted the trail to it disappears. This seemingly simple heuristic is so effective that it has been utilized to solve a number of combinatorial optimization (CO) problems, including the well know traveling salesman problem.
Swarm-based algorithms have a number of properties that make them successful at solving certain types of problems. They are versatile – the same algorithm can be applied with minimal changes to solve similar problems, robust – they keep functioning when parts are locally damaged, and population-based – positive feedback leads to autocatalytic or ‘snowball’ effects. [7]

Swarm Painting

Swarm Painting refers to swarm-based multi-agent systems in which a group of software- or hardware-based ‘painter agents’ move and deposit paint or change pixel colour values on a real or digital canvas. Swarm painting can be divided into two main categories: colour-based swarm painting and image-based swarm painting. 
To date the majority of Swarm Painting systems have adopted a colour-based painting approach. In a colour-based approach, agents paint a blank digital canvas with pre-determined or randomly chosen colours. The majority of colour-based swarm painting researchers utilize an ‘ant and pheromone’ model. In this model, a colony of virtual ants move and deposit paint on a canvas based on the distribution of virtual pheromones.
Another approach to swarm painting is to use an existing digital image as a reference for painting. The use of image files for NPR is a subfield within NPR called non-photorealistic rendering from photographs (NPRP).
The concept of using a digital image as a habitat for a colony of virtual ants was first published by Ramos at the 2nd International Workshop on Ant Algorithms (ANTS 2000). [8] In Ramos’ model, the grey level intensity of pixels in a digital image creates a pheromone map that virtual ants are attracted to. Ants deposit paint as they move and the trails they leave form a sketch-like image that contains salient features of the original image. Ramos’ primary interest was in image processing and not the creation of artistic works. In fact, the majority of research utilizing digital images as a habitat for swarm-based multi-agent systems has been concerned with non-artistic image processing tasks such as image segmentation, feature extraction, and pattern recognition.

Aesthetic Agents

Our system expands on previous research by using multiple images in conjunction with a swarm-based MAS for NPRP. Although our system references digital images for colour information it does not treat them as a habitat or environment. Instead, agents in our system are assigned a digital image that represents their aesthetic ideal. Accordingly, we refer to them as Aesthetic Agents.
On the surface, the behaviour of Aesthetic Agents does not seem to be stigmergic since the aesthetic ideal that agents are assigned can be seen as a global goal. However, the existence of multiple competing global plans produces images that are not the goal of any individual agent. Therefore, images produced by our system are the emergent result of local interactions since agents are not aware of each others goals or the image that will result from their interactions.
Aesthetic Agents are born in a toroidal digital canvas i.e. a 32-bit ARGB (Alpha Red Green Blue) bitmap image. Agents occupy a single pixel within the digital canvas and are invisible i.e. only their effect on the digital canvas is seen. When an agent is born it is assigned a 32-bit ARGB bitmap image that represents its aesthetic ideal. Aesthetic Agents are both reactive and autonomous. They are capable of ‘sensing’ the colour value of the pixel they occupy and those immediately surrounding them (Moore’s Neighbourhood) and can modify the value of the pixel they occupy.
Agents modify pixels through the interpolation of the RGB components in the pixel they occupy in the digital canvas c(x, y) with the pixel at the same location in the agent’s aesthetic ideal i(x, y). The amount of interpolation is based on a preset interpolation variable between the value 0.0 and 1.0 where 0.0 is equal to the first number, 0.1 is very near the first number, 0.5 is half-way between, etc. For example, if the interpolation variable is 0.1 (10%), the RGB colour value at c(x, y) is (0, 0, 0) and the RGB value at i(x, y) is (100, 50, 200) then the pixel at c(x, y) will be changed to (10, 5, 20) by the agent.
To initialize our system we create n agents, where n is the number of input images, and assign each agent one of the images as its aesthetic ideal. Only one agent for each aesthetic ideal is required since the offspring of agents are assigned the same aesthetic ideal as their parent. In our experiments we spawned our initial agents either in the centre of the digital canvas, c(width/2, height/2), or at random locations c(random(width), random(height)). For each iteration of the system, agents perform the following actions: Sense Colour & Move, Express Aesthetic Ideal (Modify Pixel) and Reproduce.

Experiments in Swarm Painting

We used our system to explore a number of concepts and techniques from a number of Modern Art Movements.


Since our system uses multiple images the most obvious visual technique to explore was montage. Montage (French for ‘putting together’) is a composition made up of multiple images. The technique played an important role in many Modern Art movements including Bauhaus, Dada, Constructivism, Surrealism, and Pop Art. To create a montage we simply take n images and assign each one to a different group of Aesthetic Agents. Figure 2(a) shows a montage made of an image of a skull, a lotus flower, and dice.


Impressionism was a late 19th century art movement based on the work of a group of mostly Paris-based artists including Monet, Pissarro, Manet, Sisley, and Renoir. Some of the characteristics of Impressionist paintings include small, visible brush strokes, an emphasis on light and colour over line, a focus on the overall visual effect instead of details, and a relaxed boundary between the subject and background. To explore these techniques we set different pictures of the same subject matter as the aesthetic ideals to different groups of Aesthetic Agents. Our intention was to try to combine similar elements of the same subject matter into an abstracted form. Figure 1 shows an example in which five groups of agents are given five different images of daffodils.


Cubism was an art movement in the early 20th century pioneered by Picasso and Braque. In Cubist artworks subjects are deconstructed and re-assembled in an abstracted form that often depict the subject from a multitude of viewpoints. To explore this technique we took photographs of the same subject from different angles and assigned the different perspectives as aesthetic ideals to different groups of Aesthetic Agents. Figure 2(b) shows the result of this technique and the increasingly abstract effect created as more angles and images are added.


Futurism was an artistic and social movement founded in Italy in the early 20th century by Filippo Tommaso Marinetti. The Futurists admired speed, technology, youth and violence, the car, the airplane and the industrial city – all that represented the technological triumph of humanity over nature. To the Futurists we lived in a world of constant motion, an idea that manifested in their painting manifesto:

On account of the persistency of an image upon the retina, moving objects constantly multiply themselves; their form changes like rapid vibrations, in their mad career. Thus a running horse has not four legs, but twenty, and their movements are triangular.

To explore this Futurist concept we took successive images of a subject in motion and set the images as the aesthetic ideals for different groups of Aesthetic Agents. See Figure 2(c) for an example of this technique.

Abstract Expressionism

Abstract Expressionism was a post-World War II art movement that is characterized by spontaneity, emotional intensity, and an anti-figurative abstract aesthetic. It was the first American movement to achieve global influence and was largely responsible for shifting the centre of the Western art world from Paris to New York City. Some notable painters of this style include: Jackson Pollock, Willem de Kooning, Mark Tobey, Mark Rothko, and Barnett Newman. Since we had discovered that increasing the number of competing aesthetic ideals in our system leads to increased abstraction we simply needed to use more images to create completely abstracted imagery. We found in general that around ten images is sufficient to remove all of the figurative details from a set of input images. Figure 2(d) shows an abstracted image made by assigning ten different images of a reclining nude figure to ten different groups of aesthetic agents.

The above examples demonstrate the importance of image selection to achieve a particular effect with our system. Although, some of the effects (e.g. Abstract Expressionism) can create interesting results from random image input, others like Montage require more mindful selection to achieve good results e.g. have figurative elements remain intact and still readable.

Conclusions and Future Work

In this paper we expanded upon previous research that utilized swarm-based multi-agent systems for NPRP through our use of multiple images. We successfully implemented a system that is easy to implement, versatile, and capable of producing novel, high quality artistic renderings. In doing so we demonstrated the power of biologically inspired models and metaphors to create new forms of artist expression. Furthermore, the simple implementation and effective results produced by our system makes a compelling argument for more research using swarm-based multi-agent systems for non-photorealistic rending.

We created our system using a swarm-based MAS, but we are certain that similar results could be produced using another programming methodology. Which begs the question, why use a swarm-based MAS methodology? To answer this we will adopt McCarthy’s justification of intentional systems that “although a certain approach may not be required – it can be useful when it helps us to understand and think about systems where a mechanistic explanation is difficult or impossible”. As computer systems become increasingly complex we will need more powerful abstractions and metaphors to explain their operation. This is particularly true in the case of modelling emergent phenomenon.

The dynamic nature of our swarm painting system makes it easily extensible to interactive applications. At the time of this writing we are working on a series of interactive installations in which agents are born and populations dynamically change based on input from real-world physical sensors.

In the future we would like to endow our agents with more more biologically inspired attributes and behaviours. More complex movement, feeding, and reproduction strategies will be investigated. In addition, we can extend our current model of an ‘aesthetic ideal’ to go beyond the colour values of pixels in a target image. Future agent’s aesthetic ideal could be be based on other visual elements such as contrast, brightness, and saturation or an agent could have a geometric bias towards creating certain shapes. To explore our system we used a number of Modern Art movements as inspiration for our experiments. Future work will explore the innate and unique qualities of our system. Finally, we would like to create Aesthetic Agents that inhabit a 3D world. Groups of agents could be given different 3D models as their aesthetic ideal to create emergent sculptures. Other Aesthetic Agents could add living textures to the 3D forms.

References and Notes: 
  1. B. Gooch and A. Gooch, Non-Photorealistic Rendering (Natick, USA: A.K. Peters Ltd, 2001), 250.
  2. M. Woodridge and N. Jennings, "Intelligent agents: theory and practice," Knowledge Engineering Review 2, no. 10 (1995): 115-152.
  3. A. Rao and M. Georgeff, "Modelling rational agents within a BDI-architecture," Principles of Knowledge Representation and Reasoning (KR 2004), Whistler, Canada, (June 2004).
  4. S. Bussmann and Y. Demazeau, "An agent model combining reactive and cognitive capabilities," Intelligent Robots and Systems (IROS 1994), Munich, Germany, (September 2004).
  5. E. Bonabeau et al., "Self-organization in social insects," Trends in ecology evolution 12, no. 5 (1997): 188.
  6. G. Theraulaz and E. Bonabeau, "A brief history of stigmergy," Artificial Life 5, no. 2 (1999): 97.
  7. M. Dorigo and M. Gambardella, "Ant colony system: A cooperative learning approach to the traveling salesman problem," IEEE Transactions on Evolutionary Computation 1, no. 1 (1997): 53-66.
  8. V. Ramos and F. Almeida, "Artificial ant colonies in digital image habitats - a mass behavior effect study on pattern recognition," 2nd International Workshop on Ant Algorithms (ANTS 2000), Brussels, Belgium, (September 2000).