A painterly approach to 3D computer graphics

This paper is a review and exploration of the work over the last ten years of Light Years Projects, a collaboration between Jeremy Gardiner and myself. It covers in particular how I have explored the notion of a painterly approach to real-time 3D computer graphics. It deals with the tensions, temptations and opportunities that lie in the area where landscape painting crosses with technology, concluding with some of the lessons I have learnt.

Author(s)

Introduction

For the last 10 years I have worked in the field of digital landscape painting with Jeremy Gardiner, forming the partnership Light Years Projects (www.lightyearprojects.org). Our work starts with the landscape paintings created by Gardiner, and evolves into virtual digital landscapes, through the addition of my programming. During this time we have explored new techniques alongside the development of technology relating to 3D computer graphics but we have taken care to maintain a painterly approach to our work. The opportunities and limitations provided by technology can sometime lead to tensions and influence decisions that we, as artists make, but that is not always a bad thing. This paper explores our projects in terms of a painterly approach and the role of technology in our work.

A Painterly Approach

The term painterly generally has two meanings. The first is related to ‘having the quality of expertly brushed workmanship.’ [1] The second meaning comes from the German word term malerisch and relates to the use of juxtaposition of colour and tone, as opposed to the use of line, to create form. Wöfflin in Principles of Art History, pointed out that "the outlines and forms of some paintings, such as Rembrandt, are indistinct, and as such the viewer’s eye wanders freely over the painting, giving a sense of movement. [2] In a painterly painting, brush strokes may not represent a life-like reproduction, but a deliberate choice by the artist, where they have consideration of shape, colour, texture, light and expression are all used are used to provide more than a photographic-like result. The term equally applies to modernist landscape artists such as Paul Nash, whose work overtly interprets the landscape through the eyes, mind and hand of the artist.

The paintings of Jeremy Gardiner reside in the painterly and modernist approach. [3] The abstraction of landscape, through shapes, lines and colour represent many perspectives of the landscape and also seasons, times, and eras. The physical approach to creating the paintings includes impasto, cutting, sanding, painting, rubbing and scoring. These combine to form the experience of the painting, not just as a flat picture, but also as a three-dimensional object.

The opposite of a painterly approach is a linear approach, attributed by Wöfflin to the paintings of Botticelli and Vermeer, paintings where the outlines of objects and people are clearly defined, brush strokes smoothed and (in the case of Vermeer) an attempt to create a realistic-looking image.

So why did we choose the medium of 3D computer graphics that has, for the last 20-30 years, been taking a decidedly linear approach to image making, from pre-rendered animations to real-time computer game graphics? The answer is partly because of the challenge, but particularly because of the opportunity to explore this medium in a different way. We wanted to adopt a painterly approach to 3D computer graphics and to try to avoid the goal of realism.

Non-photorealistic rendering

The strand of computer graphics known as non-photorealistic rendering has been achieving a painterly effect, a simulation of paint-strokes applied a 3D graphics or video scene, for over a decade ([4][5][6][7] for example). However, this is not what I would describe as being a painterly approach. Computer graphic techniques are becoming more sophisticated in being able to analyse scenes and simulate the application of brush strokes to them. But it is apparent that the emphasis found in the literature is mainly on the technical achievement, and underestimates the intention of any artist in taking a painterly approach. Hays and Essa, in their 2004 paper wrote “a need for non-photorealistic rendering and animation is obvious to anyone who has marveled at artistic works where complex scenes and objects are rendered using pen and brush strokes, using lines, colors, and etches.” [4] In 2008, Bhattacharjee and Narayana wrote “The intentions of an artist come out as the aesthetics and expressiveness of the painting. The accurate rendering done by computers fails to provide images with a such a feeling …  Painterly rendering … can bring artistic abstraction to the rendering and thus mix the computer generated scenes with the hand drawn elements.” [7] These descriptions, and others, seem to simplify the artistic or painterly approach as being something that is reproducible via algorithmic methods. I would argue that this is not the case. A simplified artistic effect is achievable through these methods, but this is not the same as each stroke being deliberately made as part of the creative process, and each part of the composition being considered (sometimes sub-consciously).

I’m not saying that algorithms can’t be used intentionally by an artist, in Algorithmic Art, screen, print and plotter based computer art, artists set up the rules that create the subject and content of the picture. But the above papers, take away the role of the artist in the creation of ‘painterly’ images, in much the same way as I wouldn’t claim to have created a watercolour painting by applying the watercolour filter in Adobe PhotoShop.

Light Years Projects process

In the work of Light Years Project’s we have a two-stage process. The first stage is the creation of drawing and paintings by Jeremy Gardiner. Because of this, I have not felt the need or desire to simulate brush strokes in the 3D graphics stage, and instead use scans from the paintings mixed with 3D graphical components and events for the second stage. The intention has been to show the actual paint strokes and textures within the finished digital work, and create an infinitely varied painting and an experience.

We started creating our initial project Purbeck Light Years, in 2001. We layered cutouts of scanned paintings within a 3D landscape mesh that had been created from satellite data. The terrain itself was of limited resolution and hence all small features were smoothed out, only the hills showed, including the hill of the focal subject, Corfe Castle. Technology forced our hand on the detail shown on the landscape, but the result was agreeable to us. Gardiner’s hand drawn representations of the castle, formed overlapping outlines as the viewer roamed across the landscape. We made deliberate decisions about the positioning and texturing of the line drawings. By observing the results of the interplay of the line planes, we decided upon the outcome that we wanted.

Each of the cutouts from paintings had varying transparency. These planes encircled the line drawn Castle focal point and as the viewer moved, the planes overlapped each other. This enabled the image to be viewed in two ways simultaneously; as the illusion of three dimensions, but also, as a two-dimensional dissected image of shape and colour. We selected a number of planes and positioned them randomly around the castle. We did this partly to convert our aesthetics to a rule based composition and also to enable the piece to be co-creative with the audience (as well as being a co-creative partnership between us, as the artists). [8] The audience was able to interact with the piece to create the compositions that satisfy them aesthetically.

Another aspect of our painterly approach to the project was in capturing the visual essence of being in the location. In Gardiner’s paintings, colour and light are used to portray the atmospheric changes brought on by the seasons. These variations appeared in the interactive work; the sky and ground was covered in changing textures that subtly evoke different seasons and weather conditions. The light and colour in the virtual work changed, hinting at day and night. The mist would come in and the sound of rain heard. Other natural sounds (e.g. birdsong, wind, crickets) were also audible, some specifically located in the landscape (co-creative audio composition) and other sounds that occurred at programmed intervals. So in Purbeck Light Years we applied a painterly approach to audio media as well as visual media, in terms of composition, blurring one sound into another.

Limitations of technology

Our view when working on this project was to use a painterly approach to many aspects of composition across this virtual landscape. However, there was one aspect of painterliness that we were not able to portray, due to the limitations of the platform, Macromedia Director, that I used to program the project in, which didn’t allow bump mapping or shadowing. We couldn’t show the physical texture that the original paint had, and thus revealing the painter’s marks, clearly. But also, as we mentioned earlier, we didn’t want to create an effect to achieve this either. Once this work was completed in 2002 we exhibited it over a period of five years. [9]

By 2008 technology had moved on and so had our knowledge of 3D graphics, we now chose to use a new platform, Unity Technologies’ Unity3D, which allowed bump-mapping and was easier to use for our work. Graphics cards had and still are improving, enabling us to have more detail, larger textures. Processing power had increased allowing for more computational options. We started thinking about a new project where we would be able to take advantage of newer 3D graphics technologies, in hope that it would open up more possibilities in our work. We wanted to maintain our painterly approach. But likewise, we wanted to be more ambitious.

Developing a new project

Thus we began an eighteen month long development period for our second project. New paintings were created and new programming techniques and ideas were explored. The resulting work has had several permutations since its premiere in March 2009, Jurassic Light Years (Chelsea Art Museum, New York, USA, March 2009), Light Years: Jurassic Coast (Electronic Visualisation and the Arts Conference, London, July 2009) and Light Years: Coast (Lighthouse Centre the Arts, Poole, UK, June 2010).

In terms of ambition, we were keen to expand the scale of the project, increasing the size of the landscape that it deals with, in order to create a larger experience. Whereas Purbeck Light Years dealt with a mile radius around Corfe Castle, Jurassic Light Years dealt with four miles of coastline and Light Years: Coast with fifteen miles of coast. The terrain we used was more detailed, and we were now using two-metre resolution Lidar data to create the landscape, instead of lower resolution satellite data. This meant that rough forms of trees and buildings were visible and could be lit-up and shadowed. These new projects featured the added dimension of English Channel, leading to a quest to explore different ways to represent water. The idea of moving waves and reflections was attractive, as it meant that the reflections distorted the imagery above the water. I used algorithms that fairly realistically mimicked real ocean waves. These waves were connected to an Internet data feed of local weather information. The wave heights and tides were affected directly by the information fed into it, which provided an interesting direct link to the real world.

In Jurassic Light Years, I improved the rain system from what I had used in Purbeck Light Years by using particles to simulate rain. This gave a more dramatic representation of rain, but too many particles affected the frame rate. Keeping a high frame rate was an important factor in making the experience immersive, as a stuttering image, was uncomfortable to watch. There was considerably more landscape mesh detail as well, and bump mapping on the cutouts of paintings, which showed the texture that appears in paint strokes of the original paintings.

Tensions with technology

I spent a lot of time developing these techniques, and this led to a tension. Not so much in what the technology could do, but rather in the result of what it enabled me to do.

As I have mentioned, we had taken on an increasingly ambitious project by representing an experience of a much larger area of landscape. Our painterly concept, of using overlapping planes to create a tension between the 2D and the 3D required approximately 50 planes in Purbeck Light Years but to achieve the same density of planes in Jurassic Light Years, we would have had to use thousands. Although this was technically possible, it was not artistically feasible. As a result of this issue we ended up with a much sparser landscape in Jurassic Light Years.

Light Years: Coast, set in an even larger area of land, had planes that rarely intersected each other, but showed larger sections of paintings, roughly aligning with their viewpoint in the landscape. This meant that the work was part interpreted simulation and part painterly experience. The viewer took a virtual boat ride along the coast, but the visual features were elements of paint suggesting the geology of the land.

Another point about these newer projects is to do with interactivity. They are deliberately passive, less co-creative experiences. This was a decision we made based on audience observation of Purbeck Light Years; that many people just stood and watched, instead of using the joystick to interact with the piece. The more passive viewing approach meant that we directed the viewer more. However, this was more akin to being on a boat journey. The co-creative element was reduced and the opportunity to question the two-dimensions versus three-dimensions happened less frequently, but I believe with more anticipation.

Overall Light Years: Coast has developed into a virtual landscape experience, where the weather is more clearly defined than in Purbeck Light Years, where there is greater contrast between calm and stormy conditions, enhancing the emotional impact.

Conclusion

The improving technology, and my own learning on how to harness it through programming methods such as Shader programming, have had a prominent influence on the Light Years: Coast work. Technology, or rather the excitement and opportunities of it, has affected our painterly approach, by drawing me into wanting to do something bigger (and hence better). The result was, however, different to our original project, it was more ‘linear’ on the whole. Technology had broadened and challenged our work and also created new possibilities.

The steep technical learning curve I went through has left me with a new knowledge, and control of the medium, that we as Light Years Projects can now apply to new work. By manipulating the language of 3D used in computer games, we can keep true to our original aims of applying a painterly approach to a virtual artwork, to a digital landscape painting. Our next landscape project does not need to be bigger, in terms of the landscape it explores, but will be more focused, and enable us to use the knowledge we have gained of the technology to apply a painterly approach to the work.

References and Notes: 
  1. R. Mayer, A dictionary of Art Terms and Techniques (London: A. & C. Black Ltd, 1969), 276.
  2. H. Wöfflin, Principles of Art History (Dover Publications: New York, 1932).
  3. Jeremy Gardiner, “Light Years: Jurassic Coast – An immersive 3D landscape project” (paper presented at  Electronic Visualisation and the Arts London 2009).
  4. J. Hays and I. Essa, “Image and Video Based Painterly Animation” (paper presented at 3rd International Symposium on Non-Photorealistic Animation and Rendering, 2004, pp113­­–120).
  5. A. Hertzmann and K. Perlin, “Painterly Rendering for Video and Interaction” (paper presented at 1st International Symposium on Non-Photorealistic Animation and Rendering, 2000, 7–12).
  6. L. Lin, H. Zeng et al. “Painterly animation using video semantics and feature correspondence” (paper presented at 8th International Symposium on Non-Photorealistic Animation and Rendering 2010, NPAR '10).
  7. S. Bhattacharjee, P. Narayana (paper presented at Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008, 568-575).
  8. L. Candy, “Co-Creativity in Interactive Digital Art,” (paper presented at Consciousness Reframed, Fourth International CAiiA-STAR Research Conference 2002).
  9. Anthony Head and Jeremy Gardiner, “Light Years Projects Timeline,” accessed 20th June 2011, http://www.lightyearsprojects.org/?p=198 (accessed September 2011).