automated image analysis, cultural analytics, demoscene, visual effects
Canan Hastik
canan [a] hastik.de
Researcher
University of Applied Sciences Darmstadt
Viittaaminen / How to cite: Hastik, Canan. 2014. ”Demo Age: New Views”. WiderScreen 17 (1-2). http://widerscreen.fi/numerot/2014-1-2/demo-age-new-views/
How would you explain demoscene artworks? One way is an euphoric individual statement, a personal story linked to a demo art production with an enormous amount of detailed technical background information. Another way is to browse through pouet.net or demozoo.org and show some representative works. For a clearer understanding of demo artworks both are necessary: the overview with as much contextual information as possible and some explicit details. This is how demoscene material should be accessible and explorable: in different levels of detail and through different views. The MEGA Demoage Project is dedicated to providing as many views as possible on the scene, to make the sociological and technological aspects as well as the artworks themselves searchable and comprehensible. This article wants to present some new distant as well as close views on demo art material to support communication about demoscene phenomena. Focusing on the aesthetic aspects of demos, a qualitative analysis has been done by examining visual effects, sequences, composition, and scene changes as well as other attributes like brightness and colour usage in demos.
Intro
The demoscene is a historically grown digital sub-cultural art scene producing highly elaborated real-time computer generated applications along with music and graphics. Demoscene productions can be seen as computer generated audiovisual media art. Today there are over 53.452 known demoscene productions (status 30th of December 2013), according to Pouet.net. The original source code of demoscene artworks is rarely available. Instead, different versions can be downloaded as executable binary or video files based on different compression algorithms. Particularly the demoscene is an example of networked collaborative artistic creation processes on the Internet taking advantage of technological opportunities, constantly producing new genres and promoting the development of trends. There are different types of demos. These types correspond to a sub-classification and represent specific characteristics of a demo like hardware platform characteristics, size and aesthetics.
Demos are representative for scene specific aesthetics, technical culture and the diverse usage of platforms, and they rarely communicate stories or messages (Reunanen 2010). The almost overwhelming variety of demo artworks illustrates the evolution of design principles and techniques. Until now demo artworks have not been thematically classified yet. It can be determined that demo productions are often inspired by science fiction, fantasy and comics (Borzyskowski 2000). Especially visual effects together with music shape the aesthetics of demo artworks (Botz 2011). The development of new elaborate visually and technically impressive effects depends on the underlying hardware and is one important technique for sceners. As a scene expert you have to know as many demos as possible and know at least the most important demos to get a feeling about the development of styles and aesthetics in the past.
There is no effect database you can access, and communication about demo effects is not simple. The term “effect” is used arbitrarily and a comprehensive definition of the term “demo effect” is still missing. One reason might be that digital art is still a young field and any effort to define terminology that is used within this discipline risks rapid obsolescence, is rather ambiguous or context dependent due to the fact that digital art is a mesh of many different disciplines like traditional animation, computer animation, image processing, photography, computer science, electronics, physics just to name few (Brinkmann, 1999). Another reason might be that certain technical platforms have their specific aesthetic dimensions (Monfort and Bogost 2009) which need to be opened up. It is well known that when artists develop their ideas further current definitions are often being replaced (Berger 1987), but the recognition of the particular structure of objects in visual arts is of fundamental importance. So how could someone, for example, find a certain effect in demos?
The research presented here contributes to the development of a multifaceted access to domain knowledge about the demoscene. Therefore a classical approach for generating access to content has been used and tested on demos. The goal is to develop an automated or semi-automated subject analysis methodology for demos to make scene knowledge easily accessible in the future.
Manual Subject Analysis
The classic method for generating a close view on the content of any media is manual subject analysis. The transcription in form of a protocol to transfer audiovisual content in a textual form is also necessary in film analysis. In particular, the listing of film sequences based on a timeline is used to describe the structure of a film (Faulstich 2004). Media objects are described or classified by representative terms or short annotations. These key terms form a controlled vocabulary for use in bibliographic records. For demoscene effects a collection of index terms exists (Reunanen 2010). In the field of digital composition a glossary is also available (Brinkmann 1999). With these terms typical notions of demos can be explained and described. Manual subject analysis needs to be done either by experts or professionals in the field of new media production. Ideally, artists index their demoscene productions themselves.
Within a sample case study a selection of 26 demos has been manually analysed and indexed by scene experts and games & animation students (Table 1). The visual effects of six Atari VCS demos have been identified by two sceners. The other fourteen Atari VCS demos have been analysed by students. Furthermore, six demos on different platforms have been described by students. In combination with an index term a screenshot and the exact timestamp have been documented.
|
Expert or Professional |
Demo |
|
Svolly |
doctor, tom/jones, Minute and a Bit, Sicktro |
|
JAC! |
BeamRacer, ISO |
|
Students |
hectoByte, Saigon, Tricade, Trilobit, 2600, Noice, core, Reunaviiva, Lasertime, drip.bin, Bell hopper, Sphaera Stellarum, Stella Lives! |
|
Human Traffic, Batman Forver, Chaos Theory, numb res, MGC 2011, Uncovering Static |
Table 1. Selection of demos analysed by experts and professionals.
This case study shows diverse, heterogeneous and partially incorrect results. Some demos were analysed frame by frame instead of based on the individual runtime. Particularly the students’ results do not include appropriate index terms used to describe effects and effect compositions. Nevertheless, this case study shows three levels of how effects can and should be described. Integrating screenshots of effects exemplifies the technical term or description and supports comprehension. The time-exact definition of terms and screenshots is a major aspect in indexing demos. The underlying idea is to guarantee direct access to the designated effect. Therefore, the resource needs to be clearly identifiable and thus be archived because for example a single time shift in relation to wrong frame rates and runtime will result in incorrect indexes. In Figure 1 the screenshot shows one demo effect routine which can be described using one single term.
Several effects are often combined like in Figure 2. In this case, more than one key term is necessary to describe a certain effect composition.
For more complex effect compositions a short annotation is necessary in order to describe demo graphics like in Figure 3.
Manual subject analysis offers a close view on demoscene material. The subject terms are technical terms used in contemporary media art production. This makes these terms not generally understandable. Short annotations are useful to describe complex demo effects. Sound and music, too, should be considered in textual descriptions because only the combination of visual effects and music shapes the overall atmosphere of a demo. Unfortunately, this method is much too extensive when we think about retrospectively analysing all demos published so far. Experts or, ideally, producers need to support this approach to enable access to demos especially regarding the effects used. Maybe a crowd sourcing project like Metadata Games (Metadata 2014) could be an option, but how can this elaborate process be accelerated, simplified and maybe even automated?
Automatic Indexing
Automatic indexing of large scale born-digital material, especially images, audiovisual collections or user-generated content, has emerged as a new field of study, and is now predominantly occupied by commercial companies, such as Google (Lazer 2009). Primary sources are provided in research infrastructures for exploratory analysis and empirical research of multimedia, complex and user-generated content, dialogues and interpretations (Prescott 2012). The vast amount of cultural content like conversations, opinions and other cultural activities offer new opportunities towards a better understanding of cultural processes, models and the past. Therefore, traditional paradigms need to be expanded to make new approaches (Berry 2011). But there is still no solution to the automated indexing of computer-generated art like demo artwork.
Visualisation takes a key role in accessing big cultural data and helps to question specific cultural phenomena. There are already techniques to analyse massive visual collections of still or moving images. Especially the time-lapse technique for still images, first developed by Georges Méliés in 1897, and the concept of slow motion for moving images, invented by August Musger in 1904, are indispensable. Both techniques are changing the time scale of the underlying material and result in storytelling of another dimension (Becker 2012). While time-lapse offers new perspectives through a series of still images and makes slow changes, variations and modifications visible by increasing the frequency of images, slow motion is used for fast moving images to recapture key moments or to make certain phenomena visible and document them. As slow motion makes imperceptible changes visible the method can be described as a microscopic examination of time (Storfer 1911). It seems reasonable to convert demo videos into single images to make those short audiovisual effect sequences easier to process.
Certain tools for the exploratory analysis of massive cultural datasets have been developed since 2007 by Software Studies Initiative (Lab 2007), with example projects in the field of Manga books, painting or user-created art. So why not use the software ImageJ/ImagePlot by Lev Manovich to visualize demos and generate new views for finding basic approaches to the automated analysis of demos in the future? The analysis of digital images by Lev Manovich (Manovich 2007) provides possibilities to analyse different images and image sequences. Image sequences in the form of videos need to be stripped down into single images.
Demos are generally based on different frame rates (fps), such as 25 fps, 30 fps, 50 fps or even 60 fps. Therefore, each demo needs to be processed individually to ensure that the frame count of the resource is the same as the volume of all single images. As sources, if available, youtube videos published by the original authors and alternatively emulator recordings were used. Commercial software like Adobe After Effects CS6 (Adobe 2014) and free tools like VLC media player (VideoLan 2014) or FFmpeg (ffmpeg 2014) allow to convert video files into a sequence of single images or key frames. Key frames are images within a video stream that exist as a whole instead of a delta (changed areas only) of the last image and help to identify scenes within a movie. The rate of key frames within a video stream depends on the codec, bitrate of the stream, and other factors.
For complete image sequences, the higher the frame rate of a demo is, the more single images have been extracted. In this process some demos consist of several thousand images. These single images were reduced to a manageable size considering the original aspect ratio and have been imported into ImageJ. In the following process montages, orthographic projections and histograms were generated using ImageJ and ImagePlot.
Montage
ImageJ provides different possibilities to analyse images. One basic way for an analysis is to create montages of the image sequence. Each frame can be arranged in a chronologic order. The montage of a demo resembles a storyboard that displays the general composition of each demo. Instead of having access to 35 manually indexed or annotated screenshots, these montages make it is possible to capture hundreds or thousands single images of the demo at once.
The example in Figure 4 shows a montage of the Atari VCS demo Doctor. This montage allows identifying different sections, such as text scenes, used pictures and effects. The montage clearly shows the overall usage of shapes as well as colour settings. Breaks are represented by black images. In addition, the montage gives an overview of the demo concept and the time used for credits and greetings in relation to the total runtime. It can be seen how often certain scenes have been reused.
The montage of Stella Lives!, presented in Figure 5, shows how colourful this demo is. The saturation and colour gradients are changing over the total length of the demo. Especially the yellow-brown tone present in almost every picture is obvious. It also becomes clear that there are different sections in the beginning, which are introduced or separated by text scenes, and in the end there are different effects directly after each other as in a show reel. The relative length of the scenes and the transitions between scenes are recognizable.
Another montage view of Stella Lives!, as shown in Figure 6, displays only the key frames of the different scenes within the demo. The montage is reduced to a minimum of relevant frames. The composition of the demo gets transparent and understandable at a glance. The repetitions of scenes are getting clear. In addition, the arrangement consisting of effect scenes mostly followed by a textual or credit screen becomes obvious.
The montage of Beam Racer in Figure 7 shows no certain pattern of composition compared to Stella Lives! It is recognizable that the demo is based on one single scene with a series of similar effects. The demo shows a massive presence of so-called “coder colours”, meaning the programmer of the demo probably chose the colour scheme and visuals himself instead of relying on a graphics artist. This often results in very colourful screens using the whole RGB spectrum. Most of the textual statements displayed in the demo can also be captured from the montage.
The montage of Saigon, as presented in Figure 8, includes 3540 images. It illustrates how the screen animations are changing frame by frame. Slow animations with little movement are hardly traceable in this montage, whereas fast animations are clearly noticeable. Also text, blending effects and repeats become visible. It can be recognized that this demo is more monochromatic than, for example, Beam Racer. All the scenes, transitions and effects seem well balanced colour-wise while not appearing too colourful. The scenes are well determinable.
The montage of drip.bin in Figure 9 represents the individual atmosphere of the demo. Fast movements, turnarounds and gradient changes of the cascading graphical elements increase the psychedelic atmosphere of the demo. In the middle of the demo the interval of colour changes is shorter and the gradient changes up to 90 degrees. The montage is like a rainbow-coloured kaleidoscopic mosaic. Repeating sinus lines within an always similar colour space can be noticed. The whole demo is based on frequent and iterative use of different colours. It seems that the same program code runs in a permanent loop with just a slight colour shift. At the beginning and the end the composition changes slightly.
Through the montage of Human Traffic, shown in Figure 10, a total of ten different scenes can be identified. Blue is the dominating colour in this demo, and there are only one red and two yellow-green scenes in addition. The montage indicates that the demo is quite sequential and follows a fixed rhythm. There are no big variations to the screen action.
In summary, it can be said that the montage of a demo image sequence shows the general composition of the demo artwork. Patterns, such as scenes, text, blending effects, saturation, colour gradient and changes as well as graphical elements (e.g. sinus lines) and pictures can be identified and analysed in detail by zooming into the montage. Differences and variations can be noticed easily.
Orthographic Projection
Much more detail about the spatial relationships among objects within scenes can be extracted by visualizing demo material as sliced frames, which are then put together in an orthogonal XZ- or YZ-projection view. At first glance, this visualisation seems to be confusing. Orthographic projections are mainly used to represent three-dimensional objects in two dimensions and are useful for identifying how objects are building up and moving horizontally and vertically. An orthogonal projection gives an interesting perspective of the demo as you can clearly see changes of settings and transformations in relation to time and space. The orthogonal XZ-view uses one horizontal (X) slice from the centre of each image and adds every line underneath the last one in a chronological order along the Y-axis (Z). The YZ-view uses vertical lines (Y) from the source images to display the time (Z) along the X-axis.
The orthogonal XZ-view of Human Traffic, as presented in Figure 11, gives a good impression of each section of the demo. Again, each scene becomes clearly identifiable. It is traceable how the pictures within a demo are building up horizontally line by line. In addition, with the orthogonal XZ-view, the scrolling of 2D images can be identified.
Figure 12 pictures the orthogonal XZ-view of numb res. The projection shows how particle elements are moved over the screen and then disappear. Especially the transitions of the organic objects become visible in this view. It becomes very obvious that the whole demo consists of particle effects. The movement of the particles along the X-axis can be recognized, but it is not possible to identify text or objects.
The orthogonal YZ-projection, shown in Figure 13, reveals how long each scene takes. White particles on a black background a building up to images and text. Moreover, the text is clearly readable and the images are recognizable. This indicates that objects are built-up in a horizontal way.
In summary, the orthogonal views give an interesting distant view of the demo composition. Scrolling through time also reveals the changes in the different settings at different points of the picture. With these views, demos can be analysed from a distance without watching the whole demo over and over again. Patterns, particular characteristics, styles and other technical features are recognizable.
The special Z-Projection Sum Slice function shows which areas of the screen are mainly used for the demo. In the example case, Saigon, the centre of the screen is used most frequently. The scrolling text element uses the entire width of the screen. In particular, the lettering catches the eye. It is repeatedly used in the demo and functions as a watermark across the entire presentation (Figure 14).
Other Measurements
Besides methods, such as montage and projection, which are proven to be quite useful for a visualisation of demo features, there are much more possibilities which can be reasonable for individual analyses. Some functions calculate numeric results on properties, such as brightness or colour changes, or give values for the complexity of an image. These numeric results can be used to calculate and display colour density or grayscale calibration of demos in histograms.
Colour histograms visualize the used colour spectrum and settings within a demo. With the median function the average brightness and contrast values can be determined. In live mode, for example, the demo can be interactively analysed in several dimensions and with regards to motion.
Analysing Human Traffic regarding the brightness over time, as shown in Figure 15, indicates an organic gradient of colour and brightness with smooth transformations. Some scene changes are very dark. The highlights of the demo composition and the density of used effects become clearly visible with this view.
Summary and Results
Manual subject analyses of demo effects, based on a predefined corpus of descriptors using time stamps and screenshots, are a common method to enable access to media content and to support communication about media objects. It has to be pointed out that meaningful results are caused by the combination of controlled terminology with time-exact images. Descriptions vary between one or more key terms and short annotations explaining compositions. One advantage is that sound can be included in the analysis. A disadvantage is that the analysis is reduced to the perception and knowledge of the analyst. This can lead to misinterpretations and deficits. Furthermore, this method is far too time-consuming and should only be executed by experts.
At the moment there is no way to automatically analyse, extract, annotate and describe relevant features from demos. But automated image montage tools, such as ImageJ, offer promising functions to analyse demos frame by frame with regard to time, motion, composition, colour and more. Visualising these aspects provides a good overview about the general characteristics of a demo.
To sum it up, demos can be accessed quite efficiently by using automated image analysis. When analysing sequences of effects used in demos the most helpful functions of ImageJ are montage, orthogonal projection and a gradient histogram of colour and brightness. It is no longer necessary to watch the demo or video again and again to capture as much information as possible. The automated visualisation shows key frames, coherence, repetitions and sequences explicitly. Especially for longer demos, these functions are meaningful.
However, there are a number of issues when dealing with huge amounts of images. First of all, the process needs more preparation, because it is not possible to import video material directly into ImageJ and sound cannot be processed at all. Secondly, even if demos usually have a runtime of approximately three minutes, large amounts of data must be managed. Similar to the problem that the eye has limits when viewing video sequences over time, there are also restrictions in the perception of single images. As shown in the case of demos that feature light effects and smooth transitions, the montage visualization does not work very well. In particular, small objects as well as slight and irregular movements become better visible when watching video sequences.
Still, to understand these complex visualisations some context information as well as knowledge about the technical aspects of a demo and the underlying data is necessary. In addition to the form or visual characteristics, as Lev Manovich’s concept of automatic image analysis does, the production and presentation practices as well as the socio-cultural context of demoscene artwork are also relevant.
In conclusion, it can be said both approaches provide only conditionally meaningful results by generating distant and close views on demos. But both methods can complement each other perfectly. One possible use case could be to analyse scenes in demos and extract key frames automatically and then manually describe these key frames using index terms.
Anyhow, the visualisation of massive cultural data sets provides a general overview about the underlying material and presents new views which help us to question cultural phenomena. Manual subject analyses can be seen as high quality annotation of specific material and offers the possibility to research and compare demos regarding defined aesthetic aspects. Experimenting with automatic frame-based analyses is important for finding attributes and algorithms to support the development of future fully-automated analysis tools.
Still, neither of these methods is able to carry the whole atmosphere contained in a demo. To get the right impression which is mainly based on sound and tempo of a demo it is essential to watch that demo, preferably on the original hardware, not as a video.
Future Work
This paper is the first attempt at using ImageJ for the automated analysis of demos. It should be inspiring for others, from artists to analysts. A more elaborate analysis on the results will follow soon – there is plenty more that can be said about the projections.
In the context of the MEGA Demoage Project an ontology describing the production and presentation practices as well as socio-cultural structures of the demoscene culture has been developed (Hastik 2013). This model functions as a metadata schema for integrating and merging different heterogeneous sources which are relevant for the demoscene. It offers detailed terminology and can be extended with other glossaries or classifications. Furthermore, all objects within this ontology can be described in detail including demo effects, either processed with ImageJ or annotated manually.
Acknowledgements
I would like to express my special thanks to the games and animation students at University of Applied Sciences Darmstadt from the Technology and Society practical seminar in summer term 2013: Björn Biling, Stefan Horn, Pascal Bogensprenger, Lars Möller, Jörn Dürig, Sarah Schaack, Nicola Sebastian Pirker, Hendrik Großkurth, Felica Handelmann, Bianca Galloy, and the Atari sceners Svolli and Jac! who participated in this analysis.
References
Adobe After Effects. 2014. “Animation Software.” Accessed March 06, 2014. http://www.adobe.com/de/products/aftereffects.html.
Berger, René. 1987. “Der Künstler und die Maschine: Technologischer Wandel und neue ästhetische Dimension.” In Synthesis. Visual Arts in the Electronic Culture. Offenbach: Hochschule für Gestaltung.
Berry, David M. 2011. “The Computational Turn: Thinking About the Digital Humanities.” Culture Machine. The Digital Humanities: Beyond Computing 2, pp. 1–22. Accessed March 05, 2014. http://www.culturemachine.net/index.php/cm/article/view/440/470.
Borzyskowski, George. 2000. ”The Hacker Demo Scene and Its Cultural Artifacts.” Accessed March 05, 2014. http://www.scheib.net/play/demos/what/borzyskowski.
Botz, Daniel. 2011. Kunst, Code und Maschine: Die Ästhetik der Computer-Demoszene. Bielefeld: Transcript Verlag.
Brinkmann, Ron. 1999. The Art and Sciecne of Digital Compositing. San Francisco: Morgan Kaufmann.
demozoo.org. 2013. “The new demoscene archive.” Accessed March 05, 2014. http://www.demozoo.org.
Faulstich, Werner. 2004. Einführung in die Filmanalyse. Tübingen: G. Narr-Verlag.
ffmpeg 2014. “Cross-platform solution to record, convert and stream audio and video.” Accessed March 05, 2014. http://www.ffmpeg.org.
Hastik, Canan, Steinmetz, Arnd, and Thull, Bernhard. 2013. “Ontology based Framework for Real-Time Audiovisual Art.” In 79th IFLA World Library and Information Congress, Singapore. Accessed March 06, 2014. http://library.ifla.org/87.
Lab. Software Studies Initiative. 2007. “Software Studies is a new research paradigm in the humanities and media studies.” Accessed March 05, 2014. http://softwarestudies.com.
Lazer, David, Pentland, Alex, Adamic, Lada, Aral, Sinan, Barabási, Albert-László, Brewer, Devon, Christakis, Nicholas, Contractor, Noshir, Fowler, James, Gutmann, Myron, Jebara, Tony, King, Gary, Macy, Michael, Roy, Deb, and van Alstyne, Marshall. 2009. “Computational Social Science.” Science Magazine 323(5915), pp. 721–723. DOI: 10.1126/science.1167742.
Manovich, Lev. 2007. “Founder of Software Studies Initiative.” Accessed March 05, 2014. http://www.manovich.net/index.php.
MEGA e.V. 2012. “MEGA Museum of Electronic Games & Art.” Accessed March 05, 2014. http://www.m-e-g-a.org.
Metadata Games. 2014. “A digital gaming platform for gathering data on photo, audio, and moving image artifacts.” Accessed March 05, 2014. http://www.metadatagames.org.
Monfort, Nick, and Bogost, Ian. 2009. Racing the Beam: The Atari Video Computer System. Cambridge (MA): MIT Press.
Pouet.net. 2000. “Your online demoscene resource.” Accessed March 05, 2014. http://www.pouet.net.
Prescott, Andrew. 2012. “Consumers, Creators or Commentators? Problems of Audience and Mission in the Digital Humanities.” Arts and Humanities in Higher Education: An International Journal of Theory, Research and Practice 11(1–2), pp. 61–75. Accessed March 05, 2014. doi: 10.1177/1474022211428215.
Reunanen, Markku. 2010. Computer Demos – What Makes Them Tick? Espoo: Aalto University.
Storfer, Adolf Josef. 1911. “Die Microskopierung der Zeit.” Der Kinematograph 242, p. 5.
VideoLan. 2014. “Cross-platform media player and streaming media server.” Accessed March 05, 2014. http://www.videolan.org/vlc.