MA final project research

0 The Research Book

The Research Book
The following is an excerpt from the research book to my final project Crash – A stock trader’s desk at the MA Communication Design at Central Saint Martins. Subsequent to the introduction follows chapter one that pictures particularly the theoretical research including my thoughts and hypotheses on Screen Realities, the control room, the terminology and the financial crisis whilst the second chapter deals with the early (video) tests, the final concept and the single steps (the visual representation of the stock exchange data, the graphic user interface, the programming, audio and video) to the finished multimedia piece, the actual desk and the interactive installation. Additionally to the book is the brief third chapter with pictures of the desk at the final show at Candid Gallery and selected video layers.
Many thanks go to Alekz Keck for the programming, Phillipp and Florian Koller for the soundtrack and moreover to Anne Katharina Schäfer, Peter Werner, Melanie Seyer, Hannes Mandel, Andreas Schepers at ESA, the mates at Digital Media, the DAAD, the German Academic Exchange Service and to my tutors Sadhna Jain and Rob Rae. The whole work is licensed under Creative Commons (BY, NC, SA).

1 Research
1.1 This life is not a movie, or maybe

Stills of my installation for the MACD interim show
The subject of my final project at Central Saint Martins is what I would call Screen Realities. Primarily it addresses the electronic screens that had been invented with television in the early 20s of the last century and that are dominating our perception of the world nowadays (through the tele, monitors, projections and mobile devices) and therewith the realities these electronic screens are creating and how they are perceived by the recipient.

In conclusion of my research, moving images break down into four categories: fiction, faction, simulation and operational images. The term Operational Images relates to the German filmmaker Harun Farocki who himself references to Jean-Luc Goddard. Operational Images are images without any mythologic charge. Farocki correlated the term with the quest of French philosopher Roland Barthes for something similar in language. Figurative examples of operative images are the guidance cameras on the tip of autonomous bombs capturing highly engineered images whose only purpose is technological. Men are inspecting these images merely to check the mode of operation of its parent machine. In this manner these images hold an intense functional aesthetic.

Screen Realities is a very blurry term by intention. The hypothesis is that the boundaries between the categories of moving images mentioned above are tumbling down and the different realities are increasingly merging. The main cause is the Internet. Its anarchic hierarchies and the overall democratic structure also means an equality of the different images. These forms of image taking are emancipating from their primary purpose into a universal language where all the various forms become part of our common memory.

The place where all those Screen Realities are collected is the control room – the modern editing room. These rooms are everywhere nowadays. They are like massive routers. I examined one of those huge control facilities on a trip to the ESA (European Space Agency) and their ESOC (European Space Operations Centre). On-site I was briefed about the functionality of the main and the other, smaller, control rooms that monitor the several satellites in space. Consequential the form of my final project will be an abstract recreation of a control room. The aesthetic will make use of collage techniques as a reference to pop art which always took parts of reality and reassembled them to emphasize and criticize. The work itself will examine the importance of the crash for shifts in the mass media, referencing Paul Virillio (“There is no technical invention without accidents. Each time a technology is invented, a technology of transport, of transmission, or of information, a specific accident is born.”) and Guy Debord (“In the spectacle, one part of the world represents itself to the world and is superior to it.“).

The crash is both, the witness of technological and social progress, and the culmination of the spectacle and as such the evolutionary moment, the mutation, of mass media. The same mass media that tend to over-aestheticise the crash by converging the four Screen Realities.

1.2 Screen Realities

The upper chart contextualizes the Screen Realities with theories that influenced my research, as well as with other concepts of reality, cultural techniques and associated ideas such as the increased meaning of the control room. Some of the pictured elements haven‘t had the same attention as the control room. But as they did influence my thinking they should be mentioned. Jean-Luc Godard is set up close to Harun Farocki, who was mentioned at the beginning of this research book. Even though Godard was a mostly fictional film maker he was particularly interested in the ability of film to visualize theoretical issues. Beside his own attempts Godard was very critical with the medium film and it‘s capacity for abstraction. However since the early days of film there always were film makers, mostly Russians, who tried to express theory visible in film not through words but through the medium itself. Harun Farocki shares this approach with Godard even though he montages other film sources such as the – as he defines them – Operational Images. The theoretical (and practical) centre of their work is the cutting room. There they uncover theoretical thinking in film through the association and dissociation of different (moving) image sources such as painting and photography.

Constructivism, Post-Modernism and later Pop Art became quite important recyclers of reality. Artists of these movements took parts of the reality and magnified, removed and reconstructed them. They sampled reality and duplicated the banality of everyday life in order to recontextualize its meaning. Roland Barthes wrote in his essay »The Death of the Author« in the spirit of Post-Moderism that originality is created through the rearrangement of banal, already known elements. Several years earlier the artist Max Ernst stated about the collage, the predominant technique of the later era of Post-Modernism and Pop Art especially, that it is the systematic exploitation of the accidentally or artificially provoked encounter of two or more foreign realities on a seemingly incongruous level.

1.3 The Control Room

ESA Control Rooms
At least since the blueprints of Jeremy Bentham it is known that the view (and it’s direction) is a representation of hierarchies as well. One who sees much rules much. In sociology this coherence had been widely explored, e.g., by Michel Foucault, yet the panoptic representation of moving images as an progressing process has far reaching and less investigated media-related effects.

The difference between the Screen Realities and other more comprehensive reality models such as the conceptions of realities by system theorist Niklas Luhmann is that Screen Realities are limited through technical restrictions. Lenses, image sensors, bandwith, screen resolutions – Screen Realities are a priori a restricted and limited reality. On top of these technical limitations there is the selectivity and contingency of the human operator. He‘s choosing the perspective, white balance, colour mode, depth of field, focus and in the end the edit and or the cut. (Some might call managing these settings art already.)

With the increasing amount of moving image data in these modern days, with CCTV cameras (a perverted variation of Operational Images) all over the cities e.g., the human influence on the moving image decreases. Into the shoes of the human operator steps a modern technical facility. Not only a device, but an arrangement, an architecture of devices: The control room. It‘s a necessity to manage the flood of moving images that‘s growing proportionally. 24/7 streaming replaces tapes and harddrives. And the control room replaces the editing room. Control rooms are being erected almost everywhere. The New York Stock Exchange (NYSE) for example is nothing else than a vast control room. The only purpose of the control room is the impossible: The control of reality. A wish that comes from the fear of losing public order and power at last.

In order to research modern digital control rooms I went on a field trip to the European Space Agency (ESA) in Darmstadt, Germany. Their European Space Operations Centre (ESOC) has one of the largest control rooms of Europe if not the world. My interest was not only in the process of controlling but also in the screens and the related technologies. A key questions involved the several ways users are interacting with multiple screens and what architectural and design concepts are appropriate to visualize vast amounts of abstract data.

The trip to the European Space Agency was somewhat disenchanting. I learned that they hardly ever work with moving images, but with software generated representations of reality only. The enormous ammount of columns of figures and numbers representiong positions and speed was impressing but disappointing for my research. (Consequently you could think about extending the Screen Realities with another reality: The Software Reality, which would be close to the Operational Images and simulation.) On my desperate quest for moving images one of the technicians told me the story of the new docking system on board of the new ATV module, a space transport vehicle. It docks completely automatically with laser technology. No video technology was meant to be used until the Russians intervened and insisted on an extra camera plus crosslines on the ISS. They simply wanted to see the whole docking process. Nobody at ESA had thought of that before.

Also my timing was rather bad. The day before I arrived in Darmstadt they had an unscheduled start mission control test. The day after I left two space vehicles, an Iridium-US-satellite and an abandoned russian vessel crashed some thousand miles above the Pacific – teaching me quite tangibly the contingency and incalculability of crashes. (More pictures on Flickr.)

1.4 Terminology

My initial reason for setting up a terminology was of linguistic nature. As non-english-native speaker I wanted to be sure not to err in the nomenclature. Especially the term crash has many synonyms with analogical meanings like collision, collapse or clash for example. Furthermore I wanted to clarify if the crash semantically defines a unique phenomenon and to what extent it differs from other notions such as Paul Virilios accident or the spectacle of Guy Debord. It turned out that the crash is on the interface of both mentioned phenomenons. It does has an uncontrollable destructive power, is an implemented part of any thinkable system and it also is quite attractive as an event to the mass media. (The coincidental and arbitrary nature of the crash may have done their part in its popularity.)

Caution is especially advised for the terminology. Way too often terms are being caved, manipulated and misused. A recent example is the terror term that is being hyped throughout worldwide media and whose alarming association is being misused politically to avoid discussions and replace arguments. It crops up that terms and their exact usage is necessary for a certain credibility. A credibility that also includes the phenomenon (or project) it is naming. Crash is a versatile term. It is not only being used in context with the unintentional cold deformation of different materials such as metal or conrete, it also finds its use in failing software programming, natural events such as an impact of an asteroid, and last but not least the holistic failure of the financial sector. This ambiguous ability is to a high extent suiting for a project that‘s reproducing or simulating a certain reality with the objective to emphasize and reveal a higher truth.

1.5 The Financial Crisis

Financial and economic crises are a particular kind of crash. The first financial crash that burned itself into the collective memory of mankind was the worldwide crisis in the late 1920s. In the decades following the financial crash had several smaller blooms, such as the Big Bang in 1986 which was preluded by sudden deregulation of financial market by the British government. Some financial experts say that these deregulations are one of the main reason for the ongoing crisis by the way.

The 1980s are generally seen as the decade of the rise of stock exchange and the financial wealth of its actors. Until today the image of a stock trader and successfull banker is shaped by fictional motion pictures of that time. »Wall Street« of director Oliver Stone, released one year after the London Big Bang, with Charlie Sheen as Bud Fox, a young Wall Street stockbroker in early 1980‘s New York with a strong desire to get to the top, being the most iconographic of all of them.

This decade startet a mimetic phenomenon of absurd wealth, of English shoes, Italian suits, private jets (as ironically described in Tom Wolfe‘s short story for the new Evening Standart called »The rich have feelings too…«) and German cars. The car by the way becomes a ambiguous metaphor for the crash these days. The obvious is it‘s accident and secondly with being dumped in all the oversea harbours all around the world the car became a symbol for the financial crash that is turning into a economic crisis. The automotive industry and the aerospace industry are both of this old, 1980s styled, capitalism (and in many cases both are producing their goods for both the military and the civil sector).

If the stock exchange as mentioned before is nothing but a control room and the desk of an individual banker or trader nothing else but a smaller modell of this huge architecture the question remains how they still can claim that no one saw the crash coming. It seems that there must be a certain disfunction between the way their control rooms display reality.

1.6 Case Studies

Harun Farocki – »Erkennen und verfolgen«, 2003 (International title: »War at a Distance«)
Primal Scream – »Kill All Hippies« music video 2000, by Julian Gibbs and Julian House (Intro)
»Anomaly« short film, 2003 by Noah Harris (Precursor)
Opening sequence for the motion picture »The Kingdom«, 2007 by Stephan Burle
»Digital Acoustic Cartography«, 2005 by Daniel Rothaug
»Iron Man« end sequence, 2008 by Danny Yount, Prologue Films
Harun Farocki didn‘t influence my theoretical research alone. His practical work been inspiring as well. It isn‘t about motion graphics or design, it‘s moreover his approach to the medium of the moving image itself. Farocki‘s work process is almost as writing a scientific book, starting from the research in archives all over the world to a narration that‘s customized to the medium. However he doesn‘t exclude modern technologies and does the editing meanwhile at home on a personal computer.

The »Kill All Hippies« music video for the schottish electro noise band Primal Scream by Intro is almost 10 years old by now and it‘s still ingenious regarding the usage of found footage but also the prudential combination of 3D-space and graphical elements. It‘s a great example of leaving things out.
»Anomaly« by Noah Harris of Precursor speaks graphically a similar language as the Primal Scream video. It extends this language with an elaborate and convincing information design. The extruded video footage is an interesting attempt with the deconstruction of video material. Despite their age both videos, »Kill All Hippies« and »Anomaly«, don‘t seem old, but rather timeless, which evidences their quality.

What impresses me the most about the opening sequence for the motion picture »The Kingdom« is the skilful combination of historic recordings and 2D-3D information design. Thereby Stephan Burle, the designer, uses accurately the free space (here a metaphor for the desert) in his composition to emphasize single elements such as the drill rigs. It’s a strong piece of work where charts turn into political messages.
The original purpose of Daniel Rothaug‘s pictures is, as the name »Digital Acoustic Cartography« indicates, of acoustic nature. Beyond that he uses the software Processing not only for informational processing but also for a remarkable way of deconstructing the (video) image.

»Iron Man«‘s end sequence by Prologue Films hasn‘t much to do with the prior examples, but for me it‘s one of the most compelling motion design pieces of the last two years. In the beginning it is graphically reduced to completely monochromatic outlines. These lines are a homage to comics and computerized blueprints at the same time. And additionally Prologue added advanced information graphics towards the end, similar to those in the movie, also produced by the very same company.

2 Work Process
2.1 Early Tests

Deconstruction of TV footage, DV PAL, 20 seconds each
Slight manipulation of documental footage, DV PAL 01:30 mins
Short film »The media history of crash« about the Hudson Miracle, 1280×800, 01:30 mins
Left: Extruded geographical data on a video, DV PAL, 20 seconds; Middle: Displacement map of the normalized data of a certain stock quote, still image; Right: The displacement map on actual video footage, DV PAL, 20 seconds
My very first experiments were about deconstructing the moving image itself. It seemed logical to me as it is – due to its technical specifications – the one thing that all Screen Realities have in common. Therefore I seperated the colour channels of some sample scenes in seperate layers, red, yellow and blue. Afterwards the material was meshed like a TV screen and scaled up until it was completely irrecognizable.

The second test was about the border area between the different Screen Realities. We know that the mass media tend to over-aestheticise even documentary images. With it they blur the edges between faction and fiction. This experiment was meant as a critical comment on this practice. I used documentary footage to modify little things in the picture, manipulating the message of the images. The modifications were supposed to be so slightly that the viewer doesn‘t recognise them in the first place but still feels somewhat uncomfortable and irritated watching it. To achieve such effekt, I enlarged the eyes of refugees by the amount of 50 percent for example.

For the third experiment I was concentrating on narrative structures. I set up a small media history of the crash, wish means I concentrated on the correlation between the disaster and the news coverage about it. The theory behind that was that the crash was something like the evolutionary trigger for the emerging of new media and formats. The example for my tentative short film was the ditching of a Boeing into the Hudson River in February 2009. Only three minutes after the crash the first picture was spread all over the world by a user of the micro-blogging service Twitter.

After that I dealt with the possibility of transfer data directly onto video footage. Therefore I took stock exchange data, normalized it with a mathematical formula that is also used for data processing with the Processing software, translated it into a graphical system and used that system as displacement map for the video footage.

2.2 Final Concept

After all of my attempts to conceptualize and create a pure motion graphic piece, particularly after I made the short film about the media history of crash, I realised that a explanatory video would actually receal and explain less than I wanted to. (And there also remained the risk of sounding too didactic.) I came to realize that trying to shape, to provide an experience through an interactive installation setting could be more in the sense of my research about realities and control (rooms). A non-linear, non-narrative project would not only achieve a higher immersion of the audience, it also would be more exciting as a visual experience. It would be more „realistic“ as well, since it would be dynamic and strikingly involve contingency. By recreating a certain situation of participation I aimed to explain way more about the greater scheme than with a linear moving image.
Having developed the terminology graphs I couldn‘t stop thinking of the ambiguity of the mechanical and the financial crash. With the increasing feeling of powerlessness (see also the interview in the appendix) towards the ongoing process of bailing out multi-billion-dollar companies along with the credit borrowers and home owners being somewhat blamed for the whole crisis. I decided to focus the content of my work on the financial crash. The one that was caused by a pervert system of a greedy minority and that‘s going to have economic impact on almost every family all around the globe.

Considering all this I developed the final concept of the stock trader‘s desk. A spatial installation that is an interactive and a motion graphic piece at the same time. The construction is to look like this: On a desk, that somehow is also a kind of control console, are two monitors and a operating element (a keyboard) installed. All together sculptures a working place of a stock trader. Thus the left screen displays a simulation of a trading software. With the keyboard the user/viewer checks 18 pre-selected stock quotes of some of the most wealthy worldwide operating companies. The right screen then shows elements of the real world. These elements, houses, cars, trains and planes for example are dynamically linked to the stock quotes on the left screen. If the quotes decrease the video track of the particular element plays forward towards its crash. Corresponding precisely with the stock quotes of the last 12 months, the video elements crash.

2.3 Visual Representation of the Stock Exchange Data

Unfortunately it is not possible to access live stock exchange data just like that. The sale of these data is a good business for the stock exchanges and affiliated companies. Nevertheless services as Google Finance and Yahoo Finance offer some reduced stock exchange informations in certain spreadsheets. Yahoo especially has quite recent quotes but they are difficult to embed into another software. That‘s why I decided to step „back“ to a closed period of time of the past year (still preserving the possibility of changing the programming of the software to use live data one day). It is important for me to use real stock quotes because they should contribute to the feeling of authenticity by the left screen.
To this purpose I collected the stock quotes of one year, from early March 2008 to early March 2009 (52 weeks in total) and processed them into XML-files that can be read by the software. In the beginning I had analysed 20 different stock quotes of some of the worldwide wealthiest companies. Due to performance reasons I had to reduce them to 18 however. Pretty soon the design and the arrangement of 18 quotes at the same time became a problem. Similar real applications usually present no more than three quotes at the same time. I started experimenting with different designs, used line and area, monochromatic and coloured charts. Besides I worked with different transparencies and blending modes, sorted the quotes regarding their values in different groups and aranged them again. It seemed to be impossible to create a well ordered overview. Then I switched back from experimenting with area charts to line charts, whose amplitude looks like a seismograph, and I realised that I was on the wrong quest. I realised I had to look not for order, but for chaos. Uncontrollability – would be a better fit. That‘s why I chose monochromatic line charts eventually.

2.4 Graphic User Interface (GUI)

The Graphic User Interface is basically split in two parts. The left third consists of buttons of the 18 quoted companies. These buttons include the companies‘ name, their stock exchange code, the recent quote, the date and the change. Originally the buttons should have been clickable. As most of the functions of the mouse had been similar to the keyboard and the keyboard as an input device is more direct and tangible, the mouse became redundant. A major part of the screen is occupied by the charts who impressively represent the chaotic quote changes. Initially the quotes in the charts should have been clickable as well (I liked to compare it with the scratching of vinyl discs) but it turned out to be technically impossible because of the to the connection to the 18 video tracks and their vast amount of data. However it is possible to switch between two different representations, line and area charts. Additionally to the huge chart window there is a second one on the bottom that displays the overall amount of traded capital.
In the beginning I wanted the software to look like its real examples, as a Windows98 application. After finishing several versions I realized that it would be more helpful to actually develop an interface that reminds the user of software he used before but still makes him feel to use something he couldn‘t use every day. The new look was meant to be more coolish and technical.
GUI version 0.3
GUI version 0.4
GUI version 0.5
GUI version 1.3 (final)

2.5 Programming

Luckily I could convince Alekz Keck to do the programming of the whole application. Alekz is mostly working with Flash and Action Script. For a long time now Flash isn‘t used for web applications alone. It more and more replaces even Director and is being used for interactive installations as well. A huge advantage of Flash is that it is working fairly well with video and databases and it also runs on almost every platform. What Alekz basically coded was both, the simulation of the stock trading software on the left screen (that‘s reading the stock quotes from the XML-files in real-time and calculates the charts from it) and also the videos on the right screen, that are linked to the stock quotes. In Flash both screens are set up as one single stage. As the whole application should have been operated with a keyboard without having all the possible functions of a normal keyboard, a new layout had to be made up. Basically it works like any other special keyboard that has extra key assignments.

The performance became a huge problem for the software development. Playing 18 video layers at the same time is a huge weight for almost every current personal computer. Thus even for the computer that ran the tests and will be used for the exhibtion, a Intel Quad-Core PC with 8 GB RAM. To increase the performance we did a lot of testing. At first the resolution of the videos was reduced to 640×400 pixels. Flash then scales the videos up to the needed resolution and overlays a mesh to cover potential visual distortions. In the beginning all the video was loaded into the Flash timeline as a PNG-sequence. The advantage was that the video (its speed e.g.) could be manipulated by the user while being played. But again performance issues forced us to chose another option: H.264 coded Quicktime movies. H.264 is quite a robust state of the art codec. The next problem was the way the 18 layers should have been composed. The initally used Alpha-channels turned out to cause big trouble issues and had to be cut out. We decided to use the blend mode (screen) instead. The result was fairly satisfying. The test computer played 12 clips without any problem and 18 clips showed only some seconds delay over the whole time of three minutes. That won‘t be a big issue because the clips will still be synchronised and the user will hardly become aware of the delay.

2.6 Audio

I wasn‘t looking for a traditional sound design but for something distinctive. That was the reason why I asked Philipp Koller and Florian Koller to compose the soundtrack. Both are more musicians than sound designers. Philipp plays in the Berlin-based band Ter Haar and Florian is the bassist at SDNMT. I‘m a big fan of their mostly instrumental progressive rock and i knew that they are both experimenting with electronic music. My only guidelines were that they had to produce 18 different tracks that had to illustrate – just like the video – the development of the each stock quote. And I asked them not to make it too darkish overall. The early samples were astonishing. Way better than I expected. The sound had destinctive themes and still represented the stock quote and even the different companies. With only some of the tracks activated you get a pleasing electronic song, but once they‘re all turned on, it grows into an experimental chaos.

2.7 Video

Since the very beginning the aesthetics of the video had been technically restricted. At one point I won‘t cut the amount of video layers. Less quotes also meant less chaos and with the chaos I was afraid of losing one of the main messages of the installation. That‘s why we, the programmer Alekz and me, had to look for work arounds. The loss of Alpha-channels for example seemed to be pretty difficult but we figured out other ways to use the blend mode in a similar way. Also the „screen“ blend mode has it‘s own TV aesthetic which supports my intentions. At the start I wanted the video to look more infographical but the thin lines would have been blurred through the upscaling and I decided to go for a two dimensional TV aesthetic. I had several reasons doing this. First at all the TV is still the number one news source. Even the internet, in some instances, is nothing more than a bi-directional highly networked TV (Luckily McLuhan can‘t hear me anymore.) So in a way going for a TV look was referencing my research on the Screen Realities whilst the stock trader‘s desk would be the control room.

The process of producing each layer was quite complicated. Rotoscoping each scene to take it out of it‘s original context was more difficult than expected. I was trying different new rotoscoping softwares like Motor from Imagineer Software and some After Effects scripts but in the end they couldn‘t help with the mess of a crash. For some videos I needed more than 50 different masks to rotoscope them. Afterwards the rotoscoped video was time remapped. Therefore I normalized the stock quotes of each of the 18 companies and converted it to time values. These normalized values had been applied to the video layer. And in the very end each layer got his own finishing for achieving a unique look.

3 The Final Piece

Crash - A stock trader's desk
Installation shot: Candid Gallery, June 2009
Installation shot, detail: Candid Gallery, June 2009
Moving image: Stock quote of Citigroup (March 2008 – March 2009)
Moving image: Stock quote of Procter & Gamble (March 2008 – March 2009)
Moving image: Stock quote of General Electric (March 2008 – March 2009)
Moving image: Stock quotes of Boeing, Citigroup, General Electric and Procter & Gamble (March 2008 – March 2009)

This is the online portfolio for motion and design projects and related research of Nico Roicke. I'm a graduate of MA Communication Design, pathway Digital Media, at Central Saint Martins College of Art and Design, London. Also I'm part of the motion design studios Sir ja sir and Buchstabenschubser.

Beside motion graphics (or motion design, if you prefer) I'm the co-founder of the fanzine and weblog Jackpot Baby! - New digital pop culture (all texts unfortunately in German language) and I write for Berlin's finest Webblog Spreeblick - Pop, Politics, Products & Positions (German again). Go and catch up some of my latest tweets on Twitter (English and German, it's really hard to tell) or take a glance at some of my Flickr pics and Vimeo vids.