Emptying the Haunted Air

Thesis

PDF:

playful-audition-the-everyday-experience-of-sound-in-video-games

Here is my university honours thesis, submitted at the University of Technology, Sydney in November 2008. I have made alterations so it is more suitable for online reading – alterations which include more pictures (everyone likes pictures), sound samples, as well as various edits to the original text. Truth be told, I should have submitted the thing as an online project in the first place… But I’m a bit of a traditionalist when it comes to scholarship and erudition.

I hope that , in making it available online, my work might actually be of some use, particularly for my own studies in the field of sound and video games. The PDf is probably the best way to read it, but the complete text is here as well, in all its atrociously formatted glory.

***

Playful Audition: the ‘Everyday’ Experience of Sound in Video Games

Acknowledgements

Firstly, I’d like to thank Shannon O’Neill, my thesis supervisor – his comments and suggestions during the course of my research have been invaluable. Without his wealth of knowledge about all things sound and game-related, I would certainly have had no way of moving forward. I must also thank him for his patience, for answering all of my questions, and challenging me to always seek new questions, new weaknesses in my understanding and, therefore, new ways to approach my research. There are, probably, two-dozen concepts and academic strands that simply would not have been in my thesis had Shannon not led me to them.

Most importantly, Shannon has encouraged me to look at this research not only as a short-term academic project, but as a springboard to future pursuits and studies. I approach the end of this year with many more academic interests, creative pastimes and ideas to keep my mind occupied than I did at the start of the year – I owe this to Shannon’s generosity, as well as his inquisitive and open-minded nature.

I’m also very thankful to Norie Neumark for being so patient, not just with me, but the rest of the Honours cohort as well. She provided some of the most important references in my thesis, and her feedback on my respective presentations helped my arguments take shape. The breadth of her knowledge, considering she assisted all of the Honours students in their diverse fields of research, is astonishing.

I am grateful to Stephen Muecke and Theo van Leeuwen, my Honours coursework lecturers – Stephen, for filling in all of the cultural studies gaps that three years in a production-focused course had left, and Theo, for opening my mind to alternative ways of looking at my medium of choice. The knowledge I gained from these two courses was instrumental in my research.

Thanks to my student colleagues, without whom this year would have been a very solitary journey indeed. Their feedback towards my own research, and genuine enthusiasm for everyone else’s, was very humbling.

Finally, I owe a great debt of thanks to my family, for being patient with me during this very intense year and, for my whole life, compelling me to be inquisitive and self-confident. Thanks to Matthew, in particular, for keeping my toon geared during the year, and saving me from a very somber return to online gaming.

Without all the knowledge, time and effort that these people have given me over the year, I wouldn’t have been able to commit a single word to page. This thesis reflects a great amount of work from many different people.

Part 1

Introduction: The Void in Contemporary Video Game Theory

Video games are a relatively new field of academic study, and a medium that has only recently begun to acquire legitimacy as a mode of artistic expression. In a matter of decades, and especially over this past ten years, games have cast off their status as recreation for children, and burst into the centre of aesthetic, moral and technological debates. Despite the saturated contemporary mediascape and legally-hostile environments (especially in Australia, where tight censorship laws frequently see games banned for sale), the medium has become a multi-billion dollar industry to rival film, as well as fertile artistic territory for independent game designers to explore. In this context, the need to understand this new medium is more pressing than ever, and game theorists have addressed this need by suggesting many different approaches. I argue that, of the myriad of theoretical frameworks with which one can attempt to understand games, gaming, and gamers, phenomenology fills a distinct void that is prominent in much of the academic writing on video games – the fact that their accounts do not seem to provide a path to describing the human experience of games, and the auditive experience in particular. Foregrounding human embodiment also allows us to suggest ways that sound composers can improve this auditive experience and, by extension, the entire medium.

Recent work in video game studies has been focused on devising hybrid theoretical frameworks with which to understand the medium; approaches that attempt to deal with the way games construct meaning while, at the same time, articulate the experience of playing a game. A prime example of this trend is Thomas Malaby’s account of games as “processual” (2007, p. 102), in which he makes a broad argument that games in general (and online video games in particular) cannot be separated from everyday experience – that they “are grounded in (and constituted by) human practice and are therefore always in the process of becoming” (p. 103). This, he argues, is an important approach in video game studies, in that it diverges from the modernist notion of play as being unproductive; opposed to real life (Caillois 1979, p. 10). In describing the process of playing a game, Malaby argues that the experience is one of “contrived contingency” (p. 96). In other words, a game constructs for the player a set of possibilities which produce “a mix of predictable and unpredictable outcomes” (p. 106). Alexander Galloway’s approach (2006) could be seen as an extension upon Malaby’s work in that, in place of one actor, there is instead of network of actors (or, as he calls them, “acts”), each operating its own set of contrived contingencies.

Such approaches are valuable, especially because they cast aside the casual associations of play, and describe games as legitimate, social phenomena. In spite of this, there is still a gaping hole in the emerging field of video game studies – one that I feel most long-time gamers would notice most of all. Invoking the term ‘gamer’ in this way, and implicitly including myself as a member of this group, is problematic. My use of the word ‘gamer’ throughout this essay is not intended to five the impression of exclusivity. Indeed gaming, which for decades was the pastime of a small subculture, has become a hugely popular and commercialized industry, and the medium can no longer be said to belong to any particular social group. By ‘gamer’, I simply refer to people who spend a substantial amount of their time playing games (there is need to be specific about how long), and who have some sense of the history of the medium, whether through direct experience, or from other sources. I contrast this term with ‘casual gamer’, which refers to anyone who might have played video games, but not necessarily on a regular basis; people who might not have made an emotional engagement with a game. These are vague definitions, and I have no desire to be prescriptive; people can engage with video games on many different levels. Broadly speaking, I suspect that anyone who is a gamer will be aware of it, and identify themselves as such. This, perhaps, is the only way possible to identify a gamer.

In the cases of Malaby and Galloway (and many other writers) there is only a marginal account of video games as phenomenological and sensory experiences. Of course, no writer would deny the importance of meaningful, semiotic components of a video game; Malaby’s definition of games (remembering that his argument encompasses all types of games, from digital games to gambling) describes the components of a game:

“A game is a semibounded and socially legitimate domain of contrived contingency that generates interpretable outcomes” (Malaby, 2007, p. 96)

According to this definition, video games are primarily spaces that, to varying degrees, grant the gamer agency within a complex web of computational and social possibilities. Again, taking into consideration Galloway’s work in the field, we inevitably have to accept that there is more than a single actor present in the process of playing a video game – an ‘operator’ and a ‘machine’ (2006, p. 5) – and more than one space in which actions are taking place – ‘diegesis’ and ‘nondiegesis’ (pp. 8-28). (We don’t necessarily have to accept such terminology, but Galloway’s classifications of the four ‘gamic actions’ are sufficiently distinct from one another, and he backs them up with reference to specific games and gamic moments in a way that, I feel, would seem sensible to most gamers).

Regardless of how complex, or computationally mind-boggling, this web of ‘contrived contingencies’ might become, a problem still seems to exist in such a processual account of games; a problem that makes it difficult to imagine how one might use such a theoretical tool to describe any specific video game. Harder, still, to describe the bizarre experiences on which millions of gamers have whiled away large portions of their lives. The study of video games is still developing, and gaining credibility as a field of academic research; approaches like Malaby’s and Galloway’s are useful is distinguishing video games from other creative mediums. Even more usefully, they go well beyond the paradigms of ludology (the analysis of games through the theory of play) and narratology (the study of games as a storytelling medium), and provide contemporary writers with a new way to think about video games; a postmodern framework in which games are not deemed spaces of non-work, and non-life, and therefore wasteful and unproductive. The experience of playing games is not necessarily separable from everyday life (Malaby, 2007, p. 96); it is highly creative, and by no means defined (or limited) by contrived rules.

Yet, whether one sees video games as processual, or as actions, there is a need to recognise the role that the vast number of meaningful components occupy in the process of playing a game; Galloway’s is a useful analysis in this regard. The approach that I take in this essay accepts that video games are unique as a medium due to the agency they grant a player, but I propose that phenomenological analysis is a crucial tool in describing what it is like to experience a game. Broadly speaking, I feel that, often contrary to their stated intentions, many writers sound like they have very little experience of playing games. Generally this is because, like Malaby, they are making very broad and ambitious arguments about video games as an entire medium; the names of individual video games are often dropped in casually, and sometimes not mentioned at all. In order to address this (what I consider quite bizarre) state of affairs, I will focus on specific games, such as Deus Ex (2000), BioShock (2007) and Half-Life 2 (2004), to name a few. More importantly, I seek to identify the phenomenological aspects of audition in these games that make them so compelling; how the texture, mood, timbre and colour of sound shape gamic contingency, to create experiences that are creative, poetic, and unforgettable.

The ‘Everyday’ Experience of Games

In contemporary video game studies, there are surprisingly few humanist accounts of gamic experience. Some, like Galloway and Stockburger, adopt approaches that define and theorize gamic space as a meaningful combination of digital actors. In his auditive analysis of the game environment, Axel Stockburger describes a network of ‘sound objects’, while Galloway argues that games consist of various ‘actions’ that fall into certain diegetic and nondiegetic spaces (those within the narrative world of the game, and those external to it). In both cases, although less so in the case of Galloway, an ‘ideal’ gamic experience is presupposed; these actors within the gamic space are programmed to be experienced a certain way, forever subjected to a set of specific rules.

This concept of an ‘ideal’ experience (which favours the experience as planned by game designers) might seem sensible when one regards a video game as a commercial product; indeed, most video game critics (whose opinions occupy an important place in the commercial cycle of game development) would argue that there is no other way to evaluate a game. It is impossible, in trying to evaluate its quality, to anticipate how people might use a game. In many cases, the true potential of games is only realized years after release, by software patches and modding teams (as in the case of Vampire: Bloodlines, 2004). However important traditional criticism might be to the continued commercial triumphs of genuinely high-quality video games, I am not interested in such objective scales against which games might be judged. Indeed, my primary interest is how we might describe video games as phenomenological and, more specifically, phycho- and physio-acoustic artefacts. In addressing this, it would it be very limiting to discuss only the painstakingly-woven diegetic space and rules that the game developers intended us to see (yet not always to hear, as in the graphics-oriented world of commercial video games). Michel de Certeau showed us that the everyday walker can wander aimlessly about the urban streets, in a manner not planned or directed by the utilitarian design of the modern city; an observation that I will relate to video games later in this chapter.

However, I should first make clear some of the terminology that I will use throughout my argument. According to Galloway, “video games are actions” (2006, p. 2). I would like to begin by modifying this statement: video games are actions mediated by human embodiment. Galloway himself doesn’t shy away from this notion: “A video game is a cultural object, bound by history and materiality” (p. 1). ‘Materiality’, I take it, refers to the fact that there is a physical dimension of games that exists alongside the computer code, and his terms ‘operator’ and ‘machine’ evoke (quite deliberately, I think) this physical relationship. Yet, he only discusses gamic space in regards to four key ‘acts’ that he outlines in the first chapter of his book: diegetic operator acts, diegetic machine acts, nondiegetic operator acts and nondiegetic machine acts. I will take this opportunity to mention that, throughout this essay, I use the terms ‘game’ and ‘video game’ interchangeably to refer to the same artefact, which Galloway defines simply as “consisting of an electronic computational device and a game simulated in software” (p.1). Furthermore, when using the term ‘gamic environment’, I am not only referring to the virtual, diegetic space in which a player controls an avatar. The gamic environment is a hybrid space that includes all of the semiotic components of the game, but is always mediated by human embodiment and perception. It is a space that, for me, is inseparable from the experience of everyday life.

To the ordinary man. To a common hero, an ubiquitous character walking in countless thousands on the streets.

Michel de Certeau (1984, )

There is such a thing as an everyday experience of a game, and it is vastly different from, what I shall name, the topographic experience of a game, as envisaged by the designer. The topographic experience is preempted, encoded, but eternally hypothetical and elusive; it may be plotted, but is never truly enacted. I give the example of one of my very favourite games, Deus Ex, developed by Ion Storm Austin in 2000. Upon its release, it was lauded as a remarkably open-ended game (and, even by current standards, still is), giving the player control of a cybernetically-enhanced avatar named JC Denton within virtual simulations of futuristic urban environments. Each ‘level’ (or self-contained virtual space) contains numerous characters, objectives, and problems to solve, and the manner in which the player interacts with these various diegetic actors changes the way in which the game’s plot unfolds. So, if I were to attempt to describe how I played the level in Hell’s Kitchen, New York, it might read thus:

Once the level had finished loading, I commanded my character to approach the edge of the building that I had landed upon, and looked around the environment. I then noticed a ladder that ran down the side of the building, which I descended until I was at street level. I noticed that there was a closed manhole on the ground, and remembered that my objectives required me to go into the sewer system under the street. The manhole cover was locked, and I knew I could either pick the lock, or destroy it with explosives. I looked in my inventory and…

Such a description may well be accurate when approaching video games as networks of actions, as Galloway’s approach would entail. It articulates an accumulation of actions and experiences. Indeed, my recount of a moment in Deus Ex, despite being non-poetic in the extreme, hides an incredible network of calculations and programming variables that are wondrous in their complexity. It also raises some complex ontological questions that I do not intend to discuss in this essay; specifically, the question of who we are when we are playing a game, and the nature of gaming subjectivity. Yet, such an action-oriented description isn’t terribly sophisticated, nor is it true to my actual experience. In a topographic description of Deus Ex, there is a heads up display (HUD) from which one extracts valuable information pertinent to playing the game, there are virtual cars, characters, usable objects (like boxes and weapons) and all manner of amazing diegetic and non-diegetic actors with which we can play (in the ludological sense of the word). Similarly, in the topographic city that bustles with people, there are lattices of streets, main roads and buildings, parks and districts that pertain to uses preconceived by the city planner.

The city (as conceived by Michel de Certeau) and the video game are spaces that are quite similar. I come to this conclusion not by reading the meaningful constitutive parts of the gamic environment, as a social-semiotist might be inclined, but by observing the particular ways in which people use these topographic environments at street level, so to speak. As such, in Deus Ex, which coincidentally takes place partially on the streets of New York, the manhole referenced above is actualized in two ways. Firstly, as a gamic actor; a cluster of pixels rendered on the computer screen (nondiegetic machine act; Galloway 2006, p. 28) which form a steamy manhole on the street (diegetic machine act; p. 8), creating a ‘problem’ that the player must ‘solve’ (nondiegetic operator act; p. 19) by way of one of JC Denton’s tools (diegetic operator act; p. 12). The same manhole also comes into being as a virtual and symbolic object in the way that Certeau described objects in the city: “If it is true that a spatial order organizes an ensemble of possibilities (e.g., by a place in which one can move) and interdictions (e.g., by a wall that prevents one from going further), then the walker actualizes some of these possibilities. In that way, he makes them exist as well as emerge” (1984, p. 98).

This may be meaningful for Certeau’s walker on the city streets, for whom a wall is no more an obstruction than it is a spatial signifier that is transformed into something else; a possibility to create other, illegitimate or unplanned routes through the city. However, what does it mean for a gamer, staring at a manhole in Deus Ex? The same notion applies here as for the everyday walker; ignoring the fact that we are talking about a virtual street, the contrived diegetic barrier of the manhole acts as an opportunity for the everyday gamer to create their own space using ‘tactics’ that are influenced by, but never wholly bound to, the ‘strategies’ that define the environments that game developers have laid out for us (I borrow the terms ‘tactics’ and strategies’ from Certeau). In Deus Ex, this might involve choosing one of the predetermined paths to pass through the manhole, or ignoring the manhole completely, and exploring another part of the gamic environment. It might be a time when I ignore graphical elements of the game altogether, and pay attention to the sound and music for its own sake or, perhaps, to make a smiley face on the manhole cover with bullet marks. I might even turn off the game entirely and test the veracity of Ion Storm’s claims to realistically modeled environments by comparing Deus Ex’s New York to a real map of the city. In fact, there are many millions of possibilities, and Galloway would be correct if he were to contend, here, that they are all enabled by the deliberate programming of the developers. I can walk around aimlessly, or write on the concrete walls because they have programmed the game as such; even shutting down Deus Ex, in Galloway’s analysis, would be something that is enacted within the rules of the game, by an operator and a machine. Every possibility is preconceived. Yet, the idiosyncratic way in which we actualize these possibilities is not; we ignore, negotiate and recombine the rules and structures in the gamic environment in ways that are influenced, but never determined, by these rules and structures. The gamic environment may belong to Ion Storm, as the topographic city belongs to architects, city planners and governing bodies, but it is purely hypothetical. The gamic space is entirely my own, just as the street is to the everyday walker.

In practice, this would lead us to conclude that everyday gamic space never consists of the totality of gamic actors, but a selection of them actualized by the player. This is at odds with the dominant trend in contemporary video game studies, which, just as Galloway does, tends to take a reductive approach to games, defining them by the network of their constitutive parts (often in comparison to film). In their book, Wolf and Perron adopt a similar approach: “in defining the video game, a few elements seem to appear persistently, under various names and descriptions. These elements are at the heart of what makes the video game a unique medium… The most fundamental of these elements are: an algorithm, player activity, interface and graphics” (2003, p. 14). (The lack of ‘sound’ in this breakdown is worrying, and not uncommon). No matter how elaborate this network becomes, or how incredible video games can be as technological artefacts, the problem of describing the human experience of playing a game still persists in contemporary video game theory, despite the widespread acceptance that ‘play’ is a feature of critical importance to video games.

The problem, perhaps, lies in the roots of the field of ludology – the theoretical approach to video games that concerns itself primarily with the concept of ‘play’. One of the theorists most influential to ludology, Johan Huizinga, wrote that play transpires “quite consciously outside ‘ordinary’ life” (1950, p. 13) and furthermore that it “is a voluntary activity or occupation executed within fixed limits of time and place, according to rules freely accepted but absolutely binding” (p. 28). More recently, Roger Caillois echoes this view, writing that games are “accompanied by a special awareness of a second reality or of a free unreality, as against real life” (1979, p. 10). In both cases, the ‘unreality’ or lack of ‘ordinariness’ in play is stressed. Other recent discussions of video games seem to subscribe to this notion: “To play a video game is… to interact with real rules while imagining a fictional world” (Juul, 2005, p. 1). Yet, in relating the experience of playing video games to Michel de Certeau’s discussion of the ‘ordinary man’ walking in the city, I contend that there is something distinctly ‘everyday’ about playing a game; to reiterate, the manner in which we actualize space in a videogame is the same as the way we do so in the city, regardless of the extent to which the diegetic and nondiegetic elements of the video game remove our imaginations from our immediate surroundings. Furthermore, in stark contrast to Huizinga, my relating of gamic space to everyday life as conceived by Certeau implies that the notion of a game’s rules being ‘”absolutely binding” is incorrect, for a gamer’s negotiation of a virtual space will always be, to an extent, undetermined by rules; Malaby’s notion of ‘contrived contingency’ recognises this. One of the overarching problems I am trying to address in this essay is that, in their exclusive concern with the ‘topography’ of video games and the ‘idealized’ game player who is bound by its rules, contemporary video game theorists are failing to articulate the most important and profound quality of video games; how we actually experience them. Contemporary writers such as Caillois, Wolf and Perron, and Juul therefore place an unfair amount of responsibility on the game developer as the creator of gamic experience which, in its similarity to Auteur Theory, ignores the everyday gamer as a creative agent in the video game.

Games do not contain encoded experience. They do not even contain an encoded network of possible experiences. Video game studies is a developing, exciting academic field that deals with a new and ever-changing medium, and while reducing video games to their semiotic components is useful in describing them as technological artefacts, such an approach risks losing sight of the human experience of playing a game; the embodied experience, as I shall describe in the next chapter. There is an important place for humanist frameworks in video game studies; a place that my conception of everyday gamic experience scarcely occupies. Indeed, my essay leaves many questions unanswered, the most important of which is a deliberate omission I have made – a non-human actor in a video game whose own experiences I have deemed outside the limits of my discussion. I am talking about the experience of the machine, or computer, who occupies such an important role in the act of play. I propose that phenomenology (in the Husserlian sense of studying phenomena as perceived through human consciousness), is more useful than a reductive approaches for the purposes of describing the experience of games, and while my concern is exclusively on the auditive experience, a similar approach could be adopted for a discussion of video game graphics. Furthermore, by acknowledging the fact that human audition and embodiment mediates the virtual environments in games, it is possible to propose new, more immersive (or, in the case of the aesthetics of failure, deliberately less immersive) ways of composing sound for games.

Part2

Sounding the Everyday Gamic Environment: Who is listening?

In the first chapter, I raised my concerns with the idea of presupposing an idealized experience of games, because any such notion is undermined by the idiosyncratic manner in which people play games, and actualize scenarios in a game. To talk about sound in particular, this needs to be expanded upon: even though the central premise of this essay denies that a person’s experience of sound objects as ‘actions’ can be encoded into a game by developers, I am assuming a level of uniformity in the experience of game sounds as physical events. The social conditions in which we play games are yet another quality that is unique to the medium and deserves its own study (see: Swalwell 2002, pp. 84-138), but in writing about sound, I am not taking into account the low-frequency hum of a computer tower, or the frenetic noises of a LAN party, or music in the background. These considerations would have to be addressed in a more complete analysis of the nature of gamic audition; particularly interesting is the way many gamers augment their auditive experience with background music players and community mods (short for ‘modifications’; user-made alterations to a game). Certain first-person games – generally multiplayer titles, like Valve’s Counter-Strike (1999) – demand that such socio-environmental factors be addressed in talking about the game. Indeed, Counter-Strike, for its close association with the LAN scene and highly competitive nature, might require its very own analysis. Instead, the sort of first-person experience I am concentrating on, demonstrated best by games such as Deus Ex and BioShock (2007), is solitary in nature, and played in controlled conditions. Any multiplayer experience, if it exists at all, is usually secondary to the singleplayer gameplay. The person listening to these games, in this essay, exists somewhere between the player’s perception and the implied perception of the virtual avatar.

What do we hear?

Immersion inside a virtual reality rests, according to Myron Krueger, on the extent of our physical engagement with the virtual world, and as such his artworks sought to make use of, and to extend upon, our “evolutionarily acquired embodiment” (Hansen 2006, p. 28). If Krueger is correct, that the key to a successful mediation between the virtual and the lifeworld is human embodiment, then it is important to look at the conventions of sound design in video games (or any aspect of their design) less as reconfigured extensions of filmic and narrative conventions (as Galloway and Stockburger have done), and analyse them instead within a humanist model that prioritizes the enactive qualities of games.

The acoustic spatial code of first-person video games – by which certain sounds have an absolute, unchanging spatial orientation, and others are synchronized to in-game graphical actors – bears many functional similarities to its filmic counterpart. However, it acts upon us in way that is unique to the medium, and this is particularly apparent in first-person games, as compared to other types of games. In most first-person games, sounds that are attributed to in-game events or objects serve three broad purposes: to orient us in three-dimensional space, to highlight certain events that are pertinent to the player (out of ammo, etc) and, finally, these sounds also exist for their affective, timbral qualities alone. The fact that human embodiment is so important to one’s immersion in a virtual world explains why most game developers have adhered so faithfully to the conventional acoustic spatial code in first-person games; it locates the player at the very centre of the acoustic environment. In the earliest examples of first-person games, such as Wolfenstein 3D (1992) and Doom (1993), sounds are merely increased or decreased in volume depending on how close they are to the player. Recent games, such as Half-Life 2 (2004) and Crysis (2007), support surround sound, and realistically attenuate sounds based on the virtual acoustic space and distance from the player. Regardless of the sophistication of their respective game engines, these games have all sought to locate the player in the centre of the acoustic space; the effect is not always one of realism, but invariably one of human embodiment. Mark Hansen argues that “human embodiment serves to ‘naturalize’ technical modifications of the world (and, potentially, of the body)” (2006, p. 28); in the context of video games, such modifications might include the way a computer monitor limits visual perspective of the virtual world, a game’s heads up display, or the limited range of virtual movement imposed upon the player by a conventional mouse and keyboard. These distinctly unnatural modifications to the lifeworld are mediated by the human embodiment of sounds in video games; made comprehensible and natural. Most avid gamers will have had the amusing (and indescribably frustrating) experience of watching someone who is unused to first-person games attempt to negotiate the many challenges of moving through a virtual world; often, they can’t do two things at once (such as the ‘moving and shooting’ that is common in first-person shooters), and strafing (moving left and right) is simply an impossibility. The acoustic spatial code, in the way it makes use of our embodiment, helps us to become familiar with these obstacles; as Hansen states, “embodied enaction is, quite literally, the agent through which technics has an impact on life and the lifeworld” (p. 29).

Yet, for the everyday gamer (who is necessarily a seasoned gamer), sound events do more than simply assist with navigating virtual space. Once familiar with the modifications to the lifeworld that make the virtual world possible, the everyday gamer is, all of a sudden, in a position to read the virtual soundscape in a manner that is no different to the way sound theorists have attempted to read the real-life soundscape and, as a result, empowered with an ability to manipulate and create meaning from the soundscape. However, the nature of audition in a video game is very different to that of real-life city streets, tundra or plains. R Murray Schaefer’s concepts of the ‘acoustic ecology’ and ‘acoustic design’ (1977, p. 205) have informed my attempts to interpret gamic sound as a type of musical orchestration, but his proposed methods could not be directly transposed onto games. One of his major preoccupations, for example, is with eliminating “boring or destructive sounds”, but in a game’s acoustic environment, there are no undesirable sounds to be heard, aside from those outside the direct control of designers, like sound glitches, or noises in a gamer’s local environment. Still, there is a tendency for some writing in the field – particularly narratologists – to describe the experience of games as a kind of virtual reality, in which a player’s subjectivity is merged with that of the avatar. Tanya Krzywinska’s textual approach to games is an example of this problematic way of looking at games; it begins with a description of some time spent in World of Warcraft:

“I’m on my way to Iron Forge from the night-elf outpost of Astranaar; it’s quite a journey, involving running, catching a boat, and flying on a hippogryff. My bags are full of booty robbed from the newly slain, and I’m looking to trade it for a stave with better stats” (Krzywinska 2006, p. 119)

Of course, Krzywinska isn’t suggesting here that playing a game somehow involves imagining oneself in the role of a virtual avatar – the very reference to tactical decisions (the desire for an in-game item with ‘better stats’) in the above passage is indicative of another type of complex gaming subjectivity; one that is best explored by strategic MMORPGs (massively multiplayer online role-playing games) such as World of Warcraft (2004). There are some writers who have devised approaches to this type of subjectivity in online games, known by Miroslaw Filiciak as ‘hyperidentity’ (Filiciak in Wolf & Perron (eds.) 2003, p. 87). These approaches lie outside the limits of my particular discussion, however. Suffice to say, I feel that descriptions such as Krzywinska’s incorrectly place video games within a virtual reality framework that is already pushed obsessively by game publishers, who have been forever heralding graphics technologies as delivering ‘unparalleled realism’, even dating back to the box-art on Super Nintendo games.

So, perhaps more so for the everyday gamer than the casual gamer, video games are an immersive and embodied experience and, furthermore, first-person games provide the best examples because of the way designers make use of human embodiment to construct logical, believable and negotiable worlds. The New York City of Deus Ex, however, believable as may be, is entirely different from the same city that Certeau gazed down upon from the World Trade Center. The sounds that populate it, while not out-of-place, are strange and unique; there are very few of them – the sounds of the player’s footsteps on concrete (panned to the left and right channels for each step), gunshots during combat, and the occasional sound of a car or truck that cannot be seen. There is also music, which is not subtle, but prominent in the overall mix. On top of this, these are all 16-bit sounds due to a limitation with the game’s engine (Unreal Engine 1, developed in 1998), and feature no reverberation because the effect, as it featured in the engine when Deus Ex was developed, produced rather undesirable results. Despite the fact that Deus Ex is a remarkable game, it features a conventional soundtrack for a game of its generation – the technology that powered it was still in use by commercial game developers until 2003, and the limitations that held back its sound design feature in most games from that period. The only reason I refer to Deus Ex, aside from the fact it is widely considered a great game, is because it provides an entry point to talking about some of the most basic properties and purposes of sounds in video games – characteristics that have carried through into today’s cutting-edge first-person games.

One of the most important characteristics of gamic sound is their necessary, and perceptibly obvious, repetition. From a topographic perspective, game sound effects are assets in a game data folder that are played at programmed moments, or else simply looped over and over. Multiple instancing of the same sound effects frees up system memory, and also saves development costs by minimizing the required amount of sound recording and production. It is a creative constraint placed upon sound designers, and a feature of technology that will inevitably change; until technology exists that allows game soundtracks to be synthesized entirely, however, the repetition of engine sound objects is a prominent part of the auditive experience in any game. Even though its application to games cannot be justified on the grounds of a ‘virtual’ or perceptual similarity between real life and games, the ‘Repeatable Object’ (RO) conception of sound in the real-world acoustic ecology provides a useful framework for understanding the ontology of the repeating sound and how it shapes one’s experience of the gamic soundscape. The RO ontology conceives “sounds to be kinds of objects rather than events… [They] behave like autonomous entities… typically caused by material objects, but do not essentially depend on them” (Dokic 2007, p. 392). Matthew Nudds suggests one of the motivations behind this particular phenomenology of human auditory experience:

“Normally we think of events as things that, unlike objects, cannot be re-encountered. The fact that we can re-encounter sounds suggests that they are perhaps best thought of as kinds of objects, rather than events.” (2001, p. 222)

In discussing this conception of sounds as repeatable objects, Dokic raises serious doubts about the conditions under which sounds “can exist as wholes in more than one place and time” (2007, p. 392) in the natural ecology. However, even if its application to ecological audition is dubious, the effect of certain sounds being frequently re-encountered in first-person games can be articulated by particular theorists who have been concerned with sounds as objects – of whom the most obvious, and perhaps significant, is Pierre Schaeffer, and his notions of the ‘objet sonore’ and ‘reduced listening’ from his work Traité des Objets Musicaux (Schaeffer 1966). To Michel Chion, himself a former student of Schaeffer, the most basic condition of reduced listening is that “the descriptive inventory of a sound cannot be compiled in a single hearing” (Chion 1994, p. 30). A sound has to be listened to many times over, and for this to be the case, it must be recorded; made into an object. Reduced listening is not an exercise that is possible in the real-life acoustic ecology, but could be applied to Schaeffer’s own ‘music concrète’, and also to film (as an educational exercise for sound composers) as Chion suggests (p. 31). As I have already described, sounds are frequently repeated in first-person games out of necessity, but this feature – which might otherwise be viewed as a technological limitation to developers – might also be a site of increasing compositional interest to game sound designers.

‘Everyday’ experience as acousmatic experience

Before I move on to the compositional interest of reduced listening and, as I shall explain later, Schaeffer’s so-called ‘acousmatic experience’, I wish to draw attention to an inadequacy that presents itself in some contemporary approaches to sound in games. It isn’t unusual to classify sounds as ‘objects’ in games; as I mentioned in the first chapter, Alexander Galloway regards sounds as ‘acts’ that fall into certain quadrants of “gamic action” (Galloway 2006, p. 38). Admittedly, his analysis would encompass a far greater breadth of sounds than I am choosing to look at here (for it would include sounds made external to the game environment, such as the mechanical hum of a computer fan), but it is an example of writing that describes sounds as assets within an algorithmic process, rather than for their phenomenological effect. Axel Stockburger in particular – who, unlike many other writers in the field, has given specific consideration to the role of sound in video games – classifies sounds as objects in a manner entirely different to Schaeffer’s ‘objet sonores’, and he explicitly draws attention to the contrast:

“The notion of the sound object clearly resonates with meaning from the field of music theory and its use in the game context has to be clarified.” (2003, p. 5)

Then, when he approaches the topic of reduced listening, he writes:

“It is quite obvious that a mode of reduced listening will not be achieved when we are playing an audiovisual game, simply because we are drawn to construct relations between the visual and auditory information we are receiving. Thus the way we understand sound objects is different from Schaffer’s use.” (2003, p. 5)

In light of the highly impressive achievements in sound design that certain, very recent games have delivered, the age of Stockburger’s analysis might excuse his dismissal of the possibility of reduced listening in a video game. Regardless, an outright dismissal of the possibility is ill-conceived, and shows a lack of appreciation for the phenomenological qualities of sound in games; some of which have been released in recent years, like BioShock (2007), and others that are now fairly old, like System Shock 2 (1999). The most obvious problem with Stockburger’s statements is that they aren’t entirely accurate – the way he describes the sound object in a game isn’t completely different from Schaeffer’s, and also Chion’s, use of the term. There are sounds of a certain type that he recognises in his taxonomy, mainly within his categories of ‘effect’ and ‘zone’ (2003, pp. 5-9), that occur without a visual cause in the game. According to Schaeffer’s conception of sound objects, such a phenomenon gives rise to the ‘acousmatic situation’ – an experience that can modify our listening. Deus Ex, released three years prior to Stockburger’s paper, is composed of many such sounds: the musical soundtrack when the player is in the Majestic 12 Laboratories features mechanical sounds that resemble doors, while the aforementioned section in New York city features the sounds of cars that, rather comically, are nowhere to be seen in the area’s deserted streets. For Chion, “acousmatic sound draws our attention to sound traits normally hidden from us by the simultaneous sight of the causes… the acousmatic truly allows sound to reveal itself in all its dimensions” (Chion 1994, p. 32). Stockburger doesn’t deny the possibility of the acousmatic situation in video games, although he makes no mention of it either, and based on his assertions regarding Schaeffer’s ‘sound objects’ and his own, it seems as though he hasn’t considered the possibility of an acousmatic experience in games.

Stockburger’s denial of the possibility of reduced listening in a game is not quite as easy to contradict. Reduced listening is not a natural enterprise, for it requires the listener to move beyond a casual appreciation of sound – it requires a certain amount of auditive work, and this would, in a video game, necessitate some fairly unusual modifications to the game world. It is not enough that first-person games will repeat certain sounds (gunshots are a common example); the player would have to consciously trigger these sounds over and over, listening keenly to their timbral characteristics. In Deus Ex, where ammunition is often scarce, this practice would be impractical, and at worst destructive, to other aspects of the game. On top of this, reduced listening would require that the player cease to pay attention to visual and physical elements of the game. Releasing control and sacrificing one’s vision is, in most intense first-person games, a dangerous thing indeed. At this point, it could be argued that reduced listening turns the experience of the game into a ludicrous affair – surely, a game is no longer being played at all.

However, games are multimodal artefacts, and don’t necessarily involve a hybrid audio-visual experience; they can facilitate many different levels of player interaction and input. Although there are very few games (commercial, at least) that provide such experiences, a few moments I have spent in front of a screen come to mind: in Mario Party (1999) on the Nintendo 64, for example, certain minigames required the player to revolve the controller’s joystick, or else to mash buttons, as fast as possible in order to win the task. Anyone who played the game will recall that these were maddeningly chaotic experiences. To be victorious, you had no choice but to ignore the television entirely, to ignore competitors, and basically anything else that was external to the task at hand. It was a brutal, purely physical act, and earned Mario Party a notorious reputation for destroying Nintendo 64 controllers, and causing blisters and hand injuries to the more vigorous players. Nor was it particularly fun, but it is an example of a commercially successful game where at least part of the experience deviated from the typical audio-visual interface with the game. Many adventure games from the 1990s provide examples of worlds that are purely visual experiences; they are not silent, but sound plays a very minor role in the game, due mainly to the fact that spatial orientation is less important. The Curse of Monkey Island (1997), Discworld (1995) and The Longest Journey (1999) all necessitate, and subsequently reward, a heightened attentiveness to the visual properties of the gamic composition – objects that glint, move strangely (or at all), or otherwise stand out in some way, are usually objects that need to be collected or manipulated in order to progress in the game. As a direct consequence, these three games, and many others in the adventure genre, engender a very real aesthetic appreciation of the visual-compositional qualities of the gamic environment – indeed, it is very difficult not to appreciate the quality of the artwork of The Longest Journey.

In all of these cases, the fact that the games diverge from the typical conception of games as an audio-visual medium doesn’t involve the suspension of play at any time. Malaby, in arguing that games are domains of ‘contrived contingency’, legitimizes many different types and expressions of play. I argue that, just as games can be purely visual experiences, and promote visual-aesthetic awareness, it is equally possible for a game to facilitate heightened auditive awareness. The practice of reduced listening in a game is certainly not conventional but, if one chooses to invest the time and effort to do so, can be informative and profound. It can be practiced in any game, but the best results obviously emerge when the developers have taken a more thoughtful and inspired approach to sound design; BioShock is one such game. Released in 2007, BioShock is a first-person shooter that is set in a 1940s, art deco-inspired, underwater metropolis. Capturing the feeling of being completely surrounded by water was naturally a high priority for the sound designers; there are a great many, densely-layered aquatic sounds in almost every part of the game. Yet, they are only aquatic in nature because of visual cues in the game; you know, as you are walking through the flooded halls of the Neptune’s Bounty (the third level of BioShock), that the sounds you are hearing are those of water. An attempt to separate the sound of water from its source, meaning and effect may seem pointless, but it isn’t terribly difficult. Concentrating on the sound of footsteps in the water in particular – closing one’s eyes if necessary – can reveal sonic characteristics that exist independently of its source within the game environment, or effect of signalling that your avatar’s foot has hit the surface of the water. Reduced listening will reveal two concurrent bursts of stochastic noise occurring at rhythmic intervals, and the quick rises and falls in pitch will stand out in the mix. If you listen a certain way, and tune in to certain frequency bands, this particular sound object might even resemble the timbres we associate with a hand clap. The sound need not be recorded, looped, and arranged into an overall mix composed of similarly processed sounds (as in Schaeffer’s musique concrète) – the conditions for reduced listening are already in place. Galloway’s description of his ‘first quadrant of gamic action’ is useful:

“The first quadrant is about the machinic phylum and the vitality of pure matter. Consider Yu Suzuki’s Shenmue. One plays Shenmue by participating in its process. Remove everything and there is still action, a gently stirring rhythm of life. There is a privileging of the quotidian, the simple. As in the films of Yasujiro Ozu, the experience of time is important… There is a slow, purposeful accumulation of experiences.” (Galloway 2006, pp. 8-10)

This sense of subtle movement, of inevitability, is at the heart of the everyday experience of a game; some, like The Elder Scrolls IV: Oblivion (2006) emphasize this aspect of gamic experience, but it is present in all games. If you stand still, for a moment, you’ll notice that sounds loop and repeat themselves irrespective of everything else in the game. You need not even initiate a player action (as I did in my example of footsteps) to be in an acousmatic situation in a game. Spending a few hours in BioShock, you’ll hear the same sound objects many times – you simply have to shift your attention to appreciate their unique sonic qualities. Furthermore, the fact that contemporary games such as BioShock and Metal Gear Solid 2 (2001, which Stockburger uses as the focus of his discussion) feature sounds whose spectral patterning is similar to their real-life counterparts does not mean they are inherently tied to those objects; listening to ‘jumping sounds’ of the 8-bit era of consoles (Sega Master System, Nintendo Entertainment System and the Atari 7800) would be a useful exercise here. They were invariably made of a rising synthesized tone, but the player would only associate this rising tone with the act of ascension by way of a visual connection (and, later, by convention). Given the simplicity of the sound, it would be very easy to close one’s eyes and forget this association altogether; the unvarying sonic characteristics would be easy to describe in their own right. The sounds of BioShock can also be heard in this way, but their increased complexity means that the task of reduced listening requires proportionately more work.

To date, there are few games that I have encountered – commercial or independent – that actually incorporate an attentiveness to sound, much less reduced listening, into the gameplay. Generally, games that feature moments in which one’s auditive awareness is heightened only achieve the result of promoting casual listening, for they focus the player’s attention on a sound’s source, meaning and effect. The first-person horror genre in particular is guilty of this: in F.E.A.R (2006), there are frequent moments in which the lights turn off, plunging the player into total darkness. True to the name of the game, one might be unsettled by distant (or shockingly close) scurries, and other sounds of movement. Then, in a predictable replication of a Hollywood horror film convention, the lights will snap back on; an enemy has appeared, dangerously close, and combat resumes. The player’s attention to sound fades away; the case of the mysterious sound is solved. Besides being an overused cliché with a long history in the cinema, this gameplay mechanic, which is common in Doom 3 (2004) as well, undermines the acousmatic experience. The experience described above may create a sense of palpable tension, but the opportunity for reduced listening is lost, and with it the goes chance for the gamer to identify the timbral characteristics of the soundscape that inherently cause tension, and why they might do so. Part of the work of the sound designer, who might have put a great deal of thought into the inherent qualities of their sonic materials, is wasted. This might be one of the reasons behind Stockburger’s dismissal of reduced listening – that gameplay so often disallows it. The possibility of games that not only allow, but actually reward, the act of reduced listening (just as adventure games reward attentiveness to visual composition) is potentially interesting, heretofore unexplored territory for the medium. Until such games exist, reduced listening can only be undertaken purposefully by the adventurous player, as an augmentation of everyday gamic experience. At the same time, even if they haven’t a mind to actually incorporate reduced listening into the core design of a game, developers can learn from the enterprise for, as Chion suggests, it “has the enormous advantage of opening up our ears and sharpening our power of listening” (1994, p. 31).

Beyond taxonomy

There is another, inherently flawed premise in Stockburger’s auditive analysis, quite apart from any claims he makes regarding the nature of sound objects, and reduced listening. It is a problem that suggests a lack of foresight regarding the direction in which games are heading and, unlike reduced listening (which would require a conscious effort on the part of game developers), this future direction is purely technological; inevitable, it seems. The change from software graphics processing to hardware driven processing heralded an astonishing period of evolution in video game graphics, and the technological developments have shown no sign of slowing; a game released in one particular year will make a game of the previous year look positively dated. Unfortunately, there have been fewer commercial incentives for sound in games to evolve at such a rapid pace. Perhaps it is because sounds are, quite literally, harder to show off – screenshots are a powerful promotional tool for upcoming games, but I cannot imagine many publishers releasing sound samples from their game onto the internet, any more than I would expect gamers to appreciate them in the same way as pictures.

Strangely enough, this technological development in sound to which I am referring will require developers to visit the past, when game soundtracks were synthesized entirely by sound chips in home consoles and computers. The engines of most contemporary first-person games produce and process sounds by referring to libraries of pre-recorded sounds and, as a result of this, there is no such thing as unintentional sound in games; everything is scripted, so there is always an upward limit on the number of sounds that can occur at any one time. The only reason it makes sense for Stockburger to impose a taxonomic structure on video game sound is because there are few enough sounds, all with specific purposes, that it is actually possible. The task of categorizing sounds based on their function would be infinitely more difficult for the streets of Sydney, I would imagine. What would a zone sound object be (the type in Stockburger’s analysis that accounts for in-game ambient sound), when ambient sound itself consists of innumerable ‘effect’ and ‘speech’ sound objects, coalescing and canceling at the same time? Indeed, BioShock comes very close to this effect at times; in situations where there are many sounds occurring concurrently (as in combat) the wetness and long decay of the game’s reverberation effect gives the impression of sound effects that gradually merge into the ambient soundscape. Call of Duty 4 (2007) is an even better example. The sounds of distant gunshots are attenuated so as to give the impression of distance, and they form part of the ambient sound, serving to “identify a particular locale through their pervasive and continuous presence” (Chion 1994, p. 75). They help create an impression of danger at first, but as the player moves closer to their source, the spectral attenuation of the sounds gradually changes until they take on the status of ‘effect sound objects’. Stockburger’s taxonomy is still useful then, granted one allows for the categories to be fluid, and accepts that many sounds will move freely between them. It is only an attribute of the current engine technology that makes game soundtracks ontologically different to the real-life soundscape; Stockburger’s auditive analysis relies very much on technology as the agent that shapes gamic experience when, as I have argued in referring to Myron Krueger, embodiment is much more important (Hansen 2006, p. 28).

Inevitably, the way video game soundtracks are composed and generated will change, just as graphics have, from playback engines to synthesis engines; if not driven by artistic interest, it will occur as a matter of computational efficiency. Currently, the manner in which game engines produce sound resembles the short-lived age of full motion video (FMV) gameplay, best demonstrated in games such as Rebel Assault (1993) and Mad Dog McCree (1993). Sounds are called upon as largely static objects from libraries, and facilitate little more player interactivity with the soundscape than FMV gameplay allowed with graphics. By interactivity, I mean the extent to which an in-game actor can be fundamentally altered by some sort of player input, such as a key press, or voice command. The FMV gameplay era was short-lived because it was notorious for producing games with severely limited interactivity; aside from any nostalgic value, Rebel Assault has very few redeeming features. Conversely, contemporary graphics engines, such as the Source Engine or Unreal Engine 3.0, can be said to synthesize graphics. While, at some level, they still call upon blocks of data within game directories, they operate on a more atomic level than sound engines. Textures are stitched together, polygons are rendered, and architecture is assembled, all in real time.

Even though the incorporation of digital synthesis into game engines might be years away, there has been some recent research that suggests technological and aesthetic benefits of the technique. In a paper that proposes a specific method of sound synthesis for virtual environments, Miner and Caudell draw attention to the problems with relying sole on digitized sounds (as contemporary game engines still do), for they “are static and do not dynamically change in response to user actions or to changes within a virtual environment” (2002, p. 493). Since the release of the research paper, games like Call of Duty 4 have challenged this assumption by incorporating dynamic, real-time attenuation of sounds to simulate distance, as well as to impose subtle variations onto digital waveforms, thus lessening the chance of their repetition. However, the three greatest limitations of engines based on digitized sounds still remain:

1. every sound must be scripted, leaving little room for unpredictable, stochastic events
2. real-time attenuation is being applied to a very small number of static waveforms, and
3. there is a fundamental separation between player actions and the sounds these actions produce, because the player merely triggers sounds, but doesn’t have any direct control of their timbral characteristics.

Miner and Caudell’s research is interested primarily in the first limitation; their synthesis technique is designed to produce perceptually believable stochastic, nonpitched sounds for virtual environments. Such sounds are pervasive in our real-life environment, yet missing entirely from contemporary game soundscapes. Rain, for example, which has a variable but characteristically identifiable structure, is always simulated in games by a looped and unvarying pre-recorded sound (p. 493). This first limitation will certainly be of interest to composers once virtual synthesis technology exists. In the meantime, however, there is an entirely different sort of stochastic probability that affects sound in games; one that forms a very prominent part of the everyday experience of sound.

Locating the post-digital in contemporary games

Kim Cascone used the term ‘post-digital’ to explain the aesthetic of failure in contemporary electronic music, and the interest of composers in exposing the inherent flaws of digital software. In his own words, he uses the term “because the revolutionary period of the digital information age has surely passed” (2000, p. 12). While it cannot be said that there is a post-digital aesthetic in games, the sonic hallmarks of glitch music have been a characteristic feature of video games for decades. It is only in recent years, though, as games have become increasingly large and complex, that digital failure has become an unavoidable feature of gameplay, especially for PC gamers.

Latency, more commonly knows as ‘lag’, is to an avid online gamer a thoroughly destructive experience. Similarly, choppy frame rates (the ‘slideshow’ effect), freezes, crashes, texture popping, screen tearing and sound cut-outs can disrupt the experience of playing a game. Galloway includes such occurrences in his fourth quadrant of gamic action (which he calls ‘nondiegetic machine acts’; actions that are “performed by the machine and integral to the entire experience of the game but not contained within the narrow conception of the world of gameplay” (2006, p. 28). Returning to the theory of ludology for a moment, Johan Huizinga’s writing on the nature of ‘play’ seems appropriate to mention, here. According to Huizinga, play “proceeds within its own proper boundaries of time and space according to fixed rules and in an orderly manner” (1950, p. 28). The presence of actions that completely undermine the rules of a video game (and, in the most extreme case, destroying the game by crashing or freezing) forces us to question this particular conception of ‘play’ when discussing games within the framework of ludology. Regardless of how destructive such digital failures might seem, it is nonetheless interesting that they break through into the experience of a game at the most unexpected and inopportune of times. World of Warcraft suffers from frequent ‘lag spikes’ (large but brief increases in latency) – in these moments sounds and animations within the game environment will temporarily suspend, stretched overlong until, all of a sudden, every single sound and animation that occurred within the spike will tumble out, resulting in an explosive cacophony of noises and almost certainly, if you were playing against other players, the death of your avatar. The release of Fallout 3 (2008) on the PC was plagued with issues proportionate to the enormous size of the game; you would play for a few minutes before the game suddenly froze, the last instance of sound caught in an ongoing loop – the sort you might encounter in a Squarepusher or Autechre track – and you had no choice but to manually shut down the game. It doesn’t happen all the time, and developers are always working to stamp out such glitches – their reputation rests upon it. To someone who has an interest in glitch music, the sonic artefacts that are created by freezes, glitches, latency spikes and crashes might be of aesthetic interest. Just as reduced listening can be exercised as an alternative way to experience the gamic soundscape, so too can a player open their ears and appreciate the textural qualities of digital failure, as well as the phenomenological shift of being unceremoniously torn from immersion within a game, to becoming suddenly aware of its digital apparatus. There is no post-digital sound aesthetic in games because it isn’t yet being consciously adopted by sound designers as a compositional technique, but it is nonetheless an aesthetic similarity between glitch music and the everyday experience of games.

There is no reason to believe games will ever be free of this aesthetic similarity, either. The technological and commercial forces that push the rapid evolution of video games ensure that new games will always be taxing to all but the most powerful computer systems, sewing the seeds of digital failure despite developers’ attempts to release error-free products. The late-1970s and 1980s left a legacy of game developers frequently reinventing their programming techniques, as Mark Wolf observes when discussing the history of games: “game complexity naturally evolved as programmers attempted to outdo one another” (2003, p. 54). Similar to the 1980s Demoscene, whose artists would vie to encode increasingly complex animations into small parcels of data, game programmers were constantly inventing new ways to squeeze better graphics and performance out of computers. This spirit of competition persists today, and to a much greater extent than two decades ago, given the enormous commercial stakes in the industry. The first-person genre has always been at the cutting edge of these developments – Quake (1996), Half-Life 2 and Crysis (2007) revolutionized game technology upon their respective releases, and punished computers with below-average processing power accordingly. Publishers, and the pressure they exert upon developers to release games before the engines have been fully optimized, have also played a part in ensuring glitches and digital failure remain at the forefront of everyday gamic experience.

Considering all of this, the possibility of games that are purposefully built with un-optimized engines to strain even very powerful computers – games that anticipate and attempt to shape the subsequent digital failures for the sonic results – is potentially interesting, if highly unusual, compositional territory. It would necessarily require a level of open-mindedness on the part of gamers, but no more so than of audiences listening to glitch music.

The atomic soundscape

In raising my concerns with the very premise behind Stockburger’s discussion of the auditive environment in games, I argued that the only reason it possible to categorize different types of game sound objects is because, at the moment, there are very few sounds to talk about. I also mentioned that, due partly to a video games media that fetishizes visual appeal, games are powered by technology that renders graphics by way of a granular process. It is at this juncture that I will bring up my final suggestion for game sound design – granular composition.

Granular composition is not a fundamental change in the way sound is composed for video games; rather, it is an extension of the current method, similar in nature to visual design in games. Therefore, it is feasible as a compositional principle in designing sound for current generation game engines, like Unreal Engine 3.0, but would require more time and effort on the part of the sound designer.

Curtis Roads used the term microsound to describe the previously inaccessible domain of sounds that, due to current digital technology, “now stands at the forefront of compositional interest” (Roads 2001, p.21). He was referring to a technique of composition focused on the micro time scale, “a broad class of sounds that extends from the threshold of timbre perception… up to the duration of short sound objects… [Spanning] the boundary between the audio frequency range… and the infrasonic frequency range” (p. 20-21). While sound could be composed within this time and frequency scale for video games, I am rather proposing a method of composition that is similar in spirit to micro time scale composition; design that occurs at a more atomic level than is currently being practiced by developers, and experienced by gamers.

“Microsound is ubiquitous in the natural world. Transient events unfold all around in the wild” (Roads 2001, p. 21)

An obvious quality in visual design that has changed at a startling rate over the last decade is the attention to subtlety; the gradual shift from large, fully formed objects to small grains. In some cases, this can be taken literally: detailed textures (a feature commonly associated with games based on the Unreal Engine) can only truly be appreciated when a player moves close to the textured object, and allows it to fill the visual field. The clarity of detailed textures, dating as far back as Deus Ex in 2001, can be quite astonishing. Contemporary rendering techniques have added even more detail to textures; bump mapping refers to a range of techniques that encode depth information into a texture, so that it may be realistically affected by both rendered and real-time lighting, in a way that doesn’t put too much stress on average computer systems. High dynamic range lighting (HDR) has been used in modern first-person games to great effect; some games will use the technique to introduce subtle, gradual changes to the lighting in a virtual environment, while others will use it to decimate the detail of textures with pools of perfect whiteness. There are dozens more examples, for texturing, modelling, physics and animation; to truly appreciate the extent to which programmers and designers have embraced subtlety and granularity in visual design, you need only spend a few moments playing such games as Fallout 3, Crysis and Unreal Tournament 3 (2007).

Sound design in these modern first-person games can be equally impressive, but only a few examples hint at the true potential of atomic sound composition. BioShock, as I have already discussed, was a great achievement in sound design. Part of its success owed to the fact that actors in the game were assigned numerous layers of sound, all of which could be triggered and attenuated independently of one another. In an interview with a BioShock community site Emily Ridgway, the game’s audio director, provides an insight into the astonishing level of sonic detail in the game:

“The security bot sounds were insanely complicated from an implementation standpoint. There is a sound for the blades moving through the air, there is a sound for the motor that seamlessly winds-up, loops, and ends in all the right places. Said motor sounds also dynamically pitch bend depending on how hard the bot is accelerating… It has sounds for banging into things while it’s flying. It has sounds for when the security bot is half damaged, quarter damaged, and dead. It has a sound to let you know if you can hack it or not. And also, all of the above sounds change depending on if the security bot is friendly to you or not. We don’t expect people to be able to hear all that all the time, but for us each little touch seems to make the world that much more believable.”

(Excerpt from an online interview at the Cult of Rapture webpage, 2007; see references for address)

The last point Ridgway makes is perhaps the most important. Just as the almost ridiculous level of detail in contemporary game textures, the complexity of the sound design in BioShock can only be appreciated by a discerning listener; perhaps by reduced listening, as I suggested earlier. However, even without the benefit of an appreciative listener, the micro-temporal variations in these multi-layered, multi-attenuated sounds do result in perceptible timbral variations. Roads concludes as much in his book on microsound: “Operations on one time scale generate structures which may be perceived on other time scales… To cite an example; a gradual change in particle durations results in timbre variations on a higher time scale” (2001, p. 331). BioShock’s soundscape is believable, to use Ridgway’s words, because it effectively puts the gamer in an environment populated with transient sound events. This adds another dimension to gamic embodiment; just as embodied spatial configuration of sounds can help immerse us in a virtual environment, so too can we be immersed in a game by subtle timbral configurations that appeal to the limits of our evolutionarily acquired hearing.

This compositional approach could variously be called atomic, micro or granular sound composition. Roads refers to it as ‘bottom-up’, “since the composition takes its shape from microsonic interactions”, which he contrasts with a ‘top-down’ strategy “in which a composer fills in a preplanned form” (p. 331). My only suggestion is that sound designers adopt this approach for all sounds within the gamic environment, for actors both seen and unseen, rather than just objects of interest. The individual leaves of plants, the animated fabrics of people’s clothing, and the creaks of all the wooden beams that form a building – all can be given life with the same amount sonic detail that Ridgway described. If gamers have taken the time to appreciate the minuscule details of concrete textures and the subtle play of real-time light on the surface of water, they would certainly be willing to appreciate the same detail in the auditive environment. Arguably, these quotidian moments (as Galloway identified them) are the most important to populate with sound, for it is only in these moments that one can forget all other topographic aspects of play – of killing enemies, and unlocking doors, and solving puzzles – and appreciate the everyday in gaming.

Time (and, in the case of commercial games, money) are considerable factors, but BioShock has demonstrated that thoughtful, granular sound composition can have a perceptible impact on the experience of sound, and sound designers have much to gain from investing their energy into creating equally, if not more detailed soundscapes. There are musical possibilities for granular composition as well; BioShock’s sound is dissonant, and largely textural, but it would be possible to build a game’s musical soundtrack from physically affected objects within a game environment, so that their movements create harmonic and percussive effects. The composition Quick be the Feet that Run to Mischief (2008) by Snawklor comes to mind when trying to imagine the sonic possibilities, with the added consideration that the musical sounds could be manipulated by the player in real-time, and moved around in virtual space. Effectively, a game’s musical composer would become an enabler of compositions, and of musical performances. They could design sound in a way that allows the everyday gamer to exercise an entirely different form of creativity than the type that Certeau proposed for the everyday walker on the street; artistic creativity, as well as interpretive. Without necessarily involving the manipulations of specific sounds on a micro time scale, sound designers should continue to apply the principle of granularity in composing for virtual environments.

Conclusion: Returning to the street

In writing this essay, I have attempted to address a void in contemporary video game theory that, as someone who spends a lot of time playing games, seems very obvious. Deconstructing games, thinking about the different parts that developers have assembled into virtual environments, didn’t allow me to approach games on my own terms. I have always felt, yet only now articulated, that games allow for highly idiosyncratic experiences. Despite the unique manner in which each gamer will negotiate a virtual environment, I have proposed that there is a fundamental similarity between all the different ways of experiencing games; a way of subverting rules and topographic order, a way of actualizing and creatively manipulating phenomena that resembles Michel de Certeau’s walker on the city streets. Using this as a starting point when thinking about video games allows us to understand the importance of human perception and embodiment in navigating and, as I have described, listening to virtual worlds.

Approaching gamic sound from a topographic perspective, as Stockburger does, is valuable insofar as it reveals similarities between filmic and gamic conventions. Stockburger’s sound objects are also useful in understanding sonic composition of many contemporary games. However, the problem with imposing a taxonomic structure onto game sound is that it places too much responsibility upon the game engine as the agent that shapes auditive experience. His taxonomy describes sounds before they reach the player’s ear, before the vibrations of a sound have had any effect on the gamer. Furthermore, in distancing his sound objects from those of Pierre Schaeffer and dismissing the possibility of reduced listening, Stockburger ignored the notion of the acousmatic experience when, in actuality, the conditions for such as experience of sound are well in place, for the adventurous gamer. Even if the application of reduced listening is of questionable merit to gaming (I argued that it is a very useful exercise), it is important for theorists and developers to consider that the everyday experience of a game is also an acousmatic experience, and that games can facilitate and reward a heightened attentiveness to the sonic environment.

I also proposed a number of approaches to sound design in games based on prominent aspects on everyday gamic experience. Digital failure is an unavoidable, conventionally undesirable feature of gaming, manifesting itself as crashes, freezes, latency spikes and lag. Without going as far as claiming that there is a deliberate failure-aesthetic that developers incorporate into games, it is conceivable that games could be made whose sound composition reflects this most salient aspect of gaming. I also, very briefly, suggested the role that digital synthesis will play in the future of game sound before, finally, discussing the benefits of granular approaches to sound design. There are many more aspects of the everyday experience of gaming, certainly not limited to sound, that deserve discussion. Rather than provide exhaustive accounts of the results of reduced listening in a game, or the future of digital sound synthesis, my intention is to bring embodied, human experience to the forefront of video game theory; not only for the purpose of better understanding our engagement with games, or to conceive alternative ways to compose sound for games, but also so we might think of games as a reflection of the way we engage with the world. Hopefully, it will be a part of the larger project of devising a more complete phenomenology of gaming to contrast some of the dominant approaches in contemporary video game studies.

Bibliography:

2K Boston & 2K Australia 2007, BioShock, 2K Games.

American Laser Games, Inc. 1993, Mad Dog McCree, IBM.

Bethesda Game Studios 2006, The Elder Scrolls IV: Oblivion, 2K Games & Bethesda Softworks.

Bethesda Game Studios 2008, Fallout 3, Bethesda Softworks.

Blizzard Entertainment 2004, World of Warcraft, Vivendi Universal.

Caillois, R. 1979, Man, Play, and Games, Schocken Books, New York.

Cascone, K. 2000, ‘The Aesthetics of Failure: “Post-Digital” Tendencies in Contemporary Computer Music’, Computer Music journal, vol. 24, no. 4, pp. 12-18.

de Certeau, M. 1984, ‘Walking in the City’ from The Practice of Everyday Life, Berkeley: University of California Press.

Chion, M. 1994, Audio-Vision, Columbia University Press, New York.

Cult of Rapture 2007, Sounds of Rapture, viewed 14/10/2008 2008 <http://www.2kgames.com/cultofrapture/soundsofrapture.html>.

Crytek 2007, Crysis, Electronic Arts.

Dokic, J. 2007, ‘Two Ontologies of Sound’, The Monist, vol. 90, no. 3, p. 391(312).

Epic Games 2007, Unreal Tournament 3, Midway Games.

Filiciak, M. 2003, ‘Hyperidentities: Postmodern Identity Patterns in Massively Multiplayer Online Role-Playing Games’, in Wolf, M.J.P & Perron, B. (eds.), The Video Game Theory Reader, Routledge, New York.

Funcom 1999, The Longest Journey, Empire Interactive Entertainment.

Galloway, A.R. 2006, Gaming: Essays on Algorithmic Culture, University of Minnesota Press, Minneapolis.

Hansen, M. 2006, Bodies in Code, Routledge, New York.

Hudson Soft 1999, Mario Party, Nintendo.

Huizinga, J. 1950, Homo Ludens: A Study of the Play-Element in Culture, Beacon, Boston.

id Software 1993, Doom, id Software.

id Software 1996, Quake, id Software.

id Software 2004, Doom 3, id Software & Activision.

Infinity Ward 2007, Call of Duty 4: Modern Warfare, Activision.

Ion Storm Inc. 2000, Deus Ex, Eidos Interactive.

Irrational Games & Looking Glass Studios 1999, System Shock 2, Electronic Arts.

Juul, J. 2005, Half-real: Video Games between Real Rules and Fictional Worlds, MIT Press, Cambridge.

Konami Computer Entertainment Japan 2001, Metal Gear Solid 2: Sons of Liberty, Konami.

LucasArts 1993, Star Wars: Rebel Assault, LucasArts.

LucasArts 1997, The Curse of Monkey Island, LucasArts.

Miner N.E. & Caudell T.P. 2002, ‘A Wavelet Synthesis Technique for Creating Realistic Virtual Environment Sounds, Presence, vol. 11, no. 5, pp. 493-507.

Monolith Productions 2006, F.E.A.R., Vivendi Universal.

Nudds, M. 2001, ‘Experiencing the Production of Sounds’, European Journal of Philosophy, vol. 9, no. 2, pp. 210-229.

Roads, C. 2001, Microsound, MIT Press, Cambridge.

Schaeffer, P. 1966, Traité des Objets Musicaux Seuil, Paris.

Schafer, R. M. 1977, The Soundscape, Destiny Books, Rochester.

Snawklor 2008, Quick be the Feet that Run to Mischief, Outer.

Stockburger, A. 2003, ‘The Game Environment from an Auditive Perspective’, paper presented to the DIGRA Level Up Conference, University of Utrecht, Holland, 2003.

Swalwell, M. 2002, Playing the Senses: “Better than playing (PC Games) with yourself”, in Aesthetics and Hyper/aesthetics: Rethinking the Senses in Contemporary Media Contexts, University of Technology, Sydney, pp. 84-138.

Teeny Weeny Games & Perfect 10 Productions 1995, Discworld, Psygnosis.

Troika Games 2004, Vampire: The Masquerade – Bloodlines, Activision.

Valve Corporation 1999, Counter-Strike, Vivendi Universal.

Valve Corporation 2004, Half-Life 2, Valve Corporation.

Wolf, M.J.P. 2003, ‘Abstraction in the Video Game’, in Wolf, M.J.P & Perron, B. (eds.), The Video Game Theory Reader, Routledge, New York.