Things – Page 14 – medieninitiative

Visualizing Digital Seriality, Or: All Your Mods Are Belong to Us!

In this post, I want to outline some ongoing work in progress that I’ve been pursuing as part of my postdoctoral research project on seriality as an aesthetic form and as a process of collectivization in digital games and gaming communities. The larger context, as readers of this blog will know, is a collaborative project I am conducting with Andreas Jahn-Sudmann of the Freie Universität Berlin, titled “Digital Seriality” — which in turn is part of an even larger research network, the DFG Research Unit “Popular Seriality–Aesthetics and Practice.” I’ll touch on this bigger picture here and there as necessary, but I want to concentrate more specifically in the following on some thoughts and research techniques that I’ve been developing in the context of Victoria Szabo’s “Historical & Cultural Visualization” course, which I audited this semester at Duke University. In this hands-on course, we looked at a number of techniques and technologies for conducting digital humanities-type research, including web-based and cartographic research and presentation, augmented and virtual reality, and data-intensive research and visualization. We engaged with a great variety of tools and applications, approaching them experimentally in order to evaluate their particular affordances and limitations with respect to humanities work. My own engagements were guided by the following questions: How might the tools and methods of digital humanities be adapted for my research on seriality in digital games, and to what end? What, more specifically, can visualization techniques add to the study of digital seriality?

I’ll try to offer some answers to these questions in what follows, but let me indicate briefly why I decided to pursue them in the first place. To begin with, seriality challenges methods of single-author and oeuvre or work-centric approaches, as serialization processes unfold across oftentimes long temporal frames and involve collaborative production processes — including not only team-based authorship in industrial contexts but also feedback loops between producers and their audiences, which can exert considerable influence on the ongoing serial development. Moreover, such tendencies are exacerbated with the advent of digital platforms, in which these feedback loops multiply and and accelerate (e.g. in Internet forums established or monitored by serial content producers and, perhaps more significantly, in real-time algorithmic monitoring of serialized consumption on platforms like Netflix), while the contents of serial media are themselves subject to unprecedented degrees of proliferation, reproduction, and remix under conditions of digitalization. Accordingly, an incredible amount of data is generated, so that it is natural to wonder whether any of the methods developed in the digital humanities might help us to approach phenomena of serialization in the digital era. In the context of digital games and game series, the objects of study — both the games themselves and the online channels of communication around which gaming communities form — are digital from the start, but there is such an overwhelming amount of data to sort through that it can be hard to see the forest for the trees. As a result, visualization techniques in particular seem like a promising route to gaining some perspective, or (to mix metaphors a bit) for establishing a first foothold in order to begin climbing what appears an insurmountable mountain of data. Of particular interest here are: 1) “distant reading” techniques (as famously elaborated by Franco Moretti), which might be adapted to the objects of digital games, and 2) tools for network analysis, which might be applied in order to visualize and investigate social formations that emerge around games and game series.

Before elaborating on how I have undertaken to employ these approaches, let me say a bit more about the framework of my project and the theoretical perspective on digital seriality that Andreas Jahn-Sudmann and I have developed at greater length in our jointly authored paper “Digital Seriality.” Our starting point for investigating serial forms and processes in games and gaming communities is what we call “inter-ludic seriality” — that is, the serialization processes that take place between games, establishing series such as Super Mario Bros. 1, 2, 3 etc. or Pokemon Red and Blue, Gold and Silver, Ruby and Sapphire, Black and White etc. For the most part, such inter-ludic series are constituted by fairly standard, commercially motivated practices of serialization, expressed in sequels, spin-offs, and the like; accordingly, they are a familiar part of the popular culture that has developed under capitalist modernity since the time of industrialization. Thus, there is lots of continuity with pre-digital seriality, but there are other forms of seriality involved as well.

“Intra-ludic seriality” refers to processes of repetition and variation that take place within games themselves, for example in the 8 “worlds” and 32 “levels” of Super Mario Bros. Here, a general framework is basically repeated while varying and in some cases increasingly difficult tasks and obstacles are introduced as Mario searches for the lost princess. Following cues from Umberto Eco and others, this formula of “repetition + variation” is taken here as the formal core of seriality; games can therefore be seen to involve an operational form of seriality that is in many ways more basic than, while often foundational to, the narrative serialization processes that they also display.

Indeed, this low-level seriality is matched by higher-level processes that encompass but go beyond the realm of narrative — beyond even the games themselves. What we call “para-ludic seriality” involves tie-ins and cross-overs with other media, including the increasingly dominant trend towards transmedia storytelling, aggressive merchandising, and the like. Clearly, this is part of an expanding commercial realm, but it is also the basis for more.

There is a social superstructure, itself highly serialized, that forms around or atop these serialized media, as fans take to the Internet to discuss (and play) their favorite games. In itself, this type of series-based community-building is nothing new. In fact, it may just be a niche form of a much more general phenomenon that is characteristic for modernity at large. Benedict Anderson and Jean-Paul Sartre before him have described modern forms of collectivity in terms of “seriality,” and they have linked these formations to serialized media consumption and those media’s serial forms — newspapers, novels, photography, and radio have effectively “serialized” community and identity throughout the nineteenth and twentieth centuries.

Interestingly, though, in the digital era, this high-level community-building seriality is sometimes folded into an ultra low-level, “infra-ludic” level of seriality — a level that is generally invisible and that takes place at the level of code. (I have discussed this level before, with reference to the BASIC game Super Star Trek, but I have never explicitly identified it as “infra-ludic seriality” before.) This enfolding of community into code, broadly speaking, is what motivates the enterprise of critical code studies, when it is defined (for example, by Mark Marino) as

an approach that applies critical hermeneutics to the interpretation of computer code, program architecture, and documentation within a socio-historical context. CCS holds that lines of code are not value-neutral and can be analyzed using the theoretical approaches applied to other semiotic systems in addition to particular interpretive methods developed particularly for the discussions of programs. Critical Code Studies follows the work of Critical Legal Studies, in that it practitioners apply critical theory to a functional document (legal document or computer program) to explicate meaning in excess of the document’s functionality, critiquing more than merely aesthetics and efficiency. Meaning grows out of the functioning of the code but is not limited to the literal processes the code enacts. Through CCS, practitioners may critique the larger human and computer systems, from the level of the computer to the level of the society in which these code objects circulate and exert influence.

Basically, then, the questions that I am here pursuing are concerned with the possibilities of crossing CCS with DH — and with observing the consequences for a critical investigation of digital game-based seriality. My goal in this undertaking is to find a means of correlating formations in the high-level superstructure with the infra-ludic serialization at the level of code — not only through close readings of individual texts but by way of large collections of data produced by online collectives.

As a case study, I have been looking at ROMhacking.net, a website devoted to the community of hackers and modders of games for (mostly) older platforms and consoles. “Community” is an important notion in the site’s conception of itself and its relation to its users, as evidenced in the site’s “about” page:

ROMhacking.net is the innovative new community site that aggressively aims to bring several different areas of the community together. First, it serves as a successor to, and merges content from, ROMhacking.com and The Whirlpool. Besides being a simple archive site, ROMhacking.net’s purpose is to bring the ROMhacking Community to the next level. We want to put the word ‘community’ back into the ROMhacking community.

The ROMhacking community in recent years has been scattered and stagnant. It is our goal and hope to bring people back together and breathe some new life into the community. We want to encourage new people to join the hobby and make it easier than ever for them to do so.

Among other things, the site includes a vast collection of Super Mario Bros. mods (at the time of writing, 205 different hacks, some of which include several variations). These are fan-based modifications of Nintendo’s iconic game from 1985, which substitute different characters, add new levels, change the game’s graphics, sound, or thematic elements, etc. — hence perpetuating an unofficial serialization process that runs parallel to Nintendo’s own official game series, and forming the basis of communal formations through more or less direct manipulation of computer code (in the form of assembly language, hex code, or mediated through specialized software platforms, including emulators and tools for altering the game). In other words, the social superstructure of serial collectivity gets inscribed directly into the infra-ludic level of code, leaving traces that can be studied for a better understanding of digital seriality.

But how should we study them? Even this relatively small sample is still quite large by the standards of a traditional, close reading-based criticism. What would we be looking for anyway? The various mods are distributed as patches (.ips files) which have to be applied to a ROM file of the original game; the patches are just instruction files indicating how the game’s code is to be modified by the computer. As such, the patch files can be seen, rather abstractly, as crystallizations of the serialization process: if repetition + variation is the formal core of seriality, the patches are the records of pure variation, waiting to be plugged back into the framework of the game (the repeating element). But when we do plug it back in, then what? We can play the game in an emulator, and certainly it would be interesting — but extremely time-consuming — to compare them all in terms of visual appearance, gameplay, and interface. Or we can open the modified game file in a hex editor, in which case we might get lucky and find an interesting trace of the serialization process, such as the following:

Similar to Super Star Trek with its REM comments documenting its own serial and collective genesis, here we find an embedded infratext in the hexcode of “Millennium Mario,” a mod by an unknown hacker reportedly dating back to January 1, 2000. Note, in particular, the reference to a fellow modder, “toma,” the self-glorifying “1337” comment, and the skewed ASCII art — all signs of a community of serialization operating at a level subterranean to gameplay. But this example also demonstrates the need for a more systematic approach — as well as the obstacles to systematicity, for at stake here is not just code but also the software we use to access it and other “parergodic” elements, including even the display window size or “view” settings of the hex editor:

In a sense, this might be seen as a first demonstration of the importance of visualization not only in the communication of results but in the constitution of research objects! In any case, it clearly establishes the need to think carefully about what it is, precisely, that we are studying: serialization is not imprinted clearly and legibly in the code, but is distributed in the interfaces of software and hardware, gameplay and modification, code and community.

Again, I follow Mark Marino’s conception of critical code studies, particularly with respect to his broad understanding of the object of study:

What can be interpreted?

Everything. The code, the documentation, the comments, the structures — all will be open to interpretation. Greater understanding of (and access to) these elements will help critics build complex readings. In “A Box Darkly,” discussed below, Nick Montfort and Michael Mateas counter Cayley’s claim of the necessity for executability, by acknowledging that code can be written for programs that will never be executed. Within CCS, if code is part of the program or a paratext (understood broadly), it contributes to meaning. I would also include interpretations of markup languages and scripts, as extensions of code. Within the code, there will be the actual symbols but also, more broadly, procedures, structures, and gestures. There will be paradigmatic choices made in the construction of the program, methods chosen over others and connotations.

In addition to symbols and characters in the program files themselves, paratextual features will also be important for informed readers. The history of the program, the author, the programming language, the genre, the funding source for the research and development (be it military, industrial, entertainment, or other), all shape meaning, although any one reading might emphasize just a few of these aspects. The goal need not be code analysis for code’s sake, but analyzing code to better understand programs and the networks of other programs and humans they interact with, organize, represent, manipulate, transform, and otherwise engage.

But, especially when we’re dealing with a large set of serialized texts and paratexts, this expansion of code and the attendant proliferation of data exacerbates our methodological problems. How are we to conduct a “critical hermeneutics” of the binary files, their accompanying README files, the ROMhacking website, and its extensive database — all of which contain information relevant to an assessment of the multi-layered processes of digital seriality? It is here, I suggest, that CCS can profit from combination with DH methods.

The first step in my attempt to do so was to mine data from the ROMhacking website and paratexts distributed with the patches and to create a spreadsheet with relevant metadata (you can download the Excel file here: SMB-Hacks-Dec1). On this basis, I began trying to analyze and visualize the data with Tableau. But while this yielded some basic information that might be relevant for assessing the serial community (e.g. the number of mods produced each year, including upward and downward trends; a list of the top modders in the community; and a look at trends in the types of mods/hacks being produced), the visualizations themselves were not very interesting or informative on their own (click on the image below for an interactive version):

How could this high-level metadata be coordinated with and brought to bear on the code-level serialization processes that we saw in the hexcode above? In looking for an answer, it became clear that I would have to find a way to collect some data about the code. The mods, themselves basically just “diff” files, could be opened and compared with the “diff” function that powers a lot of DH-based textual analysis (for example, with juxta), but the hexadecimal code that we can access here — and the sheer amount of it in each modded game, which consists of over 42000 bytes — is not particularly conducive to analysis with such tools. Many existing hex editors also include a “diff” analysis, but it occurred to me that it would be more desirable to have a graphical display of differences between the files in order to see the changes at a glance. My thinking here was inspired by hexcompare, a Linux-based visual diff program for quickly visualizing the differences between two binary programs:

However, the comparison here is restricted to local use on a Linux machine, and it only considers two files at a time. If this type of analysis is to be of any use for seriality studies, it will have to assess a much larger set of files and/or automate the comparison process. This is where Eric Monson and Angela Zoss from Visualization & Information Services at Duke University came in and helped me to develop an alternative approach. Eric Monson wrote a script that analyzes the mod patch files and records the basic “diff” information they contain: the address or offset at which they instruct the computer to modify the game file, as well as the number of bytes that they instruct it to write. With this information (also recorded in the Excel file linked to above), a much more useful and interactive visualization can be created with Tableau (click for an interactive version):

Here, Gannt charts are used to represent the size and location of changes that a given mod makes to the original Mario game; it is possible to see a large number of these mods at a single glance, to filter them by year, by modder, by title, or even size (some mods expand the original code), etc., and in this way we can begin to see patterns emerging. Thus, we bring a sort of “distant reading” to the level of code, combining DH and CCS. (Contrast this approach with Marino’s 2006 call to “make the code the text,” which despite his broad understanding of code and acknowledgement that software/hardware and text/paratext distinctions are non-absolute, was still basically geared towards a conception of CCS that encouraged critical engagements of the “close-reading” type. As I have argued, however, researching seriality in particular requires that we oscillate between big-picture and micro-level analyses, between distant readings of larger trends and developments and detailed comparisons between individual elements or episodes in the serial chain.)

But to complete this approach, we still need to correlate this code-based data with the social level of online modding communities. For this purpose, I used Palladio (a tool explicitly designed for DH work by the Humanities + Design lab at Stanford) to graph networks on the basis of metadata contained in Readme.txt files.

Here, I have mapped the references (“shout-outs,” etc) that modders made to one another in these paratexts, thus revealing a picture of digital seriality as an imagined community of modders.

Here, on the other hand, I have mapped references from paratextual materials associated with individual mods to various online communities that have come and gone over the years. We see early references to the defunct TEKhacks, by way of Zophar’s Domain, Acmlm’s and Insectduel’s boards, with more recent references to Romhacking.net, the most recent community site and the one that I am studying here.

As an example of how the social network and code-level analyses might be correlated, here I’ve filtered the network graph to show only those modders who refer in their paratexts to Super Mario Bros. 3 (hence bringing inter-ludic seriality to bear on their para- and infra-ludic interventions). The resulting graph reveals a small network of actors whose serializing activity involves mixing and referencing between SMB1 and SMB3, as well as between each other. The Tableau screenshot on the right then selects just these modders and reveals possible similarities and sites of serialization (for closer scrutiny with hexcompare or tools derived from the modding community itself). For example, we find that the modder AP’s SMB3-inspired patches from September 2005 and flamepanther’s SMB DX patches from Oct 2005 exhibit traces of possible overlap that deserve to be looked at in detail. The modder insectduel’s After World 8 (a mod that is referenced by many in the scene) from February 2006 shares large blocks around 31000-32000 bytes with many of the prolific modder Googie’s mods (which themselves seem to exhibit a characteristic signature) from 2004-2006. Of course, recognizing these patterns is just the beginning of inquiry, but at least it is a beginning. From here, we still have to resort to “close reading” techniques and to tools that are not conducive to a broad view; more integrated toolsets remain to be developed. Nevertheless, these methods do seem promising as a way of directing research, showing us where to look in greater depth, and revealing trends and points of contact that would otherwise remain invisible.

Finally, by way of conclusion and to demonstrate what some of this more detailed work looks like, I’d like to return to the “Millennium Mario” mod I considered briefly above. As we saw, there was an interesting infratextual shoutout and some ASCII art in the opening section of the hexcode. With Tableau, we can filter the “diff” view to display only those mods that exhibit changes in the first 500 bytes of code, and to map that section of code in greater resolution (this is done with the slider in the bottom right corner, marked “Start” — referring to the byte count at which a change in the game starts):

Here we find two distinctive (visual) matches: viz. between “Millennium Mario” and Raysyde’s “Super Mario Bros. – Remix 2” from 1999, and between ATA’s “Super Mario Bros. – Yoshi’s Quest” and Krillian’s “Mario Adventure 2,” both from 2000. The latter two mods, while clearly different from the former two, also exhibit some overlap in the changes made to the first 20 or so bytes, so it will be interesting to compare them as well.

Now we can use hexcompare for finer analysis — i.e. to determine if the content of the changed addresses is also identical (the visual match only tells us that something has been changed in the same place, not whether the same change has been made there).

Here we find that Raysyde’s “Super Mario Bros. – Remix 2” does in fact display the same changes in the opening bytes, including the reference to “toma” and the ASCII art. This then is a clear indication of infra-ludic serialization: the borrowing, repetition, and variation of code-level work between members of the modding community. This essentially serial connection (an infra-serial link) would hardly be apparent from the level of the mods’ respective interfaces, though:

When we compare “Millennium Mario” with ATA’s “Super Mario Bros. – Yoshi’s Quest,” we find the ASCII art gone, despite the visual match in Tableau’s mapping of their “diff” indications for the opening bytes:

“Yoshi’s Quest” corresponds in this respect to Krillian’s “Mario Adventure 2”:

Thus we have another clear indication of infra-ludic serialization, which would hardly have been evident other than by means of a directed filtering of the large dataset, in conjunction with a close analysis of the underlying code.

Again, however, this is just the beginning of the analysis — or more broadly of an encounter between DH and CCS. Ideally, the dataset would be expanded beyond ROMhacking.net’s database; other online communities would be mined for data; and, above all, more integrative tools would be developed for correlating social network graphs and diff-maps, for correlating community and code. Perhaps a crowdsourced approach to some of this would be appropriate; for what it’s worth, and in case anyone is inclined to contribute, my data and the interactive Tableau charts are linked above. But the real work, I suspect, lies in building the right tools for the job, and this will clearly not be an easy task. Alas, like digital seriality itself, this is work in progress, and thus it remains work “to be continued”…

Thanks finally to Eric Monson, Angela Zoss, Victoria Szabo, Patrick LeMieux, Max Symuleski, and the participants in the Fall 2014 “Historical & Cultural Visualization Proseminar 1” at Duke University for the various sorts of help, feedback, and useful tips they offered on this project!

Comics as Mediator of the Print/Digital Divide

Over at Huffington Post, Bill Kartalopoulos has an interesting article on “Why Comics are More Important than Ever” (from whence the image above is taken). I highly recommend reading the piece in full, as it offers a clear, concise, and nicely illustrated exposition of some of the core medial properties of comics, along with an argument about comics’ liminal or transitional position between print and digital media.

The upshot of Kartalopoulos’s argument, which I find quite convincing, is that comics can (or do) serve us as mediators in negotiating some of the shifts and uncertainties we experience in a world that is still undergoing large-scale digitalization — but which is not destined to become digital-only. In other words, pre-digital forms are not going away; there is no “manifest destiny” of the digital, and so we must learn to navigate between medial forms that exhibit very different affordances and demands. Comics marry aspects of both forms, so that they might be seen as privileged mediators of the contemporary (and future) media landscape. As Kartalopoulos puts it:

For more than a century, comics have demonstrated a form of communication that marries the linear sequence of typography with the global perception of an internet-like matrix of simultaneous parts. […]. As we struggle within the cognitive tug of war of our new media landscape, comics offer a useful model for a new type of reading: one that might help resolve the tensions of the current media age to move us toward new productive modes of expression and understanding.

This resonates with an argument I have made regarding the serial properties of the medium — particularly with respect to what Thierry Groensteen calls the “restrained” and “general arthrology” of the comics form: the articulations or linkages that, respectively, work to unite elements in either a linear, sequential dynamics of panel-to-panel transitions or through nonlinear, networked relations between distant panels.

I have touched on these topics in “Framing, Unframing, Reframing,” my afterword to Transnational Perspectives on Graphic Narratives. What I don’t explore in that piece, but which I had in mind when writing it, was the transitional and mediating position between digital and print forms that Kartalopoulos ascribes to comics. In the hopes that it adds something useful to the discussion, and since I’ve never published it anywhere, I offer here the concluding paragraph of a talk, called “Multistable Frames: Notes Towards a (Post-)Phenomenological Approach to Comics,” which I gave in October 2011 at a conference in Bern, Switzerland:

So effectively, what I am proposing here, in the name of a phenomenological approach, is an expansion of the general arthrology developed by Groensteen, who notes that the narrative operations of comics take root in linear sequences of contiguous panels but give rise to braidings or translinear series that establish themselves between distant panels. By following these braided networks beyond the diegesis, beyond the work, and into a plurimedial field of connectivities and the lifeworld it structures, we can appreciate the truth of a remark that Groensteen makes in the conclusion of his book. There, he writes: “comics, which marries the visual and the verbal, demonstrates a discontinuity, a staggering, and the effects of networks, and finally constitutes a sort of image bank, appear to be situated not far from the turning point between the civilization of the book and that of multimedia” (160). We can say, then, that comics are transitional between old and new media due to the emergent seriality that proliferates as a result of comics’ nested multistabilites, a seriality that Groensteen describes as a “supplementary relation” that is “inscribed like an addition that the text secretes beyond its surface” (146-147). Always vacillating between the linear narrative sequence and the translinear network, comics define their seriality as a space of the in-between: between self-enclosed books on the one hand and the total network of hypertext and convergent digital media on the other. As this in-between space of serial proliferation, comics are not assimilable to the monomedial narration of the book, and they resist as well the higher-level closure of transmedial storytelling while upsetting the exhaustive cataloguing projects of digital databases and wikis. With their plurimedial seriality, comics remain squarely in-between. With their techniques of retcon and reboot, for example, and more generally the fact of multistable framing at every level, proliferating in an unruly seriality, comics can be said to have set the stage for a consideration of the experiential gaps between old and new media. As a truly transitional medium, comics inherently confound every attempt at closure or totalization—both the self-contained book and the encyclopedic database depend on discrete categories that are incapable of accommodating the ambiguity and plurality of the multistable frame. And so, despite appearances that they might settle down, let themselves be tamed according to book-centric categories of “respectable” literature—as graphic novels—or captured and rendered coherent and manageable in the convergent space of the digital, comics remain elusive, on the move, and productive of a self-serializing dynamics of the transition. In this respect, they may be useful for understanding the parameters of a rapidly changing visual culture.

Shane Carruth’s Whiteheadian Metaphysics of Post-Cinema #SLSA14

Above, the final slide from my presentation on “Metabolic Media,” which I delivered today at the Society for Literature, Science, and the Arts conference in Dallas. I post it here because it includes one of my favorite recent discoveries: a quotation from Isabelle Stengers’s book Thinking with Whitehead that, in a discussion of Whitehead’s attempts to avoid the “bifurcation of nature” into primary and secondary qualities (as the early moderns put it) or into scientific and manifest images (in Wilfrid Sellars’s terms from the 1960s), comes surprisingly close to naming independent filmmaker Shane Carruth’s 2013 film Upstream Color — thereby unexpectedly helping us to understand the strange, post-cinematic experience of this film, which in its own way seems to reject the clear separation of subjective viewing positions and perceptual objects. Who knows? Maybe Carruth even took the title for his film from Stengers’s book. I have no evidence for this whatsoever, of course, but the resonance between the Whiteheadian project and what I call the “metabolic images” of Carruth’s film is so compelling, in my opinion, that the discovery of this quote makes it fun to speculate (idly) about the possibility…

Diagramming Media as the Environment for Life

Both aesthetically and conceptually, the diagram above is imperfect in many ways. It is, necessarily, an oversimplification; I hope that it might nevertheless serve a positive purpose by giving visible form to an otherwise somewhat abstract argument. Developed for my talk at the upcoming “Philosophy After Nature” conference in Utrecht (you can find my abstract here), the diagram could also serve as an emblem for the argument I make in Chapter 6 of Postnaturalism. In that chapter, I look at (among other things) Mark Hansen’s concept of “the medium as an environment for life” (as introduced in his paper “Media Theory,” which appeared in Theory, Culture & Society); this concept, developed in conversation with Bernard Stiegler’s philosophy of technics, has been very important for my work, and grappling with it was central for me in the process of arguing that “we have never been natural.”

In the course of developing his concept, Hansen argues that there is an asymmetrical priority of human embodiment in the transductive relation between technics and the human. In Hansen’s engagement with Stiegler, this prerogative of embodiment is seen to be at odds, to a certain extent, with Stiegler’s argument about the synchronization or industrialization of experience through the action of recording technologies. The latter embody “tertiary retentions” of experience, beyond the primary and secondary retentions that Husserl theorized as the operations, respectively, of immediate temporal experience and of recollection or memory. According to Stiegler, in a complex argument that I will not try to summarize here, tertiary retention (technical recording) injects secondary retention (memory) into primary retention (the immediate experience of the “adherent present,” from which flows also the future) — effectively instituting a pre-formatted future on a mass scale (especially in the age of live television and real-time media).

Following an objection raised by Jean-Michel Salanskis, who sees a paradox or split in Husserl’s notion of primary retention — a split between the referential aspect that aligns primary retention with conscious experience, on the one hand, and a non-referential aspect that is wholly unconscious, on the other — Hansen argues that Stiegler’s argument diminishes the robust role of embodiment in the production of temporal experience. The synchronization envisioned by Stiegler is dependent, according to Hansen, on a bracketing of embodied agency; the “mnemotechnical constitution of time” prioritized by Stiegler is thus secondary to the “corporo-technical constitution of time” that Hansen identifies as an infra-empirical condition of experience. Hence the asymmetrical privileging of human embodiment in the medial transduction of human and technical agencies.

The diagram above summarizes my own intervention in the context of these debates. Rather than reinstituting the priority of the human within the anthropotechnical transduction, my suggestion is that we conceive tertiary retention (and media technics more generally) as similarly split between a referential (“mnemotechnical” or broadly representational) and a non-referential (materially embodied) aspect. With memory flanked on both sides by a non-discrete, smooth space of matter, cognitive life is then situated squarely in a realm staked out between robustly material agencies—between the subpersonal operation of the body, on the one hand, and the subphenomenal, infra-empirical material agency of technics on the other. As the diagram tries to indicate, a certain symmetry is restored in the anthropotechnical interface, which on this model describes the joint production of empirical reality — the distributed (human and nonhuman) agency by which the phenomenal realm is demarcated from out of the unmarked environment of material flux.

DFG Research Unit “Popular Seriality” — New Website Online

Just a quick note to point out that the new website of the DFG Research Unit “Popular Seriality — Aesthetics and Practice” is now online. Check it out! LINK

“What if the camera / really do / take your soul?”

If the medium is the message, as Marshall McLuhan famously claimed, then the so-called “selfie” may be less about the face that constitutes the recognizable content of such an image, and more about a deeper, less obvious form of material-aesthetic mediation with respect to the transformation of “self” in an age of ubiquitous post-cinematic cameras.

Clearly, such acts of mediation have many levels. On the one hand, we “stage” or “perform” our selves for ourselves and for our friends (and, of course, for our Facebook “friends”); at the same time, though, we do so with an awareness of the machinery of geolocated surveillance and algorithmic facial recognition systems that we feed and help to optimize with the offering of our selfies (and the metadata they contain). Is this a self-destructive tendency or an act of defiance? Do we taunt and shake our fists at the invisible all-seeing God of Hyperinformatic Imagery (or the NSA), heroically though baselessly asserting our autonomy despite our knowledge of its baselessness? Or is it just that we have resigned ourselves to the new “situation,” in which Berkeley’s maxim esse est percipi has been made a reality through a media-technical dispositif that renders superfluous the whole apparatus of angelic and divine perceptions that Bishop Berkeley still needed to keep his system from falling apart?

But the post-cinematic camera is a post-perceptual camera. Esse is now post-percipi in the sense that networks of digital and increasingly “smart” cameras are not just collecting images of “you” or “me” but instituting radical changes in the fine-grained, “molecular” scale of temporal becoming that subtends subjective (or “molar”) perception. As I have been arguing recently (see here, for example), post-cinematic cameras produce “metabolic images” — images that operate outside of visual or perceptual registers and modulate our pre-personal relations to the environment, directly influencing us at the level of our metabolic processing of duration and relation through which our embodied agencies are defined. This has to do with (among other things) the sheer speed of computational processes, which outstrip our own cognitive and perceptual processing abilities. But it also has to do with the affective density that post-cinematic cameras themselves accrue by virtue of the gap — what Bergson would call a “center of indeterminacy,” or simply a body — that these cameras install between the input and output of images, in the space of their microtemporal computational processing. On this basis, a synchronization of human and technical temporalities is made possible at the micro-level. And perhaps this is the hidden message of the medium: the selfie is not just a paradoxical performance of self (in the way that, say, reality shows problematize authenticity), it is in fact the product of a whole new ecology of agency, an ecology of anthropotechnically co-ordinated metabolisms invisibly subtending the visible images by which we seek to represent our “selves.”

With every selfie, we experiment with this interplay of visible manifestation and invisible infrastructure. Who can we be, now, and in relation to an environment filled with rapidly proliferating digital images, where everything is in flux, nothing apparently stable? Perhaps we encounter here, and try to dispel, an old fear in a new guise: that the camera is capable of stealing our souls — both through integration into systems of surveillance, and in the dissolution of our former agencies when set in relation to the molecular, metabolic processes embodied by the post-cinematic camera. In the words of Montreal-based indie rock band Arcade Fire:

What if the camera
Really do
Take your soul?
Oh no...

Hit me with your flashbulb eyes!
Hit me with your flashbulb eyes!
You know I've got nothing to hide
You know I got nothing
No I got ... nothing

Above, my own mixed-media “reflections” on the problem of the selfie in the age of metabolic modulation. Featuring artworks by Thomas Böing (Ohne Titel [Museum König], 2006), currently on display at the impressive Kolumba — Art Museum of the Archdiocese of Cologne as part of the exhibition “show cover hide. Shrine. An exhibition on the aesthetics of the invisible,” which runs until August 25, 2014.

Images that Metabolize Time (Post-Cinematic Perspectives)

[vimeo 80345330 w=500&h=280]

Pioneering video artist Nam June Paik has been quoted as saying that video “imitates not nature but time.” Somewhat more elaborately, in his reflections on “input-time” and “output-time,” he writes: “Video art imitates nature, not in its appearance or mass, but in its intimate ‘time-structure’ … which is the process of AGING (a certain kind of irreversibility).” And elsewhere again, Paik explains his view thus: “So called ‘feedback’, video artist’s favorite word, is nothing but the scientific term for ‘aging’ … that is : enrichment in time-component or a compounded time. Like any other art, video-art also imitates the nature… but in her time-component. Ex. : in NTSC color, color is determined by time-component : that is : phase-delayline in 3.58 mega-hertz.”

Paik’s views on video’s novel relation to temporality inspired philosopher Maurizio Lazzarato to take up the question of time’s modulation in the machines of post-Fordist capitalism in his book Videofilosofia: La percezione del tempo nel post-fordisme (the first chapter of which has appeared in English in Theory, Culture & Society). And Lazzarato’s reflections (along with those of Steven Shaviro and Mark B. N. Hansen, among others) have been central to my own attempts to come to terms with the significance and experiential parameters of our shift to a properly “post-cinematic” media regime.

This past weekend (November 22-23, 2013), I had the opportunity to present my work on the topic at the excellent “Post-Cinematic Perspectives” conference organized by Lisa Åkervall and Chris Tedjasukmana from the Freie Universität Berlin. Steven Shaviro’s talk on Spring Breakers was a particular highlight for me, but I also enjoyed being exposed to thoughts on a number of topics and artworks quite outside my areas of expertise — especially a number of talks on very recent video-art pieces with which I was not previously familiar but am now inspired to seek out.

Through some serendipitous cosmic event — some alignment of the stars giving rise to an unhoped-for coincidence of spatiotemporal coordinates, intellectual and practical concerns, and the respective times of work and leisure — I also found myself confronted with Nam June Paik’s wonderful Triangle: Video-Buddha and Video-Thinker (1976/1991), currently on display in the exhibit “Body Pressure: Sculpture since the 1960s” at Hamburger Bahnhof – Museum für Gegenwart (25 May 2013 – 12 January 2014). Placing a sculpture of Buddha and a small reproduction of Rodin’s Le Penseur under the constant, real-time surveillance of two video cameras connected to four monitors — two of which face each other in a closed-circuit loop of video production and machinic reception — Paik’s Triangle provided the perfect opportunity not only to think more about the “metabolic images” that (following Paik, Lazzarato, Shaviro, and Hansen) I had been theorizing at the conference, but to put these thoughts into practice in an experimental configuration.

[vimeo 80345329 w=500&h=280]

Of course, the fact that (following the advent of the smartphone) virtually everyone today walks around with a high-definition digital video camera in their pocket has no small bearing on the significance and historicity of Paik’s work. Thus, my wife and I decided to expand Triangle‘s loop of time-metabolizing images by adding a further layer of video processing: each of us filmed different points of the sculpture’s own input/output and integrated our own video devices into the loop. The results, seen here, were synced and combined with split-screen and transparency settings (along with reverse-motion in the top video and the introduction of compression artifacts in the bottom one). In this way, we tried to expand and reflect (materially, not cognitively) on the impact of computational imaging technologies for a work like Triangle — on their radical expansion of variables (“and/ors”) which Paik described thus:

In the talk I gave over the weekend, “Nonhuman Perspectives and Discorrelated Images in Post-Cinema” (abstract here), I argued (drawing on Lazzarato’s very Paikian arguments about video):

Computational rendering processes generate unanticipated and unanticipatable images, in effect rendering post-cinematic cameras themselves strangely vibrant, uncanny. There is a dilation of affect involved, which introduces a temporal gap of hesitation or delay between perception (or recording) and action (or playback), and it amounts to a modeling or enactment of the indetermination of bodily affect through which time is generated, and by which (in Bergson’s system) life is defined. A negative view sees only the severing of the images’ indexical relations to world, hence turning all digital image production and screening into animation. But in the end, the ubiquity of “animation” that is introduced through digital rendering processes should perhaps be taken literally, as the artificial creation of (something like) life, itself equivalent with the gap of affectivity, or the production of duration through the delay of causal-mechanical stimulus-response circuits; the interruption of photographic indexicality through digital processing is thus the introduction of duration = affect = life. Discorrelated images, in this respect, are autonomous, quasi-living images in Bergson’s sense, having transcended and gained a degree of autonomy from the mechanicity that previously kept them subservient to human perception. Apparently “crazy,” because discorrelated from the molar perspectives of phenomenal subjects and objects, cameras now mediate post-perceptual flows and confront us everywhere with their own affective indeterminacy.

Another way to put this is to say that post-cinematic cameras and images are metabolic processes or agencies, and their insertion into the environment alters the interactive pathways that define our own material, biological, and ecological forms of being, largely bypassing our cognitive processing to impinge upon us at the level of our own metabolic processing of duration. Metabolism is a process that is neither in my subjective control nor even confined to my body (as object) but which articulates organism and environment together from the perspective of a pre-individuated agency. Metabolism is affect without feeling or emotion – affect as the transformative power of “passion” that, as Brian Massumi reminds us, Spinoza identifies as that unknown power of embodiment that is neither wholly active nor wholly passive. Metabolic processes are the zero degree of transformative agency, at once intimately familiar and terrifyingly alien, conjoining inside/outside, me/not-me, life/death, old/novel, as the basic power of transitionality – marking not only biological processes but also global changes that encompass life and its environment. By insinuating themselves into the molecular flows of affect, prior to the possibility of perception and action, metabolic images have a direct impact on “the way we tick” — i.e. on the material production and modulation of time and temporal experience.

In many ways, the original assemblage of Paik’s Triangle already demonstrated what I call the metabolic work of microtemporal image processing. Today, however, it provides further opportunities for experimentation with the spatial and temporal parameters of our existence in conjunction with the many cameras and screens that connect us with our contemporary environment. Expanding the variables of the work’s “and/or” configurations, our cameras can hook into Paik’s assemblage, enter into its feedback loops, but also transport those loops into larger contexts of metabolic processing and transport. Images circulate within and beyond, effectively confounding distinctions between object and process, thing and environment. By engaging with the work and aiming our cameras at it in this way, we too are hooked into the system, and our own embodied perception is displaced as it is integrated into the molar and molecular configurations that situate and underpin conscious experience. In this way, Paik’s work continues today to probe the technical modulations of time and affective life in the era of convergence, computation, and our properly post-cinematic environment.

[vimeo 80355324 w=500&h=280]

Das Zwergenproblem — and how to solve it…

What is the Zwergenproblem, or the problem of the gnome? Though this looks and sounds like one of those classic German words destined for import into the English of intellectuals (think of Zeitgeist, Weltanschauung, or Fahrvergnügen…), you won’t find the term “Zwergenproblem” in any dictionary. And yet it’s a widely shared opinion in Germany (at least in left-leaning circles) that garden gnomes — themselves typically Deutsch — are somewhat (how should I say?) problematic creatures. Their nation-based typicality is one of the problems, and they are aligned with a range of conservative political values as symbols of a bourgeois Sesshaftigkeit that finds its natural expression in the carefully mowed lawn and Garten of the proud homeowner/Dorfbewohner. This is by no means to say that gnomeownership, any more than homeownership, is a clear indication of one’s politics. Still, the associations and stereotypes are there — so much so, in fact, that garden gnomes have come to embody a downright cliché for a certain sort of lifestyle. Of course, the existence of a cliché always invites ironic appropriation as a response, and so recent times have seen the appearance of black-leather biker-gnomes, pot-smoking gnomes, and gnomes doing gnaughty things. Trying to dissociate gnomes from notions and practices of bürgerliche Spießigkeit, these have been attempts to solve what I am calling the Zwergenproblem. But none of them, it seems to me, has yet provided an adequate response, one suited to the true gravity of the situation.

Meanwhile, beyond and outside of Germany, the garden gnome has come to represent a mostly white, middle-class, suburban existence in a depoliticized any-space-whatever. Garden gnome liberationism has emerged in response — an international effort to free the gnomic proletariat from their servitude, to return gnomes to the wild, and more generally to draw attention to their plight. And yet the political edge wears thin when these efforts devolve into “pranks.” The travelling gnome prank — in which mostly white, middle-class, suburban kids steal their neighbors’ gnomes and take them on vacation, photographing them in front of famous tourist attractions — is all good fun, as far as it goes, but it doesn’t really address the core of the Zwergenproblem. Clearly, this is because the prank is situated within the same space of possibility as the “straight” form of gnomeownership that it ostensibly questions: the space of material affluence, leisure, and the freedom to travel, where the suburban home materially anchors and spiritually secures the traveller’s foray “out into the world” like the warmth of Heidegger’s Hütte accompanied him along his Holzwege. Besides, any last drop of radical potential was drained when a major Internet travel company appropriated the prank for its marketing campaign, thus transforming the “liberated” garden gnome into a symbol for digitally enabled neo-liberal capital and the transnational flows of money, bits, and bodies. Behold: the neo-liberated gnome.

This is where things get interesting, I think, and where a space for artistic intervention into the Zwergenproblem begins to disclose itself. The example of Internet-based travel, which of course depends on real money (hence real labor) and ideally gets real bodies to real places, places the garden gnome squarely in the realm of the so-called “new aesthetic” — which James Bridle describes as “a series of artefacts of the heterogeneous network, which recognises differences, the gaps in our overlapping but distant realities,” especially as concerns the intersection of material and digital realms. The neo-liberated gnome embodies capital as it flows within the control society, and it both emulates and encourages the “participatory culture” of viral marketing, whereby the unpaid immaterial labor done on social networks is appropriated and a surplus value is extracted by algorithmic means. What better mascot than the neo-liberated gnome, which bears witness to the wonders of the world, makes us hungry for travel (and for good-natured pranks), and even tempts us to donate our labor by posting pictures to Facebook et al that will serve as further advertisement for that travel company with the funny garden gnome…

The association with the “new aesthetic” is even clearer in the context of 3D printing and scanning technologies, where the garden gnome has become a symbol for that magical intersection of materially unique objects rendered digitally reproducible, as in MakerBot’s extensive use of the gnome to demonstrate their machines. There should be no doubt about it: 3D printing really does involve a revolutionary sort of realignment of the physical and the ideal, but why should the garden gnome become the symbol for this transformation? The answer, I think, has to do with the fact that before it became the mascot for the new aesthetic, the garden gnome was a mascot for a putatively “old aesthetic” that itself was secretly bound up in the appearance of industrial modernity and its “aesthetic of the new.” The garden gnome’s origins are in nineteenth-century Germany; accordingly, the creatures must be seen as an essentially modern phenomenon, and as a part of the popular culture that begins to emerge in Europe alongside industrial technologies of production and communication. And yet they feign resistance to that culture, pretending to belong to a romanticized folk culture that is pitted against the emerging commercial popular culture. The gnome, in other words, claims allegiance to the oral culture and local tradition that bequeaths to us the fairy-tale, and it aims to distinguish itself from the mass-produced industrial culture of the modern world — to which, nevertheless, it essentially and materially belongs! Now, with the advent of 3D printing and design, the possibility of digital reproduction calls the gnomes’ bluff, makes their mass and serialized nature apparent, and reveals that they have been subject from the start to the same iterative principles as the serial figures of popular culture. What’s more: the digital infrastructure democratizes the production process, putting the means of production in the hands of the many (or at least in the hands of the growing number who have access to the technology). The gnome becomes shareable across time and space, and subject to a serial process of modification. But again there is the danger that the material and immaterial labor of this “participatory culture” is subject to appropriation and exploitation.

Besides, there’s nothing very subversive about printing born-digital gnomes in order to repeat the pranks of their more earthy forebears. If we’re really going to do something about the Zwergenproblem — i.e. if we are going to address the problematic politics of the garden gnome as it exists in our transitional moment — we will have to do so with a historical consciousness, one cognizant of the gnomes’ troubled history, their relation to modern production processes, material and immaterial labor, class consciousness, nationalism, popular and high-art cultural formations, and the role of seriality in all of these constellations.

From an artistic perspective, no one is doing a better job of this at present, I think, than Karin Denson, with her “Krass People” series of gnomes, which are featured throughout this post. (Full disclosure: Karin is my wife. I’m biased. So what? No one’s got better gnomes than her!) Based on iconic figures from popular culture and modern art, her gnomes call into question the boundaries between industrial mass production, pre-modern handcraft, and modern and postmodern artistry. Half ready-made, half carefully crafted objects, the gnomes are collected from flea markets, outlet stores, garage sales, eBay, and wherever else they might be found, before they are hand painted and occasionally re-sculpted to resemble superheroes like Superman or Batman, creatures such as Nosferatu or Frankenstein’s monster, pop stars like Lady Gaga or David Bowie, figures from Star Wars or Pirates of the Caribbean, or re-imaginings of artworks by Duchamp, Miró, or Dalí. The result is a set of unique physical objects that retain strong conceptual and material links to the cultures of seriality that, since the nineteenth century, have increasingly and irrevocably problematized any notion of uniqueness or (artistic) authenticity. Finally, the objects are digitized and their images subjected to various further transformations: placed in Photoshop collages, set in motion in animated gifs, and displayed in a growing virtual gallery on tumblr. And that’s just the beginning: video, generative, and other works are in planning.

So does this answer the Zwergenproblem? Perhaps not completely. After all, can there be a truly final answer to a problem that is so thoroughly serial in both its historical genesis and its structural formation? Any answer, it would seem, would itself have to be serial rather than static. And that’s precisely what we have here: an ongoing, serially unfolding, plurimedial and mixed-reality approach that may not answer, but which first succeeds in formulating or addressing, that central problem of the modern world: das Zwergenproblem.

Finally, in case you’re interested: Karin’s gnomes are available for purchase, but there is only a very limited supply of these (problematically) unique and labor-intensive objects. Karin will be exhibiting and selling them this coming Sunday, November 3, 2013, at the Kunsthandwerkermarkt at the Kulturzentrum Faust (from 11am to 5 pm in the “Warenannahme”).

[UPDATE: There’s now an Etsy store where the gnomes can be purchased online: https://www.etsy.com/shop/KrassStuff]

Not Yet Titled: Alles in Ordnung

This weekend I had the pleasure of visiting the Museum Ludwig in Cologne, which is currently showing an exhibit called “Adjusted,” comprising a range of works by the American “Pictures Generation” artist Louise Lawler, alongside an all new permanent exhibition called “Not Yet Titled: Neu und für immer im Museum Ludwig” [Not Yet Titled: New and Forever in Museum Ludwig]. Apparently, the latter is less a permanent exhibition in the traditional sense, and more a semi-permanent configuration of museum space whose contents are scheduled to change; accordingly, it is not so much the artworks on display at any moment that constitute the object of the exhibition but rather the museum itself, its spaces and practices of exhibition, so that “Not Yet Titled” seems to defer closure in the interest of staging a quasi-permanent exhibition of flux. That’s a fairly self-reflexive undertaking, as are many of the works on display, so it’s especially refreshing to see that the museum manages to combine all this heady cross- and self-referencing with a material lightness, an architectural and visceral deferral of (en)closure that is all too often lacking in exhibitions of postmodern and semiotically complex works.

One of the exhibition’s centerpieces, Barbara Kruger’s large-scale untitled installation from 1994/1995, which you see at the top of this post (but which you also have to hear in order to appreciate fully), is a case in point: the dense commentary on our media culture, its reflexive irony and intertextuality, are embedded in a space that is at once overloaded and threatening and yet therapeutically soothing as well. The ecstatic sounds of a crowd cheering at the absurd acceptance speech for some unnamed award, where the awardee thanks his God and plays with dictatorial slogans, brainwashing tactics, sexist stereotypes and racist claims of superiority — all of these we register cognitively and appraise their political significance, but the applause enervates us directly, dangerously, on this open stage that seems as if it were designed to highlight the problematic political phenomenology of contemporary spaces: While immersed in the installation, we are able to feel the rich ambivalence of its space, which invites us both to recognize the coded nature of experience while also experiencing something that feels like a space for reflection; but the most surprising aspect of the space can only be grasped later, in a snapshot like mine above: Kruger’s installation, which positively begs to be remediated in the form of a photograph (while flaunting the fact that it can never be captured or encompassed in one), immediately collapses into a flat background, against which the human figure inevitably seems to have been added in later with Photoshop. (I swear, the picture above has not been retouched!)

Not only in Kruger’s installation, but throughout the permanent and temporary exhibitions, it was space that, for me, constituted the true attraction. Lawler’s highly self-reflexive photographs of other artists’ artworks (de- and re-contextualized to highlight the significance of exhibition practices and spatial orders) vacillated between a challenging semiotic complexity and a pleasant, almost banal decorative quality within the large open spaces of the Ludwig. Particularly refreshing was the newly commissioned Tracings series, which carries Lawler’s self-reflexive and intertextual tendencies further but abstracts them, reduces, and contributes to a clear open space. In this series, Lawler’s photos of other people’s art are reproduced again, but now in the simplified form of black contour lines upon a white background, thus transforming the high-resolution photographs and rendering the works more iconic and approachable. These oversized coloring-book pictures, which we survey upon exiting the smaller exhibition rooms in the wide monochromatic space in which we access the stairs, refer, of course, to the images we saw downstairs, but they also seem happy enough to slip into the background and assume a more functional, properly architectural role that does not force any sort of dialogue or commentary. Likewise, Lawler’s new Stretch images, which take Andy Warhol’s famous Brillo boxes as their “subject,” blown up to gigantic proportions to occupy two complete walls with anamorphically stretched photos, emphasize the space of the museum — both as an institution and as a material environment.

All in all, the current showings at the Museum Ludwig offer lots of food for thought — for reflection on the politics, history, and institution of contemporary art, for theoretically guided musings on the relations of art to mass media and our changing media of reproduction, and for thought about our own place within these configurations. If that’s what you’re looking for, you’re sure to be happy here. These exhibitions do more, though, as well, and something that many museums fail to do as they aim for more “interactive” forms of engagement (while misunderstanding “interactive” as “overstuffed and preemptively overladen with information”): namely, Lawler’s “Adjusted” and the (quasi-)permanent “Not Yet Titled” grasp the space of their own staging, clear it out for our experience, and remind us that we are bodies moving through space and time. Architectural space becomes affective space, and this affective space collides ambiguously with the content of the artworks displayed. The latter may make us know, on a cognitive and political level, that everything is not OK, but even just registering that in this wonderful space imparts a (not unproblematic) feeling, a not yet titled affect: “Alles in Ordnung” (irgendwie, vielleicht)…

Post-Cinematic Perspectives

On November 22-23, 2013, I will be participating in the conference “Post-Cinematic Perspectives,” which is being organized by Lisa Åkervall and Chris Tedjasukmana at the Freie Universität Berlin. There’s a great line-up, as you’ll see on the conference program above. I look forward to seeing Steven Shaviro again (and hearing his talk on Spring Breakers), and to meeting all the other speakers. My talk, on the morning of the 23rd, is entitled “Nonhuman Perspectives and Discorrelated Images in Post-Cinema.” The conference is open to the public, and attendance is free.