Steve Hammond’s Manual Override – Reality, Complexity & Emotion

If his cultural references seem old, it’s because he was frozen in carbonite for a bit. Gangnam style is still beyond him…

Some time ago, 2K’s Christoph Hartmann, in an article with Games Industry.biz, talked about future consoles. He expressed the opinion that photorealism is the way forward. Games lack proper emotions and this is the only way to have them. Some of the Twitterati didn’t agree and a minor kerfuffle occurred.

“Until games are photorealistic, it’ll be very hard to open up to new genres. We can really only focus on action and shooter titles; those are suitable for consoles now.”

I didn’t see anyone agree with the position, only disagree. Much of the counterargument amounted to little more than that it was ‘obviously’ wrong. But I always prefer to understand what lies behind the ‘obvious’.

Is it really wrong? And if so, why?

An image’s ability to stir the emotions is a plain fact for anyone who has visited an art gallery or watched an advert. Research has been ongoing for decades into the psychology of the means and methods of persuasion. Games are mostly built around the presentation of imagery and so their ability to manipulate us shouldn’t be a surprise to anyone.

Being interactive, adaptive and increasing in sophistication, why aren’t games already the most adept at spinning these feelings we all have? Games have the existing tools of visuals, audio, characters and story to use. Everything, indeed, that a movie can do. Everything that a novel can do. In a game, it is all wrapped up in gameplay. Taken to the extreme, a movie is simply a game with the interaction set to zero.

So let’s isolate resolution from it all. Is emotion independent of resolution? Let’s do proof by counterexample, show what we mean by graphics and even show what we mean by emotion.

In DMA Design, we were concerned about gameplay, even to the extent of giving the impression that the graphics were a secondary concern. The private joke was that our publisher – Psygnosis – did the flashy stuff and we did the playable stuff. Yet for all our concern about fun, playability and looking good (if only, sometimes, by accident), not once do I recall ever having heard a discussion of emotion. Tone, perhaps, as Body Harvest was originally supposed to feel like a B-movie. But not emotion. Perhaps the closest we ever got to this was a never-to-be project: the video game of The Lawnmower Man. I worked on the demo video.

DMA’s Big Kahuna, Dave Jones, was enthused about the idea of the player using the SNES console as a faux-computer interface, to treat it as if it was connected to a virtual computer network (this being before the web took off). The character of Jobe inhabited this network and, as envisaged, would turn up on screen without warning and yell taunts at the player, who would then be duly startled.

Being thoroughly startled, though, even if it’s starting to touch on fear as an emotion, isn’t terribly sophisticated. It’s a thrill, it’s exciting, it may even be visceral, but whatever the technology involved, it’s not much above jumping out from a cupboard.

Are games not capable of sophisticated emotion after all?

Would photorealism help? But what is photorealism anyway? Simply, that it is indistinguishable from a photograph, a claim which has been made for CG at regular intervals since the eighties. In this context it means not obviously created by a pile of circuitry and algorithms, something which looks real. And the best way to do that is with even better circuitry and algorithms. Or by having digitised actors. Immediately, this cuts out cartoons from the mix. And if you can see the problem here…

But let’s not lose sight of those emotions, which games are not generally known for. Not that they are incapable of it. After all, they have access to the full range of tools which cinema does. We are all familiar with film and how that can move us – the example given in the interview was Brokeback Mountain – so let’s take cinema as our touchstone. It is, after all, what at least some games are aspiring to be. Dave Jones has said exactly this.

Now what sort of visual improvements in a game could lead to more subtle – or indeed any – emotional involvement. (It’s worth noting here that audio is hugely important. And hugely underrated, but that’s for another time.)

The most obvious future improvement is better resolution. We’ve come from 40×25 monochrome blocks in 1977 to 1920×1080 pixels in 2012. Colour depth, at 24 bits plus 8 for an alpha channel has been around for so long that it is rarely even mentioned as a feature. Here, too, film has been making its own progress. Movies are now shot on digital formats (chemical film is effectively dead). HD is everywhere. Red Epic video cameras have made 4K resolution commonplace and higher resolutions and frame rates still are possible.

Yet that is a chimera. When we created Lemmings, the small size of the characters was – in part – a response to the animation in Beachhead. That was an 8-bit title with 8-pixel high characters in a single colour. Despite those limitations, the animation was beautifully fluid.

Now take 28 Days Later. This was shot on standard DV cameras, or in other words 720×576; about a comparable resolution to the Wii and less resolution than the sort of pocket camcorder that you can have for £40. (The reason why the film looks ‘cinematic’ is that the lens they used cost around the same as a small car.) None of which affected the emotions that the film was capable of conveying. And this is before we get to movies where the image quality is intentionally lo-fi or deliberately degraded. The Blair Witch Project and other found-footage films make low quality footage into a feature: for the purposes of verisimilitude. Creepy, unsettling. Note that these emotions are made possible by going in the opposite direction of better image quality.

I think we can dismiss resolution as being a contributor to emotional impact. As for those 40×25 character blocks? That has been the default mode for text adventure games since the 1970s. A lack of graphics and sound – in their pure form at any rate – means that the techniques of prose have to pull the player in. A text adventure is a novel with the interaction set to high. Games are a superset of all other media. And a novel, especially when it gets you right there can be particularly powerful.

What other graphical levers can we pull? What about that increased frame rate which I mentioned? The debate over the 48fps filming in The Hobbit is going to rage for a long time. Here at least, the equivalent in games is a faster refresh rate, a slow update leading to the emotion known as ‘annoyance’. Again there is no obvious impact on emotion. Proof by counterexample, again: The French film La Jetee is a 28 minute piece of cinema consisting almost entirely of a sequence of still frames. And b/w ones at that. Yet, it still manages to generate those all important emotions within the audience.

Hopefully it’s becoming obvious that a whole series of technical improvements don’t have much to do with increasing the emotional involvement of a cinema audience, and by analogy, the player.

So a film can stir subtle emotions, that’s clear enough. And it doesn’t need to be IMAX in 3D and whatever massive resolution that can be. Watching a film hand-animated no less – Spirited Away – on a 22in TV in mono sound, the ending reduced me to a flood of tears quite well enough. Cartoons are a form, remember, of which photorealism is the exact opposite. All the techniques of cinema are available to the game.

So enough about what won’t help. What will? If resolution isn’t the lever we need to pull, then what about that other bane of computer animation, expressiveness. Is that what Hartmann is really arguing? Looking at the current crop of cut-scenes/gameplay of a triple-A title, the characters are nicely rendered but are still deep within the Uncanny Valley. Perhaps the way out is to get ever closer to that ‘perfect’ rendering. More polygons! More GPU! Proof by counterexample again: if we’re concerned about the uncanny valley, why not try moving away from it altogether? To get to the truth of anything, do the opposite and see what happens.

In that corner we have Luxo Jr., the table lamp, in Pixar‘s short film of the same name. None of the usual points of reference for detecting emotion are in play. Eyes, mouth, facial expressions, and so forth, yet we know exactly what it is thinking and feeling. Clearly, the expressiveness in an animation is something other than mere pixel or polygon count. Does more sophisticated processing, more realistic art mean more empathy is possible, hence greater emotional attachment? It seems to be the characters, and their stories, we care about.

Empathy is certainly important.

Here’s a photo I took in Orkney this Spring: an entire story told in a single frame. Cutting it down to a few colours or stylising it doesn’t change it. No animation, no polygons, nothing. The story – the teddy bear is waiting for the return of his master who is lost at sea – isn’t something explicit. It was just what occurred to me when I saw it. But it had the effect of making me care about it, made me care about the outcome, which in this story will never have a happy ending. Right there are sophisticated, emotional layers. And a game can display a still image at the very least. All the techniques of the photographer are available to it.

Higher resolution will not help the player care about the protagonists in your game. When playing Sim Earth on the Amiga, I certainly started to care greatly about the icon of Gaia in the corner screaming ‘Stop it!’ at my ham-fisted attempts to save the world. But let’s suppose the story genre in question isn’t game genres – which is a consideration in the article – and instead means styles of narrative. After all, no-one is proposing an emotional connection to the blocks in Tetris. (But consider this…)

Games can quite clearly do the action-movie genre; a film with the interaction switched on. How this can be done is part of the craft of game development, there are unlimited possibilities. There is no reason that we can’t also do a soap opera with the interaction switched on. The Sims, anyone? Impose a strong narrative on this and a storytelling genre is created without having to refer to improved visuals in any way. With a protagonist, we start to care. It becomes meaningful.

And here is the important point. A large part of the emotional experience is knowing what it means. You want to do the emotional complexity of Brokeback Mountain in a computer game? All the elements already exist. Most of them go back to the dawn of cinema. More still go back to the birth of literature. Music is a powerful emotional trigger and as Cory Doctorow has said (in a different context) this predates language itself.

Try this: create a new backstory for Defender, the classic blocky pixel shoot ’em up, such that the abducted 8×8 characters are not astronauts, but toddlers. Now tell me that even an 80s game can’t generate some complex emotions. Knowing what it means – or what you think it means – is a more powerful component than the visuals. Character. Motivation. Backstory. Goal. Setback. Complication. Triumph. Defeat. Betrayal. Revelation. Catharsis. Redemption. Games have all the techniques of literature available to them.

Your best friend getting off with a someone you fancied, at a party and you went home alone. Familiar? In some of you I’ve probably triggered an emotional response and done it the space of a sentence. In fiction – any format – it’s all about creating meaning, identification and empathy. Or the opposite; alienation. You may not enjoy being alienated, but you have certainly had your emotions triggered. It can be done on a Commodore 64. It can be done on a teletype. If you’ve watched Wargames, just how much meaning is invested in that last game of ‘tic-tac-toe’?

And how did you feel when the screen abruptly went blank?

I’ve saved the ultimate refutation to last, a cinema advert from 1993. It was nothing more than two blocks of colour moving around the screen, two pixels if you will, a 2×1 resolution. Yet merely by moving the dividing line it was telling a story. The blue and yellow were getting up to stuff! Steamy stuff! Being able to control it with a joypad would only add to the sophistication. It might even make you feel odd and voyeuristic doing so. It yet is another technique available to games…

So I believe that I shown that photorealism is irrelevant when it comes to emotion. Not only that, games have the available techniques of everything else. Emotions in games, sophisticated emotions, are something which can be done right now.

A game is a superset of all other narrative forms.

But only if you don’t get distracted by the flashy graphics.