Over the last 12 months, we’ve spent a lot of time talking about nuances of communication that don’t seem to translate over video call. Sure, we can try and be analytical about it, and note how things like body language, gesture and tone of voice are some of the obvious casualties of conversing through a two-dimensional, highly compressed window. But the truth is, it’s much more of a “feeling” thing. There’s no doubt that we’re incredibly lucky to have the video communication technology that we do during this time, but you can’t help but feel there’s something missing when trying to share an intimate connection with someone over Zoom.
Sound of Light, a collaborative sound performance experience built for the Oculus Quest VR headset, is an example of not only how VR might help us to bridge that communication gap but also how it may facilitate us to communicate ideas and emotions that simply can’t be expressed to one another in a real world environment.
The project is being created by Jung In Jung, a fellow at the Dundee-based creative research and development centre InGame. Jung will be showcasing an in-development version Sound of Light at the PlayAway Festival during the VR Mini-Talks session of Wednesday 3 March at 13:00 – 14:00. She will be joined by 3D artist Paul Blackham, who is assisting on the project.
Music for Virtual Spaces
Though Jung In’s background is in sound design and sound composition, her work with interactive audio has drawn her into the games development community through her use of game technologies like controllers, headsets and the Unity engine.
“I wanted to continue my research creating interactive music or interactive sound, but in a virtual space,” she told us.
“Previously, I’d been working with contemporary dancers using hacked game controllers. So as they moved, I had programmed it to trigger different kinds of sounds and eventually create a composition.
But now, I’ve started using VR, especially the Oculus Quest, because of the real-time hand tracking feature with the front four cameras. I found that it’s a more interesting way to use physical interaction within virtual reality, rather than just using controllers.”
With the help of Blackham, Jung is creating a VR space that invites musicians and non-musicians alike to collaborate in the generation of sounds using body movements and gestures. But as well as a creation and performance tool, Jung sees Sound of Light as a means to communicate with others using sound in novel and potentially illuminating ways.
In a video of the work-in-progress build, we see two avatars forming a variety of hand signs and manipulating the environment around them to craft soundscapes together, waving playfully at one another as they work. At one point, one avatar rubs their thumb and forefinger together to make scurnchy noises on either side of the other player’s head, presumably producing an ASMR-like stereo effect in their headset speakers.
Born from Lockdown
Though Jung In had already been exploring VR prior to the coronavirus outbreak, the idea for Sound of Light came about in response to the pandemic.
“Before we created this project, we had another VR project, which was just single player. And it was actually my first time trying VR. It was mostly traditional, just me presenting myself as a musician doing a performance on the stage, that kind of style.
But then, as we all know, the pandemic started. So all my performances were of course cancelled. We were supposed to show our work in New York, actually, last April at the Babycastles gallery, but it didn’t happen. So I was just waiting for that, thinking ‘Okay, yeah, maybe at some point we will have this opportunity again’. And then I realised, ‘why am I just waiting for it, maybe I should create a virtual space where I can actually invite other people to join and perform with me’.”
Alone, Together
Meanwhile, Jung found herself becoming more interested in multiplayer games while having to stay at home, and in turn became intrigued by the possibilities for atypical social experiences in VR.
“I wasn’t really interested in any multiplayer stuff before because I just wanted to have really relaxing moments when I play a game, so I didn’t like playing with other people. But the pandemic loneliness made me just go into different kinds of things.
So I played Tender Claws’ The Under Presents, which is a kind of multiplayer roleplay kind of game. And the unique thing about this game was, you can’t speak. So everything, you have to just communicate with gestures. And I found that really weird, but really beautiful. Because, of course, it creates a very safe environment, but then all the communication is actually coming from the physical expression.
So I thought, maybe I could create a multiplayer space where people can actually just express [themselves] through sound.”
As it exists at the moment, players can speak to one another in Sound of Light, but they also have a wealth of visual and audio cues with which to enhance that communication.
On the technical side, Blackham speculates that the limitations of the Oculus Quest force players to communicate in interesting, non-conventional ways.
“The cameras that pick up your hand gestures don’t pick up hand gestures that are too far out of range, so you kind of find yourself holding your hands forwards for most of it just so that you don’t lose track of them. So in a way, you’re kind of forced to be more gestural in the way you speak, not even including the actual interactive elements of it. It’s kind of a bit of a byproduct of the actual hand tracking technology itself in some ways.”
Sharing the Experience
While many musicians have been using technology experimenting with new modes of performance, especially during the pandemic when no one can perform live, Sound of Light is somewhat unique in that it levels the playing field, erasing the distinction between performer and audience. Quite the contrast, then, to running around under Travis Scott’s mountain-sized Nikes in Fortnight or the nightmare-inducing experience of listening to Bjork from inside her mouth. Instead, everyone has the same simple avatar and the focus is simply on being creative with others.
“I think the beauty of this project is just really to express yourself through sound and using your natural hand gestures. So it doesn’t matter how you look. I mean, the avatar is very simple. So it’s all about just sharing that experience together.”
Find out more about Sound of Light during the VR Mini-Talks at PlayAway on Wednesday 3 March, 13:00 – 14:00.
