How will the Metaverse sound?
Emotion and storytelling in the Metaverse through sound
Sound plays the biggest role in providing Metaverse experiences with the right emotion and meaning. Sound effects and music tell the story in virtual 3D worlds and are important to:
Add value to an experience, usually in the form of a memory, but can also be functional for wellbeing, education or general development and entertainment
Connect with communities, artists, companies or brands
Promote escapism, escape from the current environment in the form of telepresence
The importance of sound in creating immersion
Immersion means being completely immersed in an experience. So an immersive experience is one where you feel like you are completely "in” it. This can happen at a pop concert, a movie with very good audio and visual quality, but so also when experiencing VR, a video game or in the Metaverse.
To achieve this, we need to ensure that users are stimulated. This can be done by using as many senses as possible, the most obvious ones being visual and auditory. In addition, we need to ensure that they are touched or emotionally involved in the story, which cannot be done by images alone. It requires sound. Think of music in a movie that carries and conveys emotion to the viewer.
And there are also sound effects that 'tell' more about a virtual environment, such as birds you do not see, but tell you that it is daytime and that there is no danger. For virtual 3D worlds, this concept is in principle not very different from a good movie or game. The exception is that if we want to move freely in that 3D world, the sound also moves along in such a way that it stays with the source, moves around you, just like in real life, and is therefore spatial.
What is spatial audio?
Spatial audio is the way we experience audio in real life as well, that is, sound from all directions, without the idea that the sound is coming from a speaker or from your headphones, and sounds like it is moving around you somewhere. Suppose we're talking in a podcast on this topic, and I would recommend grabbing headphones now, in a regular podcast the voices sound as if they're being played back between our ears, in our heads so to speak. But now when I play a recording of my voice that I recorded in a spatial audio way it sounds like this:
So what you can do is create a form of telepresence. That means giving the feeling of being somewhere else than you actually are. And this is an important sound technology that I apply for VR applications for well-being, art exhibitions, audio tours or metaverse applications.
Which Sound Technologies are needed for the Metaverse?
To answer this question, we first need to know exactly what the metaverse entails and whether it already exists. I think that we have already come a long way. We have an idea of the metaverse and some of us are already using techniques of the metaverse. But a metaverse, in my view, is the entire network of interconnected virtual 3-dimensional worlds, in which users can interact with each other and with their environment, using avatars. At the moment, we are not yet at this point. We already have some 3D worlds in which people can interact and play with each other, for example, Roblox, Minecraft, Decentraland, VR Space and a few more. But we can't easily teleport from one world to another yet. So basically they are sort of mini-verses, which will eventually become part of the larger Metaverse.
Different audio technologies
channel-based audio
For example, a podcast is streamed in the stereo format, suitable for playback over 2 speakers, left and right. And we usually see this in most streaming formats offered online as well. Two channels are compressed, so little data and bandwidth are required. Sometimes you can use multiple channels for a surround experience. This ranges from 5.1, 7.1.4 to sometimes as many as 22.2 surround systems. Each number represents the number of channels and each channel is directly connected to one speaker. We call this channel-based audio. As you can imagine, this system is ideal for pre-appointed rooms, such as movie theaters, but is not prevalent in every living room, let alone mobile, although surround systems do sometimes offer the ability to be translated to headphones.
Object-based audio
A great example of object-based audio is Dolby Atmos. In addition to a channel-based "bed," say the foundation of a sound environment, you can supplement it with object-based audio fragments. Think of a game, where you walk in a certain space, let's say a cave. In this, the foundation is the wind you hear and the occasional drip. Then sometimes a bat may fly over from the front right to the back left. You cannot fix this sound to a channel; it is an object that has the property of being able to switch from one speaker to another. So object-based audio can be played only over many speakers, but also translated to headphones. Something Apple, for example, is doing with their AirpodsPro. They speak of spatial audio and what they are doing, is binauralizing or translating the Dolby Atmos format to headphones or earbuds, while maintaining head tracking.
Ambisonics audio
Ambisonics audio is a technique and format in which a three-dimensional sound environment is recorded or self-created in Digital Audio Workstation (audio computer program), using an ambisonics microphone. The advantages of this technique are that this format is not tied to the number of speakers, and can therefore be played on many different speaker systems - even binauralized over headphones while retaining head tracking. It is therefore ideal for VR and 360-degree videos or fixed soundscape setups that use different speaker arrangements than the standard surround setups.
Binaural Audio
Binaural audio is meant to be played on headphones. Binaural audio can be recorded directly with ear-shaped microphones, or the dummy heads (two microphones placed in fake ears in a Styrofoam head) It can also be created by encoding surround, object-based and ambisonic files to binaural. Once this encoding is applied, you cannot add head tracking unless it is done in real-time. You now have a spatial experience, but the entire sphere of sound moves with you. This is not ideal for VR environments but would be applicable for radio.
The best way for the metaverse is a spatial audio format that automatically adapts to the user's playback system. In most cases, that will be real-time binauralized audio through the headphones, with head tracking or motion simulation. This will make sure that the user experiences a sense of telepresence, or the most natural sound environment possible, matching what the user sees or expects in that virtual environment.
What’s the impact of the metaverse on the music industry?
This a very big question, befitting of an equally big answer. In any case, we know for sure that the metaverse is going to affect the way we will consume media. It will become more immersive, or the feeling of being part of and in the middle of the experience. Attending virtual live concerts, for example. Spatial audio is in my opinion the new standard of audio experience in the metaverse anyway. Whether everyone will release their music spatially remains to be seen. Also because it requires a totally different way of producing and mixing. What we do not yet know exactly, but can only guess, is how we will consume music in the metaverse. And this will determine how much of an impact it will have on the current industry.
In any case, I do expect a lot of change.
In my opinion, the metaverse can only work if we can interact with each other within it, because the social aspect is incredibly important here. Like-minded people will find each other, and fans of artists will become communities. From these communities, new fandoms can be created, because fans will determine more what they consume instead of the current situation, where labels determine which produced tracks will be plugged into streaming services or broadcasters.
In doing so, blockchain technologies will be applied to rights, but also to releases of new tracks. So songs that are released via the blockchain as NFTs could also be proof of access to a concert, or a meet & greet with the artist, or give access to other digital assets of the artist, thanks to smart contracts.
You would also start to see digital artists who can be present in multiple places at the same time ... Anyway, the impact will be big and provide opportunities for new artists. But existing labels will also benefit from promoting their artists in the metaverse, by creating new experiences for users.
The challenges and opportunities for music producers
On the one hand, music producers will have to be more mindful of their target audiences and perhaps seek input from communities. That means they will have to embrace new techniques to make their productions stand out and be liked within existing communities. The challenge here is to create a very strong fan base.
The direct involvement of communities not only allows music producers to hear directly from users what they want, but music producers will create "real" fans when they create productions that are close to themselves. So rather than looking for virality or that one hit, their approach and productions will create a fan base that will travel with them, go on adventures or follow the story of the productions. So as a music producer, I would come up with an excellent (strategic) story that stretches over a few years, thus creating a journey experience including interaction with and for their fans. With gamification aspects, a good marketing concept and reward systems. And also distribute their music productions based on that concept, through the metaverse to their fans...
Conclusion: The future of sound in the metaverse
We are going to experience sound in the metaverse as we do in real life. The most natural representation of sound possible, including appropriate reflections. There will be more functional applications besides entertainment, in areas such as eHealth & wellbeing and art & culture. Here, the stereo format is insufficient to achieve enough immersion, so to harness the full power of audio, spatial audio must be used here as well. It has to add value to the experience. It has to contribute to the purpose of the experience and it has to sound as natural as possible. Audio should give us the feeling of being part of that virtual environment and only spatial audio can do that in the metaverse.
Can the metaverse revolutionize the music industry?
Absolutely, it can... But whether it will happen remains to be seen. Streaming services and record labels are still at the helm and have most of the money (read: power). But artists, thanks to Metaverse technologies, are fortunately getting more opportunities to monetize their creations through other channels. As stated earlier, it offers new ways of consuming music, as well as new forms of distribution, and added value in music publishing. It will make it easier for new artists to connect with existing communities and artists will be given ways to engage more directly with their fans. There will be more smaller groups of fans who want to witness/experience more of their artist. And there will be longer episodes of stories that will be told, not only through the music, but also with related media expressions, games, VR and AR experiences. And all of this will also encourage social interaction between like-minded people.
The fan-artist relationship will be more important than the current fleeting hit song structure. The artist will build an emotional bond with their fans/community over an extended period of time and offer experiences that we have not yet experienced. The technological possibilities are enormous. The dreams are sometimes too big, but it falls or succeeds with the people who do it! Technology is only a tool and should serve the creators, the artists. For they are the ones who possess the creativity to create something from nothing.