Jorge_Garcia_The audio technology behind video games_Codemotion_Berlin_2019.pdf

We are now decades far from the beeps and blops of the early video games from the 70s. Currently, video game audio has reached the quality and depth of music and movie productions, and has even surpassed them. But, how is the technology behind video game productions currently being built? A myriad of techniques such as spatial audio, effects, synthesis, streaming, mixing and codecs play a vital role in the success of a modern video game. This talk will walk behind the scenes in order to explain what can be currently achieved with audio technology and what is to come in the next few years!

About: Jorge Garcia. Audio Software Consultant - Independent

Audio software professional with more than 10 years of experience working in games, professional audio, broadcast and music. Jorge spent three years and a half working in the UK for MIDAS/Behringer, Codemasters and Activision as well as Mercury Steam and EA in Madrid, several record labels and audio companies around the world. He has worked on some well known game franchises such as FIFA, Guitar Hero and DiRT. Jorge holds a BSc in Telecommunications Engineering from the Politechnic University of Madrid and a MSc in Sound and Music computing from the Pompeu Fabra University in Barcelona.

Transcript

1. The audio technology behind video games Jorge Garcia @JGarciaMartin 12-13 November, 2019
2. Who am I? • From Madrid, Spain. • Working in audio since 2001, in game audio since 2008 • Lived and worked in the UK from 2012 until 2015 • Currently a freelance consultant and developer • Like and do aquafitness! 2 @JGarciaMartin - Codemotion Berlin 2019
3. 3 @JGarciaMartin - Codemotion Berlin 2019
4. Why am I giving this talk? • Audio programming for games is still a niche, few people in the industry do it full-time • There is a need for more and more diverse Audio developers • Hopefully help you to have better Audio in your games and projects 4 @JGarciaMartin - Codemotion Berlin 2019
5. https://captivatingsound.com/game-audio-lab-featured-on-aes-international-conference-audio-for-games-2009/ 5 @JGarciaMartin - Codemotion Berlin 2019
6. https://www.engadget.com/2015/08/03/guitar-hero-live-singing/ 6 @JGarciaMartin - Codemotion Berlin 2019
7. https://www.twitch.tv/media_molecule 7 @JGarciaMartin - Codemotion Berlin 2019
8. 10% Typical allocation of resources (CPU, memory) dedicated to Game Audio 8 @JGarciaMartin - Codemotion Berlin 2019
9. Audio Director Lead Sound Designer Manages vision for the project, budgeting, project leadership, key decision maker, content creation and implementation. Sound Designer Technical Sound Designer Content creation and implementation, systems designer and maintainer, creates and maintains tools/pipelines, technical stakeholder. Audio Programmer Audio Software Engineer Systems implementer in code, middleware integrator, engine/DSP developer, creates and maintains tools/pipelines, the most technical stakeholder. GAme audio Roles 9 @JGarciaMartin - Codemotion Berlin 2019
10. The audio programmer role in game studios • Usually shared with various teams or projects • Usually an audio generalist, have to carry out varied tasks like dealing with middleware, tooling, game-side integration, systems implementation, DSP programming, R&D etc 10 @JGarciaMartin - Codemotion Berlin 2019
11. Most studios use third-party solutions and game engines Middleware rules 11 @JGarciaMartin - Codemotion Berlin 2019
12. Wwise middleware code example (C++) // After Wwise has been initialised and banks with events have been loaded void playSound() { AkGameObjectID gameObject = 1; AK::SoundEngine::RegisterGameObj(gameObject, “GameObj1”); AK::SoundEngine::PostEvent(AKTEXT(“Play_SFX_1”), gameObject); AK::SoundEngine::UnregisterGameObj(gameObject); } void setPlayerSpeed(const float speed) { AK::SoundEngine::SetRTPCValue(“PlayerSpeed”, speed); } void gameUpdate() { AK::SoundEngine::RenderAudio(); } Play sound by posting event Update engine Set parameter 12 @JGarciaMartin - Codemotion Berlin 2019
13. Resampling 3D panning and attenuation Obstruction and occlusion simulation DSP effects and synthesis Dialog management Assets randomisation and modulation Resources management Streaming systems Audio codecs Adaptive mixing and limiting technology Ambiences management Interactive and adaptive music systems Audio tools ... Key Game Audio technologies 13 @JGarciaMartin - Codemotion Berlin 2019
14. http://open-std.org/JTC1/SC22/WG21/docs/papers/2019/p1386r0.pdf 14 @JGarciaMartin - Codemotion Berlin 2019
15. Max number of voices 16-32 (lower end, mobile) 32-64+ (PC, console) 15 @JGarciaMartin - Codemotion Berlin 2019
16. Sound engine state machine https://www.crcpress.com/Game-Audio-Programming-Principles-and-Practices/Somberg/p/book/9781498746731 16 @JGarciaMartin - Codemotion Berlin 2019
17. Listeners • Virtual microphone, placement depending on third-person or first-person • Impacts the attenuation of sources and the mix void GameEngine::update() { audioEngine->updateListener(camera->getPosition(), camera->getOrientation()); } 17 @JGarciaMartin - Codemotion Berlin 2019
18. Game audio mix overview https://magazines-static.raspberrypi.org/issues/full_pdfs/000/000/222/original/Wireframe23.pdf?1568906099 18 @JGarciaMartin - Codemotion Berlin 2019
19. Real-time audio mixing • Multiple approaches • HDR audio http://designingsound.org/2013/06/21/finding-your-way-with-high-dynamic-range-audio-in-wwise/ 19 @JGarciaMartin - Codemotion Berlin 2019
20. Open world optimisations • The CPU overload problem • Events spamming • Priority and limitation systems 20 @JGarciaMartin - Codemotion Berlin 2019
21. Open sound control integration https://www.youtube.com/watch?v=s123eCd0RUY 21 @JGarciaMartin - Codemotion Berlin 2019
22. • Double buffering • Playback latency due to e.g. pruning or codecs • Not only for music, but ambiences or dialogue Streaming 22 @JGarciaMartin - Codemotion Berlin 2019
23. Interactive audio codecs MuLaw, ADPCM AAC, ATRAC, MP3, OGG VORBIS OPUS, WMA, XMA Rule of thumb: use what the platform provides if it has hardware accelerated decoders https://arstechnica.com/gadgets/2012/09/newly-standardized-opus-audi o-codec-fills-every-role-from-online-chat-to-music/ 23 @JGarciaMartin - Codemotion Berlin 2019
24. Resampling • Fundamental tech in game audio engines • Used to e.g. save memory or change sounds pitch • Various techniques from linear to sinc interpolation • Libsamplerate from Erik de Castro (OSS thanks to Epic games) https://ccrma.stanford.edu/~jos/resample/Theory_Ideal_Bandlimited_Interpolation.html 24 @JGarciaMartin - Codemotion Berlin 2019
25. DSP effects https://docs.unity3d.com/Manual/class-AudioEffectMixer.html 25 @JGarciaMartin - Codemotion Berlin 2019
26. Reverb usage and optimisations in hitman 2 https://software.intel.com/en-us/articles/hitman-2-enhancing-reverb-on-modern-cpus https://youtu.be/hJLMDW0MXtI 26 @JGarciaMartin - Codemotion Berlin 2019
27. Reverb emulation with ray-casting • Estimate the surroundings of the environment and decide whether the player is within a closed or open environment. http://matarnoldaudio.com/masters-progress/creating-a-dy namic-reverb-system-in-unreal-engine-4/ 27 @JGarciaMartin - Codemotion Berlin 2019
28. 28 https://resonance-audio.github.io/resonance-audio/discover/concepts.html 28 @JGarciaMartin - Codemotion Berlin 2019
29. 29 http://pro.3dsoundlabs.com/category/personal-hrtfs/ https://www.mimi.io/en/blog/mimi-hearing-technologies-acquires-3d-sound-labs 29 @JGarciaMartin - Codemotion Berlin 2019
30. Spatialisation and ambisonics https://en.wikipedia.org/wiki/Ambisonics 30 @JGarciaMartin - Codemotion Berlin 2019
31. Footsteps • Random containers • Tags for different surfaces • Toe and kneel https://youtu.be/2iKcJ0PnEvM 31 @JGarciaMartin - Codemotion Berlin 2019
32. Dialog management World Guinness of Records: Star Wars The Old Republic 371k+ lines in English 1M+ including German and French https://en.wikipedia.org/wiki/Star_Wars:_The_Old_Republic 32 @JGarciaMartin - Codemotion Berlin 2019
33. Interactive music Oldie example: iMUSE in Monkey Island 2 (LucasArts, 1991) https://www.youtube.com/watch?v=7N41TEcjcvM 33 @JGarciaMartin - Codemotion Berlin 2019
34. Procedural audio and SFX synthesis https://youtu.be/FY_JaEFmIwI?t=2019 https://www.asoundeffect.com/no-mans-sky-sound-procedural-audio/ 34 @JGarciaMartin - Codemotion Berlin 2019
35. Procedural audio and SFX synthesis https://youtu.be/L4GuM15QOFE?t=1455 35 @JGarciaMartin - Codemotion Berlin 2019
36. http://tsugi-studio.com/web/en/products-gamesynth.html 36 @JGarciaMartin - Codemotion Berlin 2019 Procedural audio and SFX synthesis
37. Resampling 3D panning and attenuation Obstruction and occlusion simulation DSP effects and synthesis Dialog management Assets randomisation and modulation Resources management Streaming systems Audio codecs Adaptive mixing and limiting technology Ambiences management Interactive and adaptive music systems Audio tools ... Key Game Audio technologies 37 @JGarciaMartin - Codemotion Berlin 2019
38. If you want to learn more... 38 @JGarciaMartin - Codemotion Berlin 2019
39. Thank you! Questions? info@jorgegarciamartin.com @JGarciaMartin