The Sounds Of Android - AndroidMakers 2018

Sound on Android is a topic that is rarely covered, which is why I wanted to shed some lights in my experience with the sound management APIs on Android.

So in this talk, we'll touch on how sounds actually work programmatically, and we'll talk about how to play a sound on Android in the most simple way. We will also cover the principle of sound focus, what it is, how and when to use it. We'll see what we can do with MIDI as well. And finally, we'll go deeper in the rabbit hole and get introduced to the lower levels of sounds processing with the help of OpenSL ES and the SuperPowered library.

Transcript

1. The Sounds of Android Let’s make some noise! Android Makers 2018
2. Noise generator: Yannick Lemin • Android developer freelance • GDG Brussels Organizer • @android_leaks • Noise Maker • @theyann
3. Disclaimer
4. In the beginning…
5. d
6. So let’s make a fart app…
7. So let’s make a fart app… …in 4 steps
8. Step 1 - Create a new app
9. Step 1 - Create a new app
10. Step 1 - Create a new app #Kotlin
11. Step 2 - Add a button to the layout
12. Step 3 - Recording
13. Step 3 - Recording
14. Step 3.5 - Drop the file in the project
15. Step 4 - Play the Media
16. Step 4 - Play the Media MediaPlayer
17. Step 4.1 - MediaPlayer class MainActivity : AppCompatActivity() { private var player: MediaPlayer? = null override fun onCreate(savedInstanceState: Bundle?) { ... findViewById<View>(R.id.button_play).setOnClickListener { play() } } override fun onStop() { super.onStop() player?.release() } private fun play() { player?.release() player = MediaPlayer.create(this, R.raw.best_fart_ever).apply { setOnCompletionListener { release() } start() } } }
18. MediaPlayer • Very simple to use • Exists since API1 • Handles resources, over the network streaming, even with DRM PROS
19. MediaPlayer • Very simple to use • Exists since API1 • Handles resources, over the network streaming, even with DRM PROS • Not extensible • Dependent on OS for bug fixes and new features CONS
20. Step 4 - Play the Media
21. ExoPlayer Step 4 - Play the Media
22. class MainActivity : AppCompatActivity() { private lateinit var player: ExoPlayer override fun onCreate(savedInstanceState: Bundle?) { ... player = ExoPlayerFactory.newSimpleInstance(this, DefaultTrackSelector()) } ... private fun play() { stop() with (player) { val source = ExtractorMediaSource.Factory( DefaultDataSourceFactory(this@MainActivity, "fartheaven") ).createMediaSource(Uri.parse("asset:///best_fart_ever.m4a")) prepare(source) playWhenReady = true } } } Step 4.2 - ExoPlayer
23. ExoPlayer • Much more flexible and extensible • Better suited for complex use cases • External library • Handles caching, adaptive playback, composition, etc PROS
24. ExoPlayer • Much more flexible and extensible • Better suited for complex use cases • External library • Handles caching, adaptive playback, composition, etc PROS • More complex to apprehend • MinSdk 16 (or 19 for encryption support) • Big library CONS
25. Step 4 - Play the Media
26. Step 4 - Play the Media SoundPool
27. Step 4 - Play the Media SoundPool … a pool full of sound?
28. private fun initSoundPool() { soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) { SoundPool(1, AudioManager.STREAM_MUSIC, 0) } else { val attributes = AudioAttributesCompat.Builder() .setUsage(AudioAttributesCompat.USAGE_MEDIA) .setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION) .build() SoundPool.Builder() .setAudioAttributes(attributes.unwrap() as AudioAttributes) .setMaxStreams(1) .build() } soundPool.setOnLoadCompleteListener({ _, _, status -> loadComplete = status == 0 if (loadComplete) { play() } }) soundId = soundPool.load(this, R.raw.best_fart_ever, 1) } Step 4.3 - SoundPool
29. private fun initSoundPool() { soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) { SoundPool(1, AudioManager.STREAM_MUSIC, 0) } else { val attributes = AudioAttributesCompat.Builder() .setUsage(AudioAttributesCompat.USAGE_MEDIA) .setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION) .build() SoundPool.Builder() .setAudioAttributes(attributes.unwrap() as AudioAttributes) .setMaxStreams(1) .build() } soundPool.setOnLoadCompleteListener({ _, _, status -> loadComplete = status == 0 if (loadComplete) { play() } }) soundId = soundPool.load(this, R.raw.best_fart_ever, 1) } Step 4.3 - SoundPool
30. private fun initSoundPool() { soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) { SoundPool(1, AudioManager.STREAM_MUSIC, 0) } else { val attributes = AudioAttributesCompat.Builder() .setUsage(AudioAttributesCompat.USAGE_MEDIA) .setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION) .build() SoundPool.Builder() .setAudioAttributes(attributes.unwrap() as AudioAttributes) .setMaxStreams(1) .build() } soundPool.setOnLoadCompleteListener({ _, _, status -> loadComplete = status == 0 if (loadComplete) { play() } }) soundId = soundPool.load(this, R.raw.best_fart_ever, 1) } Step 4.3 - SoundPool
31. private fun initSoundPool() { soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) { SoundPool(1, AudioManager.STREAM_MUSIC, 0) } else { val attributes = AudioAttributesCompat.Builder() .setUsage(AudioAttributesCompat.USAGE_MEDIA) .setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION) .build() SoundPool.Builder() .setAudioAttributes(attributes.unwrap() as AudioAttributes) .setMaxStreams(1) .build() } soundPool.setOnLoadCompleteListener({ _, _, status -> loadComplete = status == 0 if (loadComplete) { play() } }) soundId = soundPool.load(this, R.raw.best_fart_ever, 1) } Step 4.3 - SoundPool
32. private fun play() { val audioManager = getSystemService(Context.AUDIO_SERVICE) as AudioManager val actualVolume = audioManager.getStreamVolume(AudioManager.STREAM_MUSIC) val maxVolume = audioManager.getStreamMaxVolume(AudioManager.STREAM_MUSIC) val volume = (actualVolume / maxVolume).toFloat() soundPool.play(soundId, volume, volume, priority, loopCount, speed) } Step 4.3 - SoundPool
33. Speaking of recording …
34. MediaRecorder
35. MediaRecorder fileName = "${externalCacheDir.absolutePath}/best_fart_ever.3gp" private fun record() { recorder = MediaRecorder().apply { setAudioSource(MediaRecorder.AudioSource.MIC) setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP) setOutputFile(fileName) setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB) try { prepare() start() } catch (e: IOException) { release() } } }
36. To play it back private fun play() { player = MediaPlayer().apply { try { setOnCompletionListener { this@MainActivity.stop() } setDataSource(fileName) prepare() start() } catch (e: IOException) { release() } } }
37. AudioFocus
38. AudioFocus …or how to make your audio app not be a jackass and work well with the platform
39. AudioFocus Your App Without
40. AudioFocus Your App Without
41. AudioFocus Your App Without
42. AudioFocus Your App Without
43. AudioFocus Your App With
44. AudioFocus Your App With
45. AudioFocus Your App With
46. AudioFocus Your App With
47. AudioFocus Your App With
48. 1. You need a listener class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener { override fun onAudioFocusChange(focusChange: Int) { when(focusChange) { AudioManager.AUDIOFOCUS_GAIN -> { // start playing or reset volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> { // reduce volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> { // pause without dropping focus } AudioManager.AUDIOFOCUS_LOSS -> { // focus was lost } } } }
49. 1. You need a listener class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener { override fun onAudioFocusChange(focusChange: Int) { when(focusChange) { AudioManager.AUDIOFOCUS_GAIN -> { // start playing or reset volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> { // reduce volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> { // pause without dropping focus } AudioManager.AUDIOFOCUS_LOSS -> { // focus was lost } } } }
50. 1. You need a listener class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener { override fun onAudioFocusChange(focusChange: Int) { when(focusChange) { AudioManager.AUDIOFOCUS_GAIN -> { // start playing or reset volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> { // reduce volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> { // pause without dropping focus } AudioManager.AUDIOFOCUS_LOSS -> { // focus was lost } } } }
51. 1. You need a listener class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener { override fun onAudioFocusChange(focusChange: Int) { when(focusChange) { AudioManager.AUDIOFOCUS_GAIN -> { // start playing or reset volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> { // reduce volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> { // pause without dropping focus } AudioManager.AUDIOFOCUS_LOSS -> { // focus was lost } } } }
52. 1. You need a listener class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener { override fun onAudioFocusChange(focusChange: Int) { when(focusChange) { AudioManager.AUDIOFOCUS_GAIN -> { // start playing or reset volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> { // reduce volume } AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> { // pause without dropping focus } AudioManager.AUDIOFOCUS_LOSS -> { // focus was lost } } } }
53. 2. Request the focus private fun requestAudioFocus() { audioFocusListener = AudioFocusListener(…) val requestResult = audioManager.requestAudioFocus(audioFocusListener, AudioManager.STREAM_MUSIC AudioManager.AUDIOFOCUS_GAIN) if (requestResult == AudioManager.AUDIOFOCUS_REQUEST_GRANTED) { // start playing } else { // handle error } }
54. 2. Request the focus … API >= 26 val attributes = AudioAttributesCompat.Builder() .setContentType(AudioAttributesCompat.CONTENT_TYPE_MUSIC) .setUsage(AudioAttributesCompat.USAGE_MEDIA) .build() // keep the request around request = AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN) .setAudioAttributes(attributes.unwrap as AudioAttributes) .setOnAudioFocusChangeListener(audioFocusListener) .build() audioManager.requestAudioFocus(request)
55. Content Type and Usage AudioAttributesCompat.CONTENT_TYPE_SPEECH AudioAttributesCompat.CONTENT_TYPE_MUSIC AudioAttributesCompat.CONTENT_TYPE_MOVIE AudioAttributesCompat.CONTENT_TYPE_SONIFICATION AudioAttributesCompat.USAGE_MEDIA AudioAttributesCompat.USAGE_VOICE_COMMUNICATION AudioAttributesCompat.USAGE_VOICE_COMMUNICATION_SIGNALLING AudioAttributesCompat.USAGE_ALARM AudioAttributesCompat.USAGE_NOTIFICATION AudioAttributesCompat.USAGE_NOTIFICATION_RINGTONE AudioAttributesCompat.USAGE_NOTIFICATION_COMMUNICATION_REQUEST AudioAttributesCompat.USAGE_NOTIFICATION_COMMUNICATION_INSTANT AudioAttributesCompat.USAGE_NOTIFICATION_COMMUNICATION_DELAYED AudioAttributesCompat.USAGE_NOTIFICATION_EVENT AudioAttributesCompat.USAGE_ASSISTANCE_ACCESSIBILITY AudioAttributesCompat.USAGE_ASSISTANCE_NAVIGATION_GUIDANCE AudioAttributesCompat.USAGE_ASSISTANCE_SONIFICATION AudioAttributesCompat.USAGE_GAME AudioAttributesCompat.USAGE_VIRTUAL_SOURCE AudioAttributesCompat.USAGE_ASSISTANT
56. 3. Abandon when you’re done // API < 26 audioManager.abandonAudioFocus(audioFocusListener) // API >= 26 audioManager.abandonAudioFocusRequest(request)
57. And how about a more complex audio app?
58. MediaSession
59. MediaSession …or how to control your stream from anything
60. MediaSession Source: https://medium.com/google-developers/understanding-mediasession-part-2-4-ad291fd32d60
61. MediaBrowserService
62. MediaBrowserService …or how to expose your media to the rest of the ecosystem
63. MediaBrowserService …or how to expose your media to the rest of the ecosystem #ecosystem
64. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99
65. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99
66. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99
67. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99
68. Going Deeper…
69. AudioTrack
70. AudioTrack … The lower level MediaPlayer
71. AudioTrack API < 26 AudioTrack(AudioManager.STREAM_MUSIC, sampleRate, // most likely 44.1 Khz AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, numberOfSamples, AudioTrack.MODE_STATIC)
72. AudioTrack.Builder() .setAudioAttributes(AudioAttributesCompat.Builder() .setUsage(AudioAttributesCompat.USAGE_MEDIA) .setContentType(AudioAttributesCompat.CONTENT_TYPE_MUSIC) .build()) .setAudioFormat(AudioFormat.Builder() .setEncoding(AudioFormat.ENCODING_PCM_16BIT) .setSampleRate(sampleRate) // most likely 44.1Khz .setChannelMask(AudioFormat.CHANNEL_OUT_MONO) .build()) .setBufferSizeInBytes(bufferSize) .build() AudioTrack API >= 26
73. AudioTrack usage fun play() { val audioTrack = initAudioTrack() val someSoundDataInByteArray = getSoundDataSomewhere() audioTrack.write(someSoundsInByteArray, startOffset, bufferSize) }
74. AudioRecord
75. AudioRecord … The lower level MediaRecorder
76. AudioEffect
77. AudioEffect … applies effects on audio!
78. Build-in Effects • Equalizer • Virtualizer • BassBoost • PresetReverb • EnvironmentalReverb
79. Effect Usage val reverb = PresetReverb(priority, audioSession).apply { preset = PresetReverb.PRESET_LARGEHALL enabled = true ... } audioTrack.apply { attachAuxEffect(reverb.id) setAuxEffectSendLevel(1.0F) ... }
80. Standard
81. Standard https://www.khronos.org/opensles/
82. Alternative SDK https://superpowered.com
83. Last but not least
84. Last but not least
85. How does it work
86. How does it work Controller
87. How does it work Controller Interface
88. How does it work Controller Computer Interface
89. How does it work Controller Computer Controller Interface
90. How does it work Controller Computer Controller Interface Controller
91. How does it work Controller Computer Controller Interface Controller
92. On Android
93. THANK YOU :)
94. Good Reads • Understanding MediaSession http://bit.ly/mediasession • Building a Video Player app in Android http://bit.ly/ vplayerandroid • ADB Podcast Episode 85 Focus on Audio http://bit.ly/ adbaudiofocus • Styling Android’s MIDI posts http://bit.ly/midiandroid