Tweet
Tweet

Slide 1

Slide 1 text

Implementing Audio for Android Caren Chang

Slide 2

Slide 2 text

Motivation

Slide 3

Slide 3 text

Motivation Creating a more interactive experience

Slide 4

Slide 4 text

Motivation Creating a more interactive experience Allow for personalization

Slide 5

Slide 5 text

Motivation Creating a more interactive experience Allow for personalization Guide users

Slide 6

Slide 6 text

How is Audio Processed?

Slide 7

Slide 7 text

How is Audio Processed? Latency - Time delay as a signal passes through a system

Slide 8

Slide 8 text

Latency - Time delay as a signal passes through a system How is Audio Processed?

Slide 9

Slide 9 text

Latency - Time delay as a signal passes through a system How is Audio Processed?

Slide 10

Slide 10 text

Latency - Time delay as a signal passes through a system How is Audio Processed?

Slide 11

Slide 11 text

How is Audio Processed? Application App audio source

Slide 12

Slide 12 text

How is Audio Processed? Application Audio Framework App audio source Resample Effects Mixer

Slide 13

Slide 13 text

How is Audio Processed? Application Audio Framework Device Hardware App audio source Resample Effects Mixer DAC

Slide 14

Slide 14 text

How is Audio Processed? Audio hardware consumes data at constant rate

Slide 15

Slide 15 text

How is Audio Processed? Audio hardware consumes data at constant rate When we pass audio files to be played, we have to make sure it gets processed in time to meet the deadlines

Slide 16

Slide 16 text

How is Audio Processed? Audio hardware consumes data at constant rate When we pass audio files to be played, we have to make sure it gets processed in time to meet the deadlines Otherwise, when deadlines are missed, you can get silent sounds or weird ‘static’ sounds

Slide 17

Slide 17 text

How is Audio Processed?

Slide 18

Slide 18 text

How is Audio Processed?

Slide 19

Slide 19 text

How is Audio Processed?

Slide 20

Slide 20 text

How is Audio Processed?

Slide 21

Slide 21 text

How is Audio Processed?

Slide 22

Slide 22 text

Implementing Audio Playback

Slide 23

Slide 23 text

Implementing Audio Playback MediaPlayer SoundPool ExoPlayer

Slide 24

Slide 24 text

Implementing Audio Playback MediaPlayer SoundPool ExoPlayer

Slide 25

Slide 25 text

MediaPlayer

Slide 26

Slide 26 text

MediaPlayer Easily fetch, decode and play audio with minimal setup

Slide 27

Slide 27 text

MediaPlayer Easily fetch, decode and play audio with minimal setup Supports :
 Local resources
 Internal URIs (from content resolver)
 External URLs (http streaming)

Slide 28

Slide 28 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 29

Slide 29 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 30

Slide 30 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 31

Slide 31 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 32

Slide 32 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 33

Slide 33 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 34

Slide 34 text

MediaPlayer void playAudioFile(String audioFilePath) {
 MediaPlayer mediaPlayer = new MediaPlayer();
 try {
 mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start();
 } catch (IOException e) {
 Log.e(TAG, "Error playing audio file: " + audioFilePath);
 mediaPlayer.release();
 }
 }

Slide 35

Slide 35 text

MediaPlayer

Slide 36

Slide 36 text

MediaPlayer MediaPlayer is state based, can easily caused undesired behaviors

Slide 37

Slide 37 text

MediaPlayer MediaPlayer is state based, can easily caused undesired behaviors mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo(); IDLE

Slide 38

Slide 38 text

MediaPlayer MediaPlayer is state based, can easily caused undesired behaviors mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo(); INITIALIZED

Slide 39

Slide 39 text

MediaPlayer MediaPlayer is state based, can easily caused undesired behaviors mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo(); PREPARED

Slide 40

Slide 40 text

MediaPlayer MediaPlayer is state based, can easily caused undesired behaviors mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
 mediaPlayer.setDataSource(audioFilePath);
 mediaPlayer.prepare();
 mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();

Slide 41

Slide 41 text

MediaPlayer : Common Errors Not handling onError

Slide 42

Slide 42 text

MediaPlayer : Common Errors Not handling onError Performing actions in wrong state

Slide 43

Slide 43 text

MediaPlayer : Common Errors Not handling onError Performing actions in wrong state Unsupported file types

Slide 44

Slide 44 text

MediaPlayer : Common Errors Not handling onError Performing actions in wrong state Unsupported file types Not releasing after use

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Implementing Audio Playback MediaPlayer SoundPool ExoPlayer

Slide 47

Slide 47 text

SoundPool

Slide 48

Slide 48 text

SoundPool Uses the MediaPlayer service to decode the audio

Slide 49

Slide 49 text

SoundPool Uses the MediaPlayer service to decode the audio Collection of samples from the APK that can be loaded into memory

Slide 50

Slide 50 text

SoundPool Uses the MediaPlayer service to decode the audio Collection of samples from the APK that can be loaded into memory Able to manage number of audio streams
 IE : able to do sound mixing

Slide 51

Slide 51 text

SoundPool Uses the MediaPlayer service to decode the audio Collection of samples from the APK that can be loaded into memory Able to manage number of audio streams
 IE : able to do sound mixing Can set priority when playing sound

Slide 52

Slide 52 text

SoundPool SoundPool mSoundPool = new SoundPool.Builder()
 .setAudioAttributes(mAudioAttributes)
 .setMaxStreams(MAX_SOUND_STREAMS)
 .build();
 
 int loadedSoundId = mSoundPool.load(audioFilePath, 1);
 
 mSoundPool.play(loadedSoundId,
 volumeToPlaySound, volumeToPlaySound,
 REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

Slide 53

Slide 53 text

SoundPool SoundPool mSoundPool = new SoundPool.Builder()
 .setAudioAttributes(mAudioAttributes)
 .setMaxStreams(MAX_SOUND_STREAMS)
 .build();
 
 int loadedSoundId = mSoundPool.load(audioFilePath, 1);
 
 mSoundPool.play(loadedSoundId,
 volumeToPlaySound, volumeToPlaySound,
 REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

Slide 54

Slide 54 text

SoundPool SoundPool mSoundPool = new SoundPool.Builder()
 .setAudioAttributes(mAudioAttributes)
 .setMaxStreams(MAX_SOUND_STREAMS)
 .build();
 
 int loadedSoundId = mSoundPool.load(audioFilePath, 1);
 
 mSoundPool.play(loadedSoundId,
 volumeToPlaySound, volumeToPlaySound,
 REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

Slide 55

Slide 55 text

AudioService.java

Slide 56

Slide 56 text

Implementing Audio Playback MediaPlayer SoundPool ExoPlayer

Slide 57

Slide 57 text

ExoPlayer

Slide 58

Slide 58 text

ExoPlayer Open source project (not part of Android framework)

Slide 59

Slide 59 text

ExoPlayer Open source project (not part of Android framework) Pros:
 Ability to update player along with application

Slide 60

Slide 60 text

ExoPlayer Open source project (not part of Android framework) Pros:
 Ability to update player along with application Fewer device specific issues across different versions of Android

Slide 61

Slide 61 text

ExoPlayer Open source project (not part of Android framework) Pros:
 Ability to update player along with application Fewer device specific issues across different versions of Android Ability to customize and extend the player

Slide 62

Slide 62 text

ExoPlayer : Talks ExoPlayer : Flexible Media Playback for Android
 Google IO ‘17 Advanced Exoplayer
 Effie Barak

Slide 63

Slide 63 text

Being a good (App) Citizen

Slide 64

Slide 64 text

AudioFocus

Slide 65

Slide 65 text

AudioFocus Audio from different apps can play simultaneously (system does not have control over which streams play)

Slide 66

Slide 66 text

AudioFocus Audio from different apps can play simultaneously (system does not have control over which streams play) If individual apps don’t properly handle audio focus it will result in a bad user experience

Slide 67

Slide 67 text

AudioFocus : Scenario

Slide 68

Slide 68 text

AudioFocus : Scenario User is currently in your app and playing audio

Slide 69

Slide 69 text

AudioFocus : Scenario User is currently in your app and playing audio User goes into another media app that plays audio

Slide 70

Slide 70 text

AudioFocus : Scenario User is currently in your app and playing audio User goes into another media app that plays audio If not properly handled, both apps will now play their audio stream and nothing will be heard

Slide 71

Slide 71 text

AudioFocus : Proper Flow

Slide 72

Slide 72 text

AudioFocus : Proper Flow Request audio focus

Slide 73

Slide 73 text

AudioFocus : Proper Flow Request audio focus Start playback after audio focus is successfully granted

Slide 74

Slide 74 text

AudioFocus : Proper Flow Request audio focus Start playback after audio focus is successfully granted Handle audio focus changes through onAudioFocusChangedListener()

Slide 75

Slide 75 text

AudioFocus : Intentions

Slide 76

Slide 76 text

AudioFocus : Intentions AUDIOFOCUS_GAIN : Indefinite AUDIOFOCUS_TRANSIENT : temporary (phone call) AUDIOFOCUS_TRANSIENT_MAY_DUCK : temporary (notification)

Slide 77

Slide 77 text

AudioFocus : Implementation AudioFocusRequest mAudioFocusRequest =
 new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)
 .setAudioAttributes(mAudioAttributes)
 .setAcceptsDelayedFocusGain(true)
 .setOnAudioFocusChangeListener(...)
 .build();
 
 int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

Slide 78

Slide 78 text

AudioFocus : Implementation AudioFocusRequest mAudioFocusRequest =
 new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)
 .setAudioAttributes(mAudioAttributes)
 .setAcceptsDelayedFocusGain(true)
 .setOnAudioFocusChangeListener(...)
 .build();
 
 int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

Slide 79

Slide 79 text

AudioFocus : Implementation AudioFocusRequest mAudioFocusRequest =
 new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)
 .setAudioAttributes(mAudioAttributes)
 .setAcceptsDelayedFocusGain(true)
 .setOnAudioFocusChangeListener(...)
 .build();
 
 int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

Slide 80

Slide 80 text

AudioFocus : Implementation AudioFocusRequest mAudioFocusRequest =
 new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)
 .setAudioAttributes(mAudioAttributes)
 .setAcceptsDelayedFocusGain(true)
 .setOnAudioFocusChangeListener(...)
 .build();
 
 int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);


Slide 81

Slide 81 text

Sounds on the June Oven

Slide 82

Slide 82 text

Sounds on the June Oven Common sounds vs Special sounds

Slide 83

Slide 83 text

Sounds on the June Oven Common sounds vs Special sounds Themes

Slide 84

Slide 84 text

Sounds on the June Oven Common sounds vs Special sounds Themes Emergency sounds

Slide 85

Slide 85 text

Sounds on the June Oven Common sounds (tap, cancel, accept, back) are eagerly loaded into a SoundPool for low latency playback.

Slide 86

Slide 86 text

Sounds on the June Oven Common sounds (tap, cancel, accept, back) are eagerly loaded into a SoundPool for low latency playback. All other sounds are played with the MediaPlayer

Slide 87

Slide 87 text

Sounds on the June Oven Common sounds (tap, cancel, accept, back) are eagerly loaded into a SoundPool for low latency playback. All other sounds are played with the MediaPlayer Sounds can request to be played immediately

Slide 88

Slide 88 text

Problems Encountered

Slide 89

Slide 89 text

Problems Encountered Creating pleasant experience on sliders

Slide 90

Slide 90 text

Problems Encountered Creating pleasant experience on sliders ‘Wake’ sound

Slide 91

Slide 91 text

Problems Encountered Creating pleasant experience on sliders ‘Wake’ sound Playing sound on theme changed

Slide 92

Slide 92 text

Thanks! @calren24