Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Implementing Audio on Android

Caren
November 05, 2017
250

Implementing Audio on Android

Caren

November 05, 2017
Tweet

Transcript

  1. Implementing Audio for Android
    Caren Chang

    View full-size slide

  2. Motivation
    Creating a more interactive experience

    View full-size slide

  3. Motivation
    Creating a more interactive experience
    Allow for personalization

    View full-size slide

  4. Motivation
    Creating a more interactive experience
    Allow for personalization
    Guide users

    View full-size slide

  5. How is Audio Processed?

    View full-size slide

  6. How is Audio Processed?
    Latency - Time delay as a signal passes
    through a system

    View full-size slide

  7. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?

    View full-size slide

  8. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?

    View full-size slide

  9. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?

    View full-size slide

  10. How is Audio Processed?
    Application App audio source

    View full-size slide

  11. How is Audio Processed?
    Application
    Audio Framework
    App audio source
    Resample
    Effects
    Mixer

    View full-size slide

  12. How is Audio Processed?
    Application
    Audio Framework
    Device Hardware
    App audio source
    Resample
    Effects
    Mixer
    DAC

    View full-size slide

  13. How is Audio Processed?
    Audio hardware consumes data at constant rate

    View full-size slide

  14. How is Audio Processed?
    Audio hardware consumes data at constant rate
    When we pass audio files to be played, we have to
    make sure it gets processed in time to meet the
    deadlines

    View full-size slide

  15. How is Audio Processed?
    Audio hardware consumes data at constant rate
    When we pass audio files to be played, we have to
    make sure it gets processed in time to meet the
    deadlines
    Otherwise, when deadlines are missed, you can get
    silent sounds or weird ‘static’ sounds

    View full-size slide

  16. How is Audio Processed?

    View full-size slide

  17. How is Audio Processed?

    View full-size slide

  18. How is Audio Processed?

    View full-size slide

  19. How is Audio Processed?

    View full-size slide

  20. How is Audio Processed?

    View full-size slide

  21. Implementing Audio Playback

    View full-size slide

  22. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View full-size slide

  23. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View full-size slide

  24. MediaPlayer
    Easily fetch, decode and play audio with minimal
    setup

    View full-size slide

  25. MediaPlayer
    Easily fetch, decode and play audio with minimal
    setup
    Supports :

    Local resources

    Internal URIs (from content resolver)

    External URLs (http streaming)

    View full-size slide

  26. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  27. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  28. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  29. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  30. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  31. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  32. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View full-size slide

  33. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors

    View full-size slide

  34. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    IDLE

    View full-size slide

  35. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    INITIALIZED

    View full-size slide

  36. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    PREPARED

    View full-size slide

  37. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();

    View full-size slide

  38. MediaPlayer : Common Errors
    Not handling onError

    View full-size slide

  39. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state

    View full-size slide

  40. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    Unsupported file types

    View full-size slide

  41. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    Unsupported file types
    Not releasing after use

    View full-size slide

  42. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View full-size slide

  43. SoundPool
    Uses the MediaPlayer service to decode the audio

    View full-size slide

  44. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory

    View full-size slide

  45. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    Able to manage number of audio streams

    IE : able to do sound mixing

    View full-size slide

  46. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    Able to manage number of audio streams

    IE : able to do sound mixing
    Can set priority when playing sound

    View full-size slide

  47. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();


    int loadedSoundId = mSoundPool.load(audioFilePath, 1);


    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

    View full-size slide

  48. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();


    int loadedSoundId = mSoundPool.load(audioFilePath, 1);


    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

    View full-size slide

  49. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();


    int loadedSoundId = mSoundPool.load(audioFilePath, 1);


    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

    View full-size slide

  50. AudioService.java

    View full-size slide

  51. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View full-size slide

  52. ExoPlayer
    Open source project (not part of Android framework)

    View full-size slide

  53. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application

    View full-size slide

  54. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    Fewer device specific issues across different
    versions of Android

    View full-size slide

  55. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    Fewer device specific issues across different
    versions of Android
    Ability to customize and extend the player

    View full-size slide

  56. ExoPlayer : Talks
    ExoPlayer : Flexible Media Playback for Android

    Google IO ‘17
    Advanced Exoplayer

    Effie Barak

    View full-size slide

  57. Being a good (App) Citizen

    View full-size slide

  58. AudioFocus
    Audio from different apps can play simultaneously
    (system does not have control over which streams
    play)

    View full-size slide

  59. AudioFocus
    Audio from different apps can play simultaneously
    (system does not have control over which streams
    play)
    If individual apps don’t properly handle audio focus it
    will result in a bad user experience

    View full-size slide

  60. AudioFocus : Scenario

    View full-size slide

  61. AudioFocus : Scenario
    User is currently in your app and playing audio

    View full-size slide

  62. AudioFocus : Scenario
    User is currently in your app and playing audio
    User goes into another media app that plays audio

    View full-size slide

  63. AudioFocus : Scenario
    User is currently in your app and playing audio
    User goes into another media app that plays audio
    If not properly handled, both apps will now play their
    audio stream and nothing will be heard

    View full-size slide

  64. AudioFocus : Proper Flow

    View full-size slide

  65. AudioFocus : Proper Flow
    Request audio focus

    View full-size slide

  66. AudioFocus : Proper Flow
    Request audio focus
    Start playback after audio focus is successfully
    granted

    View full-size slide

  67. AudioFocus : Proper Flow
    Request audio focus
    Start playback after audio focus is successfully
    granted
    Handle audio focus changes through
    onAudioFocusChangedListener()

    View full-size slide

  68. AudioFocus : Intentions

    View full-size slide

  69. AudioFocus : Intentions
    AUDIOFOCUS_GAIN : Indefinite
    AUDIOFOCUS_TRANSIENT : temporary (phone call)
    AUDIOFOCUS_TRANSIENT_MAY_DUCK : temporary
    (notification)

    View full-size slide

  70. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    View full-size slide

  71. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    View full-size slide

  72. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    View full-size slide

  73. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);


    View full-size slide

  74. Sounds on the June Oven

    View full-size slide

  75. Sounds on the June Oven
    Common sounds vs Special sounds

    View full-size slide

  76. Sounds on the June Oven
    Common sounds vs Special sounds
    Themes

    View full-size slide

  77. Sounds on the June Oven
    Common sounds vs Special sounds
    Themes
    Emergency sounds

    View full-size slide

  78. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.

    View full-size slide

  79. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    All other sounds are played with the MediaPlayer

    View full-size slide

  80. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    All other sounds are played with the MediaPlayer
    Sounds can request to be played immediately

    View full-size slide

  81. Problems Encountered

    View full-size slide

  82. Problems Encountered
    Creating pleasant experience on sliders

    View full-size slide

  83. Problems Encountered
    Creating pleasant experience on sliders
    ‘Wake’ sound

    View full-size slide

  84. Problems Encountered
    Creating pleasant experience on sliders
    ‘Wake’ sound
    Playing sound on theme changed

    View full-size slide

  85. Thanks!
    @calren24

    View full-size slide