Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Implementing Audio on Android

Caren
November 05, 2017
190

Implementing Audio on Android

Caren

November 05, 2017
Tweet

Transcript

  1. Implementing Audio for Android
    Caren Chang

    View Slide

  2. Motivation

    View Slide

  3. Motivation
    Creating a more interactive experience

    View Slide

  4. Motivation
    Creating a more interactive experience
    Allow for personalization

    View Slide

  5. Motivation
    Creating a more interactive experience
    Allow for personalization
    Guide users

    View Slide

  6. How is Audio Processed?

    View Slide

  7. How is Audio Processed?
    Latency - Time delay as a signal passes
    through a system

    View Slide

  8. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?

    View Slide

  9. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?

    View Slide

  10. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?

    View Slide

  11. How is Audio Processed?
    Application App audio source

    View Slide

  12. How is Audio Processed?
    Application
    Audio Framework
    App audio source
    Resample
    Effects
    Mixer

    View Slide

  13. How is Audio Processed?
    Application
    Audio Framework
    Device Hardware
    App audio source
    Resample
    Effects
    Mixer
    DAC

    View Slide

  14. How is Audio Processed?
    Audio hardware consumes data at constant rate

    View Slide

  15. How is Audio Processed?
    Audio hardware consumes data at constant rate
    When we pass audio files to be played, we have to
    make sure it gets processed in time to meet the
    deadlines

    View Slide

  16. How is Audio Processed?
    Audio hardware consumes data at constant rate
    When we pass audio files to be played, we have to
    make sure it gets processed in time to meet the
    deadlines
    Otherwise, when deadlines are missed, you can get
    silent sounds or weird ‘static’ sounds

    View Slide

  17. How is Audio Processed?

    View Slide

  18. How is Audio Processed?

    View Slide

  19. How is Audio Processed?

    View Slide

  20. How is Audio Processed?

    View Slide

  21. How is Audio Processed?

    View Slide

  22. Implementing Audio Playback

    View Slide

  23. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View Slide

  24. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View Slide

  25. MediaPlayer

    View Slide

  26. MediaPlayer
    Easily fetch, decode and play audio with minimal
    setup

    View Slide

  27. MediaPlayer
    Easily fetch, decode and play audio with minimal
    setup
    Supports :

    Local resources

    Internal URIs (from content resolver)

    External URLs (http streaming)

    View Slide

  28. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  29. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  30. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  31. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  32. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  33. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  34. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }

    View Slide

  35. MediaPlayer

    View Slide

  36. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors

    View Slide

  37. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    IDLE

    View Slide

  38. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    INITIALIZED

    View Slide

  39. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    PREPARED

    View Slide

  40. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();

    View Slide

  41. MediaPlayer : Common Errors
    Not handling onError

    View Slide

  42. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state

    View Slide

  43. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    Unsupported file types

    View Slide

  44. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    Unsupported file types
    Not releasing after use

    View Slide

  45. View Slide

  46. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View Slide

  47. SoundPool

    View Slide

  48. SoundPool
    Uses the MediaPlayer service to decode the audio

    View Slide

  49. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory

    View Slide

  50. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    Able to manage number of audio streams

    IE : able to do sound mixing

    View Slide

  51. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    Able to manage number of audio streams

    IE : able to do sound mixing
    Can set priority when playing sound

    View Slide

  52. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();


    int loadedSoundId = mSoundPool.load(audioFilePath, 1);


    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

    View Slide

  53. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();


    int loadedSoundId = mSoundPool.load(audioFilePath, 1);


    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

    View Slide

  54. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();


    int loadedSoundId = mSoundPool.load(audioFilePath, 1);


    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);

    View Slide

  55. AudioService.java

    View Slide

  56. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer

    View Slide

  57. ExoPlayer

    View Slide

  58. ExoPlayer
    Open source project (not part of Android framework)

    View Slide

  59. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application

    View Slide

  60. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    Fewer device specific issues across different
    versions of Android

    View Slide

  61. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    Fewer device specific issues across different
    versions of Android
    Ability to customize and extend the player

    View Slide

  62. ExoPlayer : Talks
    ExoPlayer : Flexible Media Playback for Android

    Google IO ‘17
    Advanced Exoplayer

    Effie Barak

    View Slide

  63. Being a good (App) Citizen

    View Slide

  64. AudioFocus

    View Slide

  65. AudioFocus
    Audio from different apps can play simultaneously
    (system does not have control over which streams
    play)

    View Slide

  66. AudioFocus
    Audio from different apps can play simultaneously
    (system does not have control over which streams
    play)
    If individual apps don’t properly handle audio focus it
    will result in a bad user experience

    View Slide

  67. AudioFocus : Scenario

    View Slide

  68. AudioFocus : Scenario
    User is currently in your app and playing audio

    View Slide

  69. AudioFocus : Scenario
    User is currently in your app and playing audio
    User goes into another media app that plays audio

    View Slide

  70. AudioFocus : Scenario
    User is currently in your app and playing audio
    User goes into another media app that plays audio
    If not properly handled, both apps will now play their
    audio stream and nothing will be heard

    View Slide

  71. AudioFocus : Proper Flow

    View Slide

  72. AudioFocus : Proper Flow
    Request audio focus

    View Slide

  73. AudioFocus : Proper Flow
    Request audio focus
    Start playback after audio focus is successfully
    granted

    View Slide

  74. AudioFocus : Proper Flow
    Request audio focus
    Start playback after audio focus is successfully
    granted
    Handle audio focus changes through
    onAudioFocusChangedListener()

    View Slide

  75. AudioFocus : Intentions

    View Slide

  76. AudioFocus : Intentions
    AUDIOFOCUS_GAIN : Indefinite
    AUDIOFOCUS_TRANSIENT : temporary (phone call)
    AUDIOFOCUS_TRANSIENT_MAY_DUCK : temporary
    (notification)

    View Slide

  77. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    View Slide

  78. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    View Slide

  79. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    View Slide

  80. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();


    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);


    View Slide

  81. Sounds on the June Oven

    View Slide

  82. Sounds on the June Oven
    Common sounds vs Special sounds

    View Slide

  83. Sounds on the June Oven
    Common sounds vs Special sounds
    Themes

    View Slide

  84. Sounds on the June Oven
    Common sounds vs Special sounds
    Themes
    Emergency sounds

    View Slide

  85. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.

    View Slide

  86. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    All other sounds are played with the MediaPlayer

    View Slide

  87. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    All other sounds are played with the MediaPlayer
    Sounds can request to be played immediately

    View Slide

  88. Problems Encountered

    View Slide

  89. Problems Encountered
    Creating pleasant experience on sliders

    View Slide

  90. Problems Encountered
    Creating pleasant experience on sliders
    ‘Wake’ sound

    View Slide

  91. Problems Encountered
    Creating pleasant experience on sliders
    ‘Wake’ sound
    Playing sound on theme changed

    View Slide

  92. Thanks!
    @calren24

    View Slide