Implementing Audio on Android

Transcript

1. Implementing Audio for Android
   Caren Chang
   

   View Slide

2. Motivation
   

   View Slide

3. Motivation
   Creating a more interactive experience
   

   View Slide

4. Motivation
   Creating a more interactive experience
   Allow for personalization
   

   View Slide

5. Motivation
   Creating a more interactive experience
   Allow for personalization
   Guide users
   

   View Slide

6. How is Audio Processed?
   

   View Slide

7. How is Audio Processed?
   Latency - Time delay as a signal passes
   through a system
   

   View Slide

8. Latency - Time delay as a signal passes
   through a system
   How is Audio Processed?
   

   View Slide

9. Latency - Time delay as a signal passes
   through a system
   How is Audio Processed?
   

   View Slide

10. Latency - Time delay as a signal passes
    through a system
    How is Audio Processed?
    

    View Slide

11. How is Audio Processed?
    Application App audio source
    

    View Slide

12. How is Audio Processed?
    Application
    Audio Framework
    App audio source
    Resample
    Effects
    Mixer
    

    View Slide

13. How is Audio Processed?
    Application
    Audio Framework
    Device Hardware
    App audio source
    Resample
    Effects
    Mixer
    DAC
    

    View Slide

14. How is Audio Processed?
    Audio hardware consumes data at constant rate
    

    View Slide

15. How is Audio Processed?
    Audio hardware consumes data at constant rate
    When we pass audio files to be played, we have to
    make sure it gets processed in time to meet the
    deadlines
    

    View Slide

16. How is Audio Processed?
    Audio hardware consumes data at constant rate
    When we pass audio files to be played, we have to
    make sure it gets processed in time to meet the
    deadlines
    Otherwise, when deadlines are missed, you can get
    silent sounds or weird ‘static’ sounds
    

    View Slide

17. How is Audio Processed?
    

    View Slide

18. How is Audio Processed?
    

    View Slide

19. How is Audio Processed?
    

    View Slide

20. How is Audio Processed?
    

    View Slide

21. How is Audio Processed?
    

    View Slide

22. Implementing Audio Playback
    

    View Slide

23. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer
    

    View Slide

24. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer
    

    View Slide

25. MediaPlayer
    

    View Slide

26. MediaPlayer
    Easily fetch, decode and play audio with minimal
    setup
    

    View Slide

27. MediaPlayer
    Easily fetch, decode and play audio with minimal
    setup
    Supports :

    Local resources

    Internal URIs (from content resolver)

    External URLs (http streaming)
    

    View Slide

28. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

29. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

30. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

31. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

32. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

33. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

34. MediaPlayer
    void playAudioFile(String audioFilePath) {

    MediaPlayer mediaPlayer = new MediaPlayer();

    try {

    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start();

    } catch (IOException e) {

    Log.e(TAG, "Error playing audio file: " + audioFilePath);

    mediaPlayer.release();

    }

    }
    

    View Slide

35. MediaPlayer
    

    View Slide

36. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    

    View Slide

37. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    IDLE
    

    View Slide

38. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    INITIALIZED
    

    View Slide

39. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    PREPARED
    

    View Slide

40. MediaPlayer
    MediaPlayer is state based, can easily caused
    undesired behaviors
    mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);

    mediaPlayer.setDataSource(audioFilePath);

    mediaPlayer.prepare();

    mediaPlayer.start(); or mediaPlayer.pause() or mediaPlayer.seekTo();
    

    View Slide

41. MediaPlayer : Common Errors
    Not handling onError
    

    View Slide

42. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    

    View Slide

43. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    Unsupported file types
    

    View Slide

44. MediaPlayer : Common Errors
    Not handling onError
    Performing actions in wrong state
    Unsupported file types
    Not releasing after use
    

    View Slide

45. View Slide

46. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer
    

    View Slide

47. SoundPool
    

    View Slide

48. SoundPool
    Uses the MediaPlayer service to decode the audio
    

    View Slide

49. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    

    View Slide

50. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    Able to manage number of audio streams

    IE : able to do sound mixing
    

    View Slide

51. SoundPool
    Uses the MediaPlayer service to decode the audio
    Collection of samples from the APK that can be loaded
    into memory
    Able to manage number of audio streams

    IE : able to do sound mixing
    Can set priority when playing sound
    

    View Slide

52. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();

    

    int loadedSoundId = mSoundPool.load(audioFilePath, 1);

    

    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);
    

    View Slide

53. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();

    

    int loadedSoundId = mSoundPool.load(audioFilePath, 1);

    

    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);
    

    View Slide

54. SoundPool
    SoundPool mSoundPool = new SoundPool.Builder()

    .setAudioAttributes(mAudioAttributes)

    .setMaxStreams(MAX_SOUND_STREAMS)

    .build();

    

    int loadedSoundId = mSoundPool.load(audioFilePath, 1);

    

    mSoundPool.play(loadedSoundId,

    volumeToPlaySound, volumeToPlaySound,

    REGULAR_PRIORITY_SOUND, NO_LOOP, DEFAULT_PLAYBACK_RATE);
    

    View Slide

55. AudioService.java
    

    View Slide

56. Implementing Audio Playback
    MediaPlayer
    SoundPool
    ExoPlayer
    

    View Slide

57. ExoPlayer
    

    View Slide

58. ExoPlayer
    Open source project (not part of Android framework)
    

    View Slide

59. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    

    View Slide

60. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    Fewer device specific issues across different
    versions of Android
    

    View Slide

61. ExoPlayer
    Open source project (not part of Android framework)
    Pros:

    Ability to update player along with application
    Fewer device specific issues across different
    versions of Android
    Ability to customize and extend the player
    

    View Slide

62. ExoPlayer : Talks
    ExoPlayer : Flexible Media Playback for Android

    Google IO ‘17
    Advanced Exoplayer

    Effie Barak
    

    View Slide

63. Being a good (App) Citizen
    

    View Slide

64. AudioFocus
    

    View Slide

65. AudioFocus
    Audio from different apps can play simultaneously
    (system does not have control over which streams
    play)
    

    View Slide

66. AudioFocus
    Audio from different apps can play simultaneously
    (system does not have control over which streams
    play)
    If individual apps don’t properly handle audio focus it
    will result in a bad user experience
    

    View Slide

67. AudioFocus : Scenario
    

    View Slide

68. AudioFocus : Scenario
    User is currently in your app and playing audio
    

    View Slide

69. AudioFocus : Scenario
    User is currently in your app and playing audio
    User goes into another media app that plays audio
    

    View Slide

70. AudioFocus : Scenario
    User is currently in your app and playing audio
    User goes into another media app that plays audio
    If not properly handled, both apps will now play their
    audio stream and nothing will be heard
    

    View Slide

71. AudioFocus : Proper Flow
    

    View Slide

72. AudioFocus : Proper Flow
    Request audio focus
    

    View Slide

73. AudioFocus : Proper Flow
    Request audio focus
    Start playback after audio focus is successfully
    granted
    

    View Slide

74. AudioFocus : Proper Flow
    Request audio focus
    Start playback after audio focus is successfully
    granted
    Handle audio focus changes through
    onAudioFocusChangedListener()
    

    View Slide

75. AudioFocus : Intentions
    

    View Slide

76. AudioFocus : Intentions
    AUDIOFOCUS_GAIN : Indefinite
    AUDIOFOCUS_TRANSIENT : temporary (phone call)
    AUDIOFOCUS_TRANSIENT_MAY_DUCK : temporary
    (notification)
    

    View Slide

77. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();

    

    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);
    

    View Slide

78. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();

    

    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);
    

    View Slide

79. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();

    

    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);
    

    View Slide

80. AudioFocus : Implementation
    AudioFocusRequest mAudioFocusRequest =

    new AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)

    .setAudioAttributes(mAudioAttributes)

    .setAcceptsDelayedFocusGain(true)

    .setOnAudioFocusChangeListener(...)

    .build();

    

    int focusRequest = mAudioManager.requestAudioFocus(mAudioFocusRequest);

    

    View Slide

81. Sounds on the June Oven
    

    View Slide

82. Sounds on the June Oven
    Common sounds vs Special sounds
    

    View Slide

83. Sounds on the June Oven
    Common sounds vs Special sounds
    Themes
    

    View Slide

84. Sounds on the June Oven
    Common sounds vs Special sounds
    Themes
    Emergency sounds
    

    View Slide

85. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    

    View Slide

86. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    All other sounds are played with the MediaPlayer
    

    View Slide

87. Sounds on the June Oven
    Common sounds (tap, cancel, accept, back) are
    eagerly loaded into a SoundPool for low latency
    playback.
    All other sounds are played with the MediaPlayer
    Sounds can request to be played immediately
    

    View Slide

88. Problems Encountered
    

    View Slide

89. Problems Encountered
    Creating pleasant experience on sliders
    

    View Slide

90. Problems Encountered
    Creating pleasant experience on sliders
    ‘Wake’ sound
    

    View Slide

91. Problems Encountered
    Creating pleasant experience on sliders
    ‘Wake’ sound
    Playing sound on theme changed
    

    View Slide

92. Thanks!
    @calren24
    

    View Slide