Picking up the pieces - Overcoming Device Fragmentation

Talk given at Droidcon London, Oct 2012. Jan Berkel and Jon Schmidt.

Transcript

1. Overcoming Device Fragmentation Jan Berkel Jon Schmidt
2. • Launched in 2008 • Upload, record, share original audio content • Receive feedback from the community directly on the waveform • 10+ million official app installs over all platforms “Unmute the web”
3. SoundCloud on Android Connect to and receive updates from your account. Discover, listen, and interact with SoundCloud’s library of audio.
4. SoundCloud on Android Record sounds to your Android device, then upload and share to SoundCloud.
5. Playback + Recording Core functionality of the app
6. LESSONS LEARNED: PLAYBACK
7. v1.0
8. “Love it” “Why does this app suck so bad?”
9. “There was a problem playing that Sound” SoundCloud on Android (2010 -?)
10. MediaPlayer player = new MediaPlayer(); player.setDataSource( "http://api.soundcloud.com/tracks/123/stream" ); player.prepare(); player.start();
11. player.start() MediaPlayer App Framework mediaplayer.cpp libmedia MediaPlayerService AudioTrack audiotrack.cpp Audio Flinger / HAL ALSA? Kernel SoundCloud Java C / C++
12. OpenCore MediaPlayerService Stagefright
13. “All non-trivial abstractions, to some degree, are leaky” J Spolsky Why do we have to care?
14. OpenCore Since the beginning Mature, stable, but limited
15. OpenCORE seeking is broken
16. Stagefright Default on Android 2.2 and beyond (but OpenCore support continued)
17. We support Android 2.1 +
18. if (Build.VERSION.SDK_INT > Build.VERSION_CODES.GINGERBREAD) { isStagefright = true; } That should be easy Not exactly...
19. [media.stagefright.enable-player]: [false] [media.stagefright.enable-meta]: [false] [media.stagefright.enable-scan]: [false] [media.stagefright.enable-http]: [true] [media.stagefright.enable-rtsp]: [false] /system/build.prop
20. Frankenstein Builds
21. MediaPlayer mp = new MediaPlayer(); mp.setDataSource(String.format("http://127.0.0.1:%d/", socketPort)); mp.prepare(); mp.start(); Connect to a Local Socket if (res.indexOf("User-Agent: stagefright") >= 0) { isStagefright = true; } Read the user agent
22. v2.0
23. Stream Proxy MediaPlayer Stream Proxy Connection Manager Data Storage Socket Manager Chunk Manager
24. if (mProxy == null) { mProxy = new StreamProxy(getApp()).init().start(); } mMediaPlayer.setDataSource( String.format("http://127.0.0.1:%d/%s", mProxy.getPort(), path) ); http://code.google.com/p/npr-android-app Stream Proxy
25. • Control the connection logic • caching layer The Good The Bad • Still no control over buffer • Introduced new streaming problems Stream Proxy
26. HTTP CLIENT Y U NO WORK!? HttpClient is broken
27. try { return getHttpClient().execute(target, request); } catch (NullPointerException e) { request.abort(); throw new BrokenHttpClientException(e); } HttpClient is broken
28. • Write our own MediaPlayer • Request prioritization • Mobile friendly codecs Plans for the future
29. “this is an experimental feature and might not work on all devices”
30. LESSONS LEARNED: RECORDING
31. AUDIO CODEC CODER-DECODER mp3, aac, flac, ogg ...
32. December 2010 available codecs on Android... AMR NB
33. record raw audio, encode later.
34. open source, no patents, good quality Java port = slow
35. Second Iteration AAC
36. if (Build.SDK_INT >= 10) // use built-in AAC else // slow vorbis encoding Only available on 2.3.3
37. and everything was good. until we had to rewrite.
38. NEW RECORDING
39. REQUIREMENTS user is able to trim fade in/out
40. REQUIREMENTS pause recording and resume later more accurate waveform
41. We needed a new recording infrastructure.
42. AudioRecord Sound Recorder Vorbis Encoder Amplitude Analyzer Vorbis Decoder Native Layer ̇⽃ AudioTrack
43. arm-v5 CPU flavours arm-v7a x86 mips
44. AudioRecord Sound Recorder Vorbis Encoder Amplitude Analyzer Vorbis Decoder Native Layer ̇⽃ AudioTrack Wav Writer
45. JellyBean new low-level media APIs
46. Device API Level CPU Fragmentation
47. Putting it back together
48. CRASH REPORTS
49. BETA TESTING
50. READ THE SOURCE CODE
51. HACKS
52. DO IT YOURSELF
53. Jan Berkel Jon Schmidt http://snd.sc/droidcon-2012 www.soundcloud.com Thanks!