[COSCUP 2013][FxOS] Multi channel & Audio competing

Transcript

1. Web Developer or OS Developer?
   Experience from FirefoxOS
   Alive COSCUP 2013
   
2. me
   @alivedise http://alivedise.github.io http://github.com/alivedise alive@mozilla.com alegnadise@gmail.com ~2.5year Frontend Exp. ᡒ㘸COSCUPҰఆཁ์䤕
   
3. Operating System
   ?
   http://www.flickr.com/photos/klaikong/6041270516/
   
4. Operating System
   
5. FirefoxOS….OS?
   Storage Management fileSystem(yet) DeviceStorageWatcher Memory Management Page Visibility mozAppType User Interface Multi Window Management Statusbar System Overlay App Overlay Web Activity Keyboard Hardware Resource Management Wi-Fi Radio GPS Bluetooth Power CPU WakeLock Audio/Media Multi-Tasking mozBrowser remote=true Card view
6. • Multi Channel • Audio Competing • Background Play Audio Management
7. The beginning of everything..
   http://bugzil.la/805333
   
8. 805333
   
9. Why do we need multi- channel on OS?
   
10. Sounds within FxOS - <audio/>
    
11. Sounds within FxOS - radio
    
12. Sounds within FxOS - alarm
    
13. Sounds within FxOS - voice
    
14. The world of single channel..
    Peace
    
15. But..
    Too peaceful.
    
16. This means:
    When you mute the device, your alarm also is muted. So..
    
17. Overslept ̇̇̇~
    Zzzzz..
    
18. How does OS avoid that?
    Different settings for audios on different purposes.
    
19. On the other hand..
    Competing
    
20. On Desktop browser, we don’t need competing.
    
21. How about mobile OS?
    
22. Cases
    • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.
23. It’s all about UX.
    User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..
    
24. Android solution: audio focus
    • http://developer.android.com/training/managing -audio/audio-focus.html • App is responsible to request the audio focus on demand and release the focus actively.
25. Problems
    • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …
    
26. Proposal
    • New attribute for media tags. • Background play and competing needs permission check. • Expose current occupied media info to system.
27. Page Visibility
    • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs. • A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)
    
28. mozAudioChannelType
    telephony alarm notification content normal(default)
29. Multi channel volume settings
    • ringer, desktop-notification audio.volume.notification • music, FMRadio audio.volume.content • alarm audio.volume.alarm • telephony audio.volume.voice • Bluetooth SCO audio.volume.bt_sco
30. Usage
    OR
    manifest.webapp
    app.html
    app.js
    
31. Interrupted Event
    
32. Current Competing Policies
    •  Channel with higher priority interrupts the lower channel. •  Foreground page is always playable. But interruption occurs or not depends on the channel of the media element. •  When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!) •  Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’. content normal
33. Gecko side mozAudioChannelType audio App nsIDOMHTMLAudioElement nsIDOMHTMLMediaElement nsHTMLMediaElement nsIAudioChannelAgentCallback nsIAudioChannelAgent nsBuiltinDecoder nsMediaOmxDecoder nsMediaDecoder DataSource MediaStreamSource MediaResource - SetReadMode() create IOMX OMX MediaExtractor OMXCodec MediaSource - Tell() - Seek() - Read() - GetLength() hw codec sw codec MediaBuffer - CanPlayChanged() - "mozinterruptbegin" event - "mozinterruptend" event OmxDecoder nsHTMLAudioElement nsIChannel create nsMediaOmxReader nsBuiltinDecoderReader create create MediaOmxStateMachine nsBuiltinDecoderStateMachine nsDecoderStateMachine create nsIThread nsIThread control create nsDocument - NotifyOwnerDocumentActivityChanged() FileMediaResource ChannelMediaResource nsAudioStream nsNativeAudioStream libsydneyaudio AudioTrack AudioFlinger - Write() - Pause() - Resume() - IsPaused() - Drain() - GetMinWriteSize() - GetPosition() - SetVolume() platform: Firefox OS(gonk) branch: Firefox OS 1.01 (b2g18) will be replaced to libcubeb in v1.2 Audio Streams Diagram
    https://github.com/sotaroikeda/firefox-diagrams
    
34. Since API is implemented..
    World peace?
    
35. No…
    Don’t forget the dependency tree.
    
36. Incoming Issues
    • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …
37. Policy for Silent mode
    • Easy for single channel • Multi channels • The default channel for volume rockers is ‘notification’. (Same as Android) • Silent mode interesting channels • notification • content/normal • Alarm is not affected by silent mode.
38. Policy for video element
    • <video> doesn’t need to play at background. • Currently, use ‘visibilitychange’ event. • <video> by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.
    
39. Background play issues
    • Existing web pages: “Hey, I’m a music site!” • grooveshark • youtube • … • How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.
40. CE Certification
    •  EN 50332-1 •  Output level through earphone should be under 85 dBA. •  Prompt a warning when earphone is plugged and the volume is greater than 85 dBA. •  The cool down time of CE warning popup is continuous 20 hours music listening.
    
41. OS level policy for CE
    • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.
    
42. After all..
    
43. UX logic in backend and frontend
44. Find out solutions for existing web sites without new API.
45. The scope is undefined before you look into it.
46. To be honest, I wrote few code in these issues.
    But lots of online/offline discussions.
    
47. The position of Frontend
    https://speakerdeck.com/josephj/f2e-the-keystone
    
48. • It doesn’t matter you are making a web site, a web app or an operating system. • Frontend is the one connecting UX and Backend, and struggle for ideal design and real requirement.
    
49. Thanks!
    Not the end, to be continued..