WebRTC @ Medeo

An overview of how we use WebRTC at Medeo, the problems we faced at times where both the standard and APIs were still evolving, and how we got around them. This deck also mentions how things could be done now and includes advice for situations where you get stuck.

Transcript

1. WebRTC @ Medeo The best of two exciting years Photo credit: TrevorMay.ca
2. What is Medeo? • Connects patients with their healthcare providers online ◦ Video visits ◦ Secure messaging • Walk-in clinic by Equinoxe • SAAS product for healthcare providers • Available on Chrome, iOS and Android
3. Features built on top of WebRTC • Self-test • Packet loss detection • Stats collection w/ ELK and RabbitMQ • HTML5 Fullscreen API • Background audio mode (iOS only) • Screenshots via HTML5 Canvas
4. A quick demo
5. Timeline 01/2013 - Launch as private beta with proprietary video technology 03/2013 - Official launch 04/2013 - WebRTC for Chrome (behind a feature flag, with fallback) 06/2013 - WebRTC for Chrome (default, with fallback) 09/2013 - WebRTC for Firefox (with fallback when TURN would be needed) 11/2013 - WebRTC for Android 01/2014 - WebRTC for iOS 03/2014 - WebRTC for Chrome is the only supported Browser 12/2014 - WebRTC Statistics
6. • Browser – Browser ◦ Firewall / NAT traversal (TURN/STUN) ◦ Crashes between Firefox - Chrome • Browser – Mobile ◦ Media constraints support of getUserMedia() API ◦ Availability of DTLS vs. SDES encryption ◦ Codec support (VP8, H264, Opus, ISAC) Enforce Chrome until WebRTC is ready yet An evolving standard
7. An evolving standard (cont’d) • Incomplete Native APIs • Mobile-specific features were missing ◦ Pause the capturer ◦ Switch the camera ◦ We worked around it • iOS was audio-only when we started ◦ No video capturer or renderer ◦ No getStats API ◦ We rolled our own Should be all good by now
8. • Huge build system with lots of dependencies ◦ Requires Linux for Android, ◦ Mac OS X for iOS ◦ Lack of documentation ◦ Breaks frequently • Very time consuming ◦ Slow checkout, long build time (esp. fat builds) • 32-bit only (until recently) Building the native code package
9. • Manually ◦ Instructions at webrtc.org/native-code (give it a try) ◦ Look at talk/build/build_ios_libs.sh for generating iOS FAT builds • github.com/pristineio/webrtc-build-scripts ◦ Vagrant support for building Android from non-Linux machines • libjingle_peerconnection on cocoapods Building the native code (cont’d)
10. Performance on mobile (esp. iOS) • Lots of audio and video jitter (+crashes) caused by • Lack of hardware acceleration (VP8) • Poor bandwidth estimation (videoBwe) • Poor CPU adaptation • Resource-intensive audio codec (OPUS)
11. Performance on mobile (cont’d) • Use ISAC if one party is on mobile ◦ You have to edit the session description for that • Limit frame rate to 15 fps • Limit resolution to 320x240 • Update to trunk (~r9200)? ◦ Improved videoBwe and CPU adaption, etc.
12. The user-media prompt
13. Chrome flags (from webrtc.org/testing): • --allow-file-access-from-files ◦ allows getUserMedia() to be called from file:// URLs. • --use-fake-ui-for-media-stream ◦ avoids the need to grant camera/microphone permissions. • --use-fake-device-for-media-stream ◦ feeds a test pattern to getUserMedia() instead of live camera input. • --use-file-for-fake-video-capture=path/to/file.y4m ◦ feeds a Y4M test file to getUserMedia() instead of live camera input. Integration testing
14. What to do if I’m stuck? • Issue tracker (Star-based prioritization) • discuss-webrtc • Reference implementation ◦ AppRTCDemo, apprtc.appspot.com, etc. • Read source code (and/or patch it) • Study W3C and IETF drafts ◦ TURN, STUN, ICE, WebRTC 1.0 & 1.1, etc. • Or just wait…
15. Thank you!