Statistics and Metrics in WebRTC

WebRTC Meetup Yokohama,
Varun Singh
CEO callstats.io

Transcript

1. Statistics in WebRTC Varun Singh CEO 04. 11. 2015
2. Announcement ¥ 0 plan for developers 2
3. http://geek-and-poke.com/
4. Why measure? 4 audio only call
5. Performance Monitoring Endpoint Endpoint Middlebox MCU, SFU, TURN, … Monitor Monitor Monitor Third party Monitor Operations & Management System Implement RTCP Extension Reports http://tools.ietf.org/html/rfc6792 https://tools.ietf.org/html/draft-ietf-xrblock-rtcweb-rtcp-xr-metrics
6. HTML APIs • HTMLVideoElement Extensions • http://www.w3.org/TR/2014/CR-media-source-20140717/#htmlvideoelement-extensions • droppedVideoFrames, corruptedVideoFrames, totalFrameDelay • WebRTC Statistics API • (work in progress) http://www.w3.org/TR/2015/WD-webrtc-stats-20150206/ • PeerConnection Metrics: RTP, datachannel • Transport Metrics: active and passive candidates • Security cipher suite
7. Edge Stats (ORTC) Lots of data still set to zero. but encouraging to see the data
8. cross browser • adapter.js is your friend
9. Monitoring What? • Per-stream media quality • Transport quality • Annoyances
10. What to measure? • Network metrics • bits per second, RTT, jitter, packet losses, … • Multimedia pipeline metrics • playout delay, frames metrics, … • Quality Models from metrics • Annoyances • resolution/frame rate changes, interface changes, … • failures (NATs, insufficient capacity for carrying media, …) • user feedback 10 CAVEAT: not all WebRTC applications care about the same metrics
11. Low hanging fruit • Endpoint monitoring (Traffic lights) • colors to show connection is active/inactive • off/online • good/bad/ugly
12. Transport Metrics
13. Diagnostics • characterise an impairment? • we start with an hypothesis… • implemented as decision trees • observe how often it occurs • within a call, across calls, geographies, … • How to characterise quality of the calls? 13
14. Disruptions Disruption: loss of connectivity when network interfaces change, low available capacity,or high delay The light grey vertical lines show disruption, highlighted by the red bounding boxes.
15. Intermittent Disruptions
16. Disruptions and user behaviour User Behaviour: The user tries to correct for the disruption by turning on and off video Zoom in to the events to view disruptions
17. Endpoint Churn 25% of the conferences has some users leave and rejoin.
18. Session Failure
19. Annoyances Setup times: waiting for the conference to begin: NAT traversal, negotiation, etc. Note: gathering and connectivity checks can happen concurrently
20. • Analytics for WebRTC • 3 lines of code —> 5 minutes. 1. include <script> 2. initialize() —> needs registration keys 3. addNewFabric(pc, …) —> call started • optionally: send user events: audio muted/unmuted, video paused/resumed, call terminated, call held. • optionally: reportError() —> call failed to be setup due to SDP or Firewalls. • optionally: associateMstWithUserID() —> bridges using multiple media streams in a single peerconnection may want to correlate mediastream quality across endpoints. • optionally: collect and send user feedback http://www.callstats.io/api/