Build WebRTC iOS Gateway on Browser

Building WebRTC - iOS Safari Gateway on PC Web Browser, using webaudio, canvas, websocket.
This is a presentation for WebRTC Meetup Tokyo 14 at Mar. 23, 2017

Transcript

1. WebRTC Meetup Tokyo #14
   Build WebRTC - iOS Gateway on Browser
   INFOCOM CORPORATION
   Masashi Ganeko
   @massie_g
   Mar. 23th, 2017
   1
   

   View Slide

2. About me
   •  Masashi Ganeko / @massie_g
   –  working for a research team of INFOCOM CORPORATION
   •  WriRng a series of WebRTC on HTML5Experts.jp (HTML5 tech site)
   –  h[ps://html5experts.jp/series/webrtc2016/
   •  English PresentaRon for WebRTC (2013-2017)
   –  h[ps://speakerdeck.com/mganeko
   •  Japanese PresentaRon for WebRTC (2013-2017)
   –  h[p://www.slideshare.net/mganeko
   2
   

   View Slide

3. Topic of today
   •  iOS Browser does not support WebRTC yet
   –  Webkit is on the way, but Safari may take much longer
   •  It is possible to communicate today, with other features
   –  Canvas, Web Audio, WebSocket
   •  Thanks for preceding trials:
   –  Audio: "Using WebAudio for pseudo WebRTC" @leader22
   •  h[p://leader22.github.io/slides/webaudio_tokyo-1/
   –  Video: Snapshoto from "Introducing WebRTC SFU Sora" @voluntas
   •  h[ps://gist.github.com/voluntas/0d6621d15947a24e710b0610093a5d20
   3
   

   View Slide

4. DEMO
   •  PC Chrome ßà iOS Safari
   •  Source code
   –  h[ps://github.com/mganeko/webrtcexpjp/tree/master/gateway
   •  (There are some problems with handling portrait
   photo)
   4
   

   View Slide

5. PC Chrome à iOS Safari (1) Video
   5
   getUserMedia()
   MediaStream
   
   
   Blob JPEB
   setInterval()
   drawImage()
   toBlob()
   Worker
   WebSocket
   (socket.io)
   Blob
   JPEB
   socket.io
   Server
   postMessage()
   Worker
   WebSocket
   (socket.io)
   ArrayBuffer
   postMessage()
   ArrayBuffer
   
   Blob
   createObjectURL()
   PC Chrome iOS Safari
   

   View Slide

6. PC Chrome à iOS Safari (2) Audio
   6
   getUserMedia()
   MediaStream
   Worker
   WebSocket
   (socket.io)
   socket.io
   Server
   Worker a
   WebSocket
   (socket.io)
   PC Chrome iOS Safari
   WebAudio
   ScriptProcessor [Float32Array]
   MediaStreamAudioSourceNode
   [Uint8Array]
   compress
   postMessage()
   ArrayBuffer
   (decompress)
   WebAudio
   6
   AudioBufferSource
   ArrayBuffer
   decoceAudioData()
   AudioDesRnaRonNode
   postMessage()
   

   View Slide

7. PC Chrome ß iOS Safari : Image
   7
   
   toDataURL ()
   Worker
   WebSocket
   (socket.io)
   DataURL
   socket.io Server
   postMessage()
   
   
   
   
   drawImage()
   DataURL
   toBlob ()
   
   DataURL
   img.src
   Worker
   WebSocket
   (socket.io)
   postMessage()
   DataURL
   iOS Safari
   PC Chrome
   

   View Slide

8. Reduce data size: Video/Image
   •  While using TCP, delay will occur on high traffic condiRon
   –  It is not preferable for real-Rme video/audio communicaRon
   –  Too keep enough margin, data should be reduced
   •  Image 240x180 / sec
   –  RGB RAW Image …　240x180x24bit = 1 Mbit, 129 kb
   –  PNG … about 760 kbit, 96 kb
   –  JPEG … about 110 kbit, 14 kb
   –  → reduce frequency to once per 2 sec. 　… 56 kbit/sec
   8
   

   View Slide

9. Reducing Image data
   9
   1/8
   

   View Slide

10. Reduce data size: Audio
    10
    •  Sampling rate of WebAudio is plaporm dependent. ( NOT browser
    dependent )
    –  44.1kHz (Mac)　←　Use this rate for calculaRon
    –  48kHz (Windows, Linux, iOS)
    •  Compression
    –  WebAudio original data …44.1kHz × 32bit float = 1411 kbps
    –  16bit liner PCM … 44.1kHz × 16bit Integer = 705 kbps
    •  CD is 44.1kHz ×16bit × 2ch(Stereo) = 1411 kbps
    –  Thin out samples to reduce rate to 22 kHz … 350 kbps
    –  μ-law compression: 22kHz, 8bit Integer … 175 kbps
    •  G.711 (PCMU) : 8kHz, 8bit Integer … 64kbps
    –  mp3 compression … use 36 kbps bitrate
    

    View Slide

11. 11
    1/40
    Reducing Audio data
    

    View Slide

12. μ-law algorithm
    •  Irreversible audio compress algorithm
    •  Reduce bits of each sample
    •  Used in G.711 for VoIP
    –  G.711: Sampling rate 8kHz → bitrate 64kbps
    •  Wikipedia
    –  h[ps://en.wikipedia.org/wiki/G.711
    –  PCMU … Use μ-law in North America and Japan
    –  PCMA … Use A-law in Europe
    •  22kHz μ-law is enough for WiFi, but not enough for mobile
    •  Ex) Trying primiRve audio compression with WebAudio (Japanese lang.)
    –  h[p://qiita.com/massie_g/items/49183a03b015b9ea27eb
    12
    

    View Slide

13. Logarithm representaRon in μ-law
    13
    Linear 8 steps (3bit PCM)
    Logarithm 8 steps: fine for small, rough
    for big
    

    View Slide

14. μ-law algorithm
    14
    μ = 255（8ビット）
    

    View Slide

15. PCMU/PCMA in WebRTC
    15
    [Firefox SDP]
    a=rtpmap:109 opus/48000/2
    a=rtpmap:9 G722/8000/1
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    [Chrome SDP]
    a=rtpmap:111 opus/48000/2
    a=rtcp-v:111 transport-cc
    a=fmtp:111 minpRme=10;useinbandfec=1
    a=rtpmap:103 ISAC/16000
    a=rtpmap:104 ISAC/32000
    a=rtpmap:9 G722/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:106 CN/32000
    a=rtpmap:105 CN/16000
    a=rtpmap:13 CN/8000
    a=rtpmap:126 telephone-event/8000
    

    View Slide

16. G.722
    •  Sampling rate:16kHz, Bitrate:48, 56, 64kbps
    •  ADPCM (adapRve differenRal pulse code modulaRon)
    •  Compression algorithm
    –  Calculate diff of previous sample
    •  Should be small value (than sample)
    –  Use logarithm for difference
    •  Fine for small value
    •  Rough for big value
    16
    Linear PCM ADPCM
    ケータイWatch ケータイ用語の基礎知識より
    h[p://k-tai.watch.impress.co.jp/cda/arRcle/keyword/2936.html
    

    View Slide

17. Use lamejs for MP3 Compression
    •  To reduce audio size, use MP3 compression
    •  lamejs
    –  h[ps://github.com/zhuker/lamejs
    •  Not streaming, short length MP3 files
    •  Input: Array of 16bit integer
    –  Specify channels(Mono/Stereo), Sampling rate for input data
    •  Output: Array of 8bit integer
    –  Specify bitrate (kbps) for output
    –  Min limit：　bitrate >= 32kbps(22kHz input), bitrate >= about 36kbps (24kHz Input)
    17
    

    View Slide

18. Sample from GitHub
    var channels = 1, sampleRate = 44100, kbps = 128; // mono, 44.1kHz, 128kbps
    var mp3encoder = new lamejs.Mp3Encoder(channels, sampleRate, kbps);
    var sampleBlockSize = 1152; // be[er to use 　576 * n
    var samples = new Int16Array(44100); // 1sec sample
    var mp3Data = [];
    for (var i = 0; i < samples.length; i += sampleBlockSize) {
    sampleChunk = samples.subarray(i, i + sampleBlockSize);
    var mp3buf = mp3encoder.encodeBuffer(sampleChunk);
    if (mp3buf.length > 0) { mp3Data.push(mp3buf); }
    }
    var mp3buf = mp3encoder.flush(); //finish wriRng mp3
    if (mp3buf.length > 0) { mp3Data.push(new Int8Array(mp3buf)); }
    18
    

    View Slide

19. Tips to use lamejs
    •  Reduce overhead of MP3 data
    –  Header + Tag = 132byte
    –  Longer block of audio to convert
    –  Max limit of block for WebAudio ScriptProcessor : 16384 sample for each event
    •  à 370 ms (44.1kHz), 340 ms (48kHz)
    19
    Header Compressed data MP3 tag
    4byte 128byte
    1500 - 2000 byte
    •  Reduce sample before compress：　44.1kHz→22.05kHz, 48kHz→24kHzに
    •  Consider Min limit for output bitreate
    •  bitrate >= 32kbps(22kHz input), bitrate >= about 36kbps (24kHz Input)
    •  Use WebWorker for compress … compression Rme is 50ms - 100ms / block
    

    View Slide

20. A|er reducing data → stable transfer
    •  Image … 56kbps
    •  Audio … 36 kbps
    20
    92kbps Total
    1/40
    1/8
    

    View Slide

21. Combine Browser MCU and iOS Gateway
    21
    Browser
    A
    Browser
    B
    Browser
    C
    MCU
    iOS Safari
    Image/2sec, MP3 Audio
    Image
    WebRTC Meetup Tokyo #11
    Build WebRTC MCU on browser
    h[ps://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser
    h[ps://speakerdeck.com/mganeko/build-webrtc-mcu-on-browser
    Browser
    

    View Slide

22. DEMO
    •  PC Chrome ßà Browser MCU ßà iOS Safari
    22
    

    View Slide

23. Inside of MCU Server：Video
    23
    RTCPeerConnecRon A
    MediaStream
    
    element
    element
    drawImage()
    
    element
    drawImage()
    Worker
    WebSocket
    (socket.io)
    DataURL
    Blob JPEB
    toBlob()
    Blob JPEB
    from iOS
    to iOS
    setInterval()
    Keep drawing with
    requestAnimaRonFrame()
    

    View Slide

24. Inside of MCU Server：Audio
    24
    RTCPeerConnecRon A
    MediaStream
    MediaStreamAudioSourceNode MediaStreamAudioSourceNode
    MediaStreamAudioSourceNode
    ScriptProcessor
    mix
    Worker
    WebSocket
    (socket.io)
    MP3
    [Uint8Array]
    [Uint16Array]
    [Flaoat32Array]
    lamejs
    to iOS
    create with AudioContext .
    createMediaStreamSource()
    

    View Slide

25. Inside of MCU Server: Recording
    25
    MediaStream
    (Video A+B+C+D)
    element Same as sending to remote peers
    Web Audio API
    MediaStream
    (Audio A+B+C)
    Mix all audio
    for recording
    videoTracks[] MediaStreamTrack
    MediaStreamTrack
    aidioTracks[]
    MediaStream
    new MediaStream()
    MediaStream.addTrack()
    MediaStream.addTrack()
    MediaRecorder
    WebM
    

    View Slide

26. Conclusion
    •  iOS Browser does not support WebRTC yet… So What?
    –  The GOAL is not WebRTC, but "CommunicaRon" with iOS user
    •  Try everything what you have now!
    –  Most advantage or "Web"RTC is flexibility to combine with other Web
    features, such as:
    –  Canvas, WebAudio, WebSocket
    •  It was interesRng to follow history of audio codecs
    26
    

    View Slide

27. Thank you!
    27
    

    View Slide