Extreme testing of WebRTC Applications

This material is presented at WebRTC Meetup Japan:
https://44145dc9ce5b767cd2eb8b7683.doorkeeper.jp/events/33154

You can see video from:
https://youtu.be/02wAz0Ii-ZU?t=1h33m21s

Transcript

1. "Extreme" Tes+ng of WebRTC Applica+ons

   2015/11/4(Wed) @WebRTC Meetup Japan
   iwashi86(Yoshimasa Iwase)
   atakigawa(Akinori Takigawa)
   

   View Slide

2. ■Name
   Yoshimasa Iwase / @iwashi86
   ■Work
   NTT Communications,
   DevOps Engineer of SkyWay
   ■Name
   Akinori Takigawa / @kenkenkenken8
   ■Work
   CTO of Nefrock Inc.
   

   View Slide

3. Copyright © NTT Communica+ons Corpora+on. All rights reserved. 3
   SkyWay = WebRTC plaUorm (CPaaS)
   

   View Slide

4. TL; DR
   •  Tes+ng WebRTC plaUorm/app is super hard
   WebRTCプラットフォーム/アプリのテストは超大変
   •  Let’s automate it! and we did it!
   そこで自動化したく、やってみた！
   

   View Slide

5. TL; DR
   •  Tes+ng WebRTC plaUorm/app is super hard
   WebRTCプラットフォーム/アプリのテストは超大変
   •  Let’s automate it! and we did it!
   そこで自動化したく、やってみた！
   

   View Slide

6. Why so tough?
   なぜ大変？
   

   View Slide

7. Reason 1 : Manual opera+on (手動)
   Video 1 Video2
   ① ② ③
   

   View Slide

8. Reason 1+α : Tes+ng signaling is not enough
   Video 1 Video2
   　　Signaling Succeeded
   　　Video s+ll black
   シグナリング単体じゃ不十分
   

   View Slide

9. Reason 2 : Various condi+ons (Clients)
   + iOS, Android, Na+ve …
   条件多すぎ（クライアントだけで)
   

   View Slide

10. Reason 2+α : Aggressive update of clients
    like, gUM on HTTP just doesn’t work
    アップデート早すぎ(クライアントだけで)
    gUMはHTTPで動かなくなったり
    6-8 weeks update!
    

    View Slide

11. Reason3 : Various condi+ons (Network)
    NAT-X
    ① Endpoint Independent Filtering X Endpoint Independent Mapping
    … over 80 cases (NATタイプだけで発着を考えると80種類超)
    条件多すぎ（ネットワークの種類も)
    

    View Slide

12. Reason3 : Various condi+ons (Network)
    NAT-X
    NAT-Y
    UDP blocked!
    ① Endpoint Independent Filtering X Endpoint Independent Mapping
    … over 80 cases (NATタイプだけで発着を考えると80種類超)
    条件多すぎ（ネットワークの種類も)
    

    View Slide

13. Reason3 : Various condi+ons (Network)
    NAT-X
    NAT-Y
    UDP blocked!
    ① Endpoint Independent Filtering X Endpoint Independent Mapping
    … over 80 cases (NATタイプだけで発着を考えると80種類超)
    条件多すぎ（ネットワークの種類も)
    Just tes+ng in the same machine isn’t enough
    

    View Slide

14. TL; DR
    •  Tes+ng WebRTC plaUorm/app is super hard
    WebRTCプラットフォーム/アプリのテストは超大変
    •  Let’s automate it! and we did it!
    そこで自動化したく、やってみた！
    

    View Slide

15. Ubuntu with Xvn
    Architecture Overview
    Ubuntu with Xvn
    NAT
    instance
    NAT
    instance
    Intranet1 Intranet2
    Web Server
    (hosts test
    app page)
    Virtual Internet
    VPC
    Network
    Segment
    TURN
    server
    

    View Slide

16. Browser1
    DataChannel Tes+ng
    Browser2
    

    View Slide

17. MediaChannel Tes+ng
    Browser1 Browser2
    video
    (gUM)
    video
    (gUM)
    

    View Slide

18. Browser x Browser x NAT type x NAT type
    ・Browsers
    ・ Chrome Stable, Chrome Beta, Chrome Unstable
    ・ Firefox Stable (, Firefox Unstable)
    ・NAT types
    ・Theore+cally 9 types (RFC4787)
    ・some types not configurable with iptables
    -> wrote it from scratch
    ・4 * 4 * 9 * 9 = 1296
    uh…
    

    View Slide

19. NAT types
    hup://www.netmanias.com/en/?m=view&id=techdocs&no=6065
    

    View Slide

20. Ubuntu with Xvn
    Architecture Overview
    Ubuntu with Xvn
    NAT
    instance
    NAT
    instance
    Intranet1 Intranet2
    Web Server
    (hosts test
    app page)
    Virtual Internet
    VPC
    Network
    Segment
    TURN
    server
    

    View Slide

21. Ubuntu
    with Xvn
    Architecture Overview (Detailed ver.)
    Intranet1 Intranet2
    Web Server
    (hosts test
    app page)
    Virtual Internet
    VPC
    Network
    Segment
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    16 clients
    NAT
    Type
    9
    NAT
    Type
    8
    NAT
    Type
    7
    NAT
    Type
    6
    NAT
    Type
    5
    NAT
    Type
    4
    NAT
    Type
    3
    NAT
    Type
    2
    NAT
    Type
    1
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    Ubuntu
    with Xvn
    16 clients
    NAT
    Type
    9
    NAT
    Type
    8
    NAT
    Type
    7
    NAT
    Type
    6
    NAT
    Type
    5
    NAT
    Type
    4
    NAT
    Type
    3
    NAT
    Type
    2
    NAT
    Type
    1
    TURN server
    

    View Slide

22. Test result and Packet visualiza+on
    see -> hup://status.skyway.io
    

    View Slide

23. Thanks!!
    

    View Slide