VideoMark Analysis

ICICE 合同研究会 (CQ 研究会) 201911 月の招待講演にて利用したスライド

Transcript

1. [Invited Talk] Collecting and analyzing video
   streaming services' quality data with Web browser.
   @ CQ 201911
   WebDINO Japan - Tomoya ASAI
   NTT Network Technology Labs - Takuto KIMURA
   

   View Slide

2. about:speaker
   Tomoya ASAI - dynamis webdino.org
   CTO
   dynamis @ Community
   Twitter: @dynamitter
   facebook.com/dynamis
   speakerdeck.com/dynamis
   note: red panda (fire fox, lesser panda)
   2
   @
   

   View Slide

3. about: WebDINO Japan
   Web, Browser, Open
   ( : )
   (Chromium, Firefox )
   ( : Gecko )
   Web & Open
   Web IoT
   : 2017 Mozilla Japan WebDINO Japan
   https://www.webdino.org/
   3
   

   View Slide

4. Topics
   Web VideoMark Project
   Measurement and Open Data
   Early Data Analysis
   Next Steps
   4
   

   View Slide

5. Acknowledgment
   Web VideoMark WebDINO Japan
   貢
   (QoE )
   NTT
   貢
   GitHub 貢
   https://vm.webdino.org/notice
   5
   

   View Slide

6. Web VideoMark Project
   

   View Slide

7. ಈըαʔϏεͷ
   ਪఆମײ඼࣭஋
   ར༻ऀͷ

   ϒϥ΢βʔ
   ֶज़ݚڀ༻σʔλ

   ΍ௐࠪ෼ੳ݁Ռ͸

   Φʔϓϯσʔλʹ
   ʲΫϥΠΞϯτ৘ใʳ
   w ୺຤छผ

   1$PSϞόΠϧ
   
   w ϒϥ΢βόʔδϣϯ
   w 04όʔδϣϯ౳
   ஫ࢯ໊ɾॅॴ౳ͷݸਓ
   ಛఆ৘ใ͸ؚ·ͳ͍

   6TFS"HFOU͔Βͷਪଌ
   ʲωοτϫʔΫ৘ใʳ
   w ௨৴ࣄۀࣾɾ*41
   w ஍Ҭ ౎ಓ෎ݝ
   ౳
   ஫ૹ৴ݩ*1ΞυϨεಗ
   ໊Խ ຤ඌΦΫςοτ
   ࡟আ
   ޙʹ(FP*1Ͱਪଌ
   ʲ࠶ੜ඼࣭ཁҼʳ
   w ಈը৘ใϏοτϨʔτɺ
   ղ૾౓ɺϑϨʔϜϨʔτ౳
   w ࠶ੜ৘ใ࠶ੜͷதஅɺ
   όοϑΝϦϯά౳
   w ૢ࡞৘ใಈըͷγʔΫɺ
   Ұ࣌ఀࢭɾ࠶։౳
   8FC7JEFP.BSL֦ுػೳ΍ϒϥ΢βʔͰࢹௌ৘ใΛऔಘ
   ಈը࠶ੜ࣌ͷ
   ମײ඼࣭஋Λ

   dͷ਺஋Ͱ

   ਪఆͯ͠ฦ͢
   2P&
   2VBMJUZPG&YQFSJFODF
   8FC
   7JEFP.BSL
   αʔόʔ
   ࢹௌ৘ใΛ෼ੳ
   ಈը഑৴αʔϏε΁ΞΫηε
   )5.-ɺಈըϑΝΠϧ౳Λड৴
   ௨৴ࣄۀऀɾ*41
   αʔϏεվળ༻ʹ
   ௐࠪσʔλΛఏڙ
   ʂ
   ಈըࢹௌ࣌ͷ
   2P&౳Λ֬ೝ
   ϓϥΠόγʔอޢॲཧ ֶज़ݚڀ༻σʔλ࡞੒
   ಗ໊Խॲཧ
   ಈը഑৴αʔϏεɾ
   $%/αʔόʔ
   Web VideoMark
   

   View Slide

8. Web VideoMark
   :
   8
   GCP
   Internet
   sodium.js
   nginx/ uent-d
   

   

   sodium.js 

   PC Chrome

   +
   

   (Android)
   

   endpoint
   log file
   SodiumServer
   AggregateServer
   QoE 

   

   QoE 

   GeoIP 

   (IP )
   QoE Server
   QoE
   

   mongoDB
   mongoDB
   

   QoE 

   

   QoE
   

   View Slide

9. Web VideoMark
   Paravi, TVer, YouTube
   CQ IOT
   https://vm.webdino.org/opendata
   Web VideoMark Project by WebDINO Japan http://vm.webdino.org/
   9
   

   View Slide

10. (58%, sandvine ) ( )
    (Chrome) (Web VideoMark) Android
    (VideoMark Browser)
    ( )
    10
    

    View Slide

11. CQ
    CQ 

    https://www.webdino.org/updates/news/201902051051/
    11
    

    View Slide

12. Web VideoMark
    ( ) 

    ( )
    : 2019 10
    12
    

    View Slide

13. ( )
    ( ) QoE
    15
    QoE ⭐
    13
    

    View Slide

14. View Slide

15. ( MacBook )
    Paravi, TVer QoE (3.2) YouTube 3
    YouTube QoE 4,9, 3.5-3.6, 3.2 , Paravi TVer QoE 4.5
    :
    15
    

    View Slide

16. Web VideoMark
    QoS QoE
    16
    

    View Slide

17. QoE
    ISP APN DL
    CM
    YouTube
    ( )
    17
    

    View Slide

18. note: (QoE)
    (QoE)
    (QoE: Quality of Experience)
    (1 5, MOS)
    QoE
    QoE (ITU)
    Web VideoMark ITU
    (NTT
    )
    18
    

    View Slide

19. : API ( QoE)
    PC VM URL "stats"
    19
    

    View Slide

20. ( )
    PC/Android Web VideoMark
    https://vm.webdino.org/
    ( )
    (Jetson Nano)
    LAN OK
    : https://vm.webdino.org/contact
    20
    

    View Slide

21. https://vm.webdino.org
    21
    

    View Slide

22. Measurement

    & Open Data
    

    View Slide

23. https://vm.webdino.org/opendata
    IP URL
    Android UserAgent
    UA
    (a,b,c)
    https://vm.webdino.org/opendata
    23
    

    View Slide

24. IP
    URL
    User Agent
    User Agent Android
    ID ( 1 )
    24
    

    View Slide

25. IP ( )
    , , ISP (GeoIP )
    APN, PLMN (Android )
    URL
    CDN IP [*]
    URL [*]
    [*]
    ( )
    etc...
    Representation [**]
    Codec
    Representation
    /
    etc...
    QoE ( )
    (RTT) [*]
    [*]
    etc...
    [*]
    etc...
    CM [*]
    etc...
    [*] [**]
    25
    

    View Slide

26. SIM ID
    GDPR
    IP
    GPS
    User Agent
    URL
    26
    

    View Slide

27. ( )
    2019-04-07T06:38:58+00:00 {"date":"2019-04-07T06:38:58.253Z","startTime":
    297957833.29999995,"endTime":297962836.1000001,"session":"8b37d168-7343-45a1-a507-
    f8c230a0ebd4","location":"https://www.youtube.com","userAgent":"Mozilla/5.0 (Linux;
    Android 9; XXX) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.124 Mobile
    Safari/537.36","appVersion":"5.0 (Linux; Android 9; XXX) AppleWebKit/537.36 (KHTML,
    like Gecko) Chrome/70.0.3538.124 Mobile Safari/537.36","sequence":58463,"video":
    [{"property":{"uuid":"fd65101f-a8df-47c6-a804-3ac719e04ca3","id":"","class":
    ["video-stream","html5-main-video"],"src":"blob:https://www.youtube.com/a1551960-
    f053-492e-b945-84b075c8598f","domainName":"r7---sn-5n5ip-
    ioqe.googlevideo.com","width":0,"height":0,"videoWidth":1280,"videoHeight":
    720,"mediaSize":82.384398,"defaultPlaybackRate":1,"playbackRate":1,"playStartTime":
    1554511611888.0,"playEndTime":1554511694180.0,"currentPlayPos":
    82.384398,"currentPlayTime":82.245},"playback_quality":[],"event_play":
    [],"event_play_delta":[],"event_pause":[],"event_pause_delta":[],"event_seeking":
    [],"event_seeking_delta":[],"event_seeked":[],"event_seeked_delta":
    [],"event_ended":[],"event_ended_delta":[],"event_stalled":
    [],"event_stalled_delta":[],"event_progress":[],"event_progress_delta":
    [],"event_waiting":[],"event_waiting_delta":[],"event_canplay":
    [],"event_canplay_delta":[]}],"resource_timing":
    [],"REMOTE_ADDR":"180.235.59.0","country":"JP","subdivision":"13","isp":"Asahi
    Net"}
    5
    27
    

    View Slide

28. ( , mongo DB )
    session:
    session id: uuid
    video id: uuid
    service: youtube
    user agent: XXXXXXX
    qoe: 4.2
    connection:
    - type: cellular
    effectiveType: 4g
    downlink: 1.55
    downlinkMax: 100
    rtt: 150
    apn: NTT DOCOMO
    plmn: NTT DOCOMO
    network: # IPͱGeoIP৘ใ
    - server host:
    xxx.yyy.com
    server ip: x.y.z.0
    client ip: x.y.z.0
    location: tokyo
    isp: ocn
    video:
    video id: xxxxxx
    duration: 160.3
    representations:
    - resolution:
    height: 1080
    width: 1920
    video codec: webm
    audio codec: opus
    video bitrate: 12345
    audio bitrate: 123
    play history:
    start time: 1234567890
    end time: 1234567890
    throughput history:
    # DL࣌ࠁͱ଎౓ɾRTT
    - dl time: 1234567890
    throughput: 123456
    rtt: 1234
    event history:
    - event type: seek
    time: 12.3
    representation
    history:
    # ৄࡉׂѪ
    frame drop history:
    # ৄࡉׂѪ
    buffer history:
    # ৄࡉׂѪ
    cm history:
    - duration: 12:3
    start time: 1234567890
    end time: 1234567890
    yml
    28
    

    View Slide

29. https://vm.webdino.org
    29
    

    View Slide

30. Initial Data Analysis
    

    View Slide

31. 3/1 4/30 (2 )
    23645
    10 12K
    1000 ISP
    ISP
    :
    31
    

    View Slide

32. QoE
    ISP QoE
    32
    

    View Slide

33. ( )
    PC/Mobile
    ISP
    33
    

    View Slide

34. View Slide

35. ( )
    5
    3 9
    ( - ) / 1.0
    1.0 CM
    35
    

    View Slide

36. ( )
    X 0.93-1.0 CM (CM )
    Y, Z 1.0 ( )
    Y 0-0.03 CM
    36
    Y
    X Z
    Y
    

    View Slide

37. 1.0
    ( - ) /
    CM 1.0
    Y CM 1.0-1.15
    Z CM 1.0-1.15
    Y, Z 1.5
    37
    X Z
    Y
    

    View Slide

38. PC
    PC 0.0
    CM
    38
    PC Mobile
    

    View Slide

39. QoE
    ( ) (QoE) 

    ( )
    CM
    39
    

    View Slide

40. /
    20 ( 130 )
    5, 10 5 5
    5,10,15... 5,10,15... ( )
    15 /
    40
    

    View Slide

41. QoE
    

    View Slide

42. ( )
    (bps) ( )
    貢
    ( )
    42
    

    View Slide

43. ( )
    QoE
    QoE
    QoE
    43
    

    View Slide

44. B ( )
    C ( ) 

    ( )
    44
    A
    B
    C
    A
    B
    C
    

    View Slide

45. B ( )
    C ( ) QoE
    45
    A
    B
    C
    

    View Slide

46. ISP
    OCN ( ) Asahi Net ( )
    Asahi Net ( ) WiFi OCN ( )
    ( )
    46
    ISP X
    ISP Y
    ISP Z
    ISP X
    ISP Y
    ISP Z
    

    View Slide

47. ISP
    ARTERIA ( ), Asahi Net ( )
    PC vs Mobile QoE ( ) ISP
    ( )
    47
    ISP X
    ISP Y
    ISP Z
    

    View Slide

48. QoE ( , )
    QoE 1.0 5.0 30
    貢 1080p, 720p
    ABR
    48
    A
    720p30 1.5~2Mbsp
    480p30 1M
    1080p 2~3Mbps
    1080p 5Mbps
    

    View Slide

49. QoE
    A PC
    PC QoE 5 1080p30 ( )
    PC 3.5 QoE PC
    49
    A (PC) A (mobile)
    

    View Slide

50. QoE
    B, C QoE 4.6-4.73 720p
    ( )
    B QoE 4.73 1080p
    ( )
    50
    B C
    

    View Slide

51. ISP QoE
    OCN PC
    4K, 8K, 60fps
    ARTERIA 

    51
    OCN ARTERIA
    

    View Slide

52. ISP QoE
    ISP X QoE
    Asahi Net Android
    52
    Asahi Net
    ISP X
    

    View Slide

53. ISP QoE
    ISP Y QoE 1 , Z QoE
    53
    ISP Z
    ISP Y
    

    View Slide

54. View Slide

55. ( , )
    2060p30, 1080p30, 720p30, 480p30
    ABR
    (ABR )
    55
    A
    

    View Slide

56. A PC
    PC 1080p30 FSP
    PC 1080p30
    56
    A (PC) A (mobile)
    

    View Slide

57. B, C B
    B 720p 2Mbps C 1080p 5Mbps
    57
    B C
    

    View Slide

58. ISP
    OCN PC
    貢
    58
    OCN ARTERIA
    

    View Slide

59. ISP
    Asahi Net Android
    59
    Asahi Net
    ISP X
    

    View Slide

60. ISP
    ( )
    60
    ISP Z
    ISP Y
    

    View Slide

61. View Slide

62. ( , )
    62
    A
    

    View Slide

63. A PC
    PC PC
    63
    A (PC) A (mobile)
    

    View Slide

64. B
    C
    64
    B C
    

    View Slide

65. ISP
    ISP
    ( )
    65
    OCN ARTERIA
    

    View Slide

66. ISP
    66
    Asahi Net
    ISP X
    

    View Slide

67. ISP
    67
    ISP Z
    ISP Y
    

    View Slide

68. QoE
    (
    )
    QoE
    QoE
    ( )
    68
    

    View Slide

69. ISP ( )
    PC/Android
    : OSS
    69
    

    View Slide

70. OSS
    YouTube
    (
    )
    70
    

    View Slide

71. QoE
    QoE
    QoE
    QoE
    : 1080p, 30fps, audio 128kbps ( )

    video 3380kbps => 4.813, 5000kbps => 4.882
    QoE (
    )
    :
    QoE
    71
    

    

    View Slide

72. Web VideoMark
    ( )
    CM (
    CM CM )
    ISP
    QoE
    ISP
    72
    

    View Slide

73. Next Steps
    

    View Slide

74. : APN PLMN etc.

    : RTT etc.
    CM
    : CM etc.
    CM
    QoE CM (
    )
    74
    

    View Slide

75. 3 10
    YouTube +
    QoE
    75
    

    View Slide

76. ( )
    較
    Chrome
    76
    

    View Slide

77. mongo DB
    ( )
    77
    

    View Slide

78. etc...
    : https://vm.webdino.org/contact
    78
    

    View Slide

79. ( )
    ( )
    PC/Android Web VideoMark
    https://vm.webdino.org/
    ( )
    (Jetson Nano)
    LAN OK
    : https://vm.webdino.org/contact
    79
    

    View Slide

80. https://vm.webdino.org
    80
    

    View Slide

81. vol.119 IEICE
    (IEICE )
    © 2019 IEICE , vol.119, no. 298, CQ-2019-92, Nov. 2019
    (
    )
    ®
    貢
    81
    

    View Slide