Tweet
Tweet

Slide 1

Slide 1 text

[Invited Talk] Collecting and analyzing video streaming services' quality data with Web browser. @ CQ 201911 WebDINO Japan - Tomoya ASAI NTT Network Technology Labs - Takuto KIMURA

Slide 2

Slide 2 text

about:speaker Tomoya ASAI - dynamis webdino.org CTO dynamis @ Community Twitter: @dynamitter facebook.com/dynamis speakerdeck.com/dynamis note: red panda (fire fox, lesser panda) 2 @

Slide 3

Slide 3 text

about: WebDINO Japan Web, Browser, Open ( : ) (Chromium, Firefox ) ( : Gecko ) Web & Open Web IoT : 2017 Mozilla Japan WebDINO Japan https://www.webdino.org/ 3

Slide 4

Slide 4 text

Topics Web VideoMark Project Measurement and Open Data Early Data Analysis Next Steps 4

Slide 5

Slide 5 text

Acknowledgment Web VideoMark WebDINO Japan 貢 (QoE ) NTT 貢 GitHub 貢 https://vm.webdino.org/notice 5

Slide 6

Slide 6 text

Web VideoMark Project

Slide 7

Slide 7 text

ಈըαʔϏεͷ ਪఆମײ඼࣭஋ ར༻ऀͷ
 ϒϥ΢βʔ ֶज़ݚڀ༻σʔλ
 ΍ௐࠪ෼ੳ݁Ռ͸
 Φʔϓϯσʔλʹ ʲΫϥΠΞϯτ৘ใʳ w ୺຤छผ
 1$PSϞόΠϧ w ϒϥ΢βόʔδϣϯ w 04όʔδϣϯ౳ ஫ࢯ໊ɾॅॴ౳ͷݸਓ ಛఆ৘ใ͸ؚ·ͳ͍
 6TFS"HFOU͔Βͷਪଌ ʲωοτϫʔΫ৘ใʳ w ௨৴ࣄۀࣾɾ*41 w ஍Ҭ ౎ಓ෎ݝ ౳ ஫ૹ৴ݩ*1ΞυϨεಗ ໊Խ ຤ඌΦΫςοτ ࡟আ ޙʹ(FP*1Ͱਪଌ ʲ࠶ੜ඼࣭ཁҼʳ w ಈը৘ใϏοτϨʔτɺ ղ૾౓ɺϑϨʔϜϨʔτ౳ w ࠶ੜ৘ใ࠶ੜͷதஅɺ όοϑΝϦϯά౳ w ૢ࡞৘ใಈըͷγʔΫɺ Ұ࣌ఀࢭɾ࠶։౳ 8FC7JEFP.BSL֦ுػೳ΍ϒϥ΢βʔͰࢹௌ৘ใΛऔಘ ಈը࠶ੜ࣌ͷ ମײ඼࣭஋Λ
 dͷ਺஋Ͱ
 ਪఆͯ͠ฦ͢ 2P& 2VBMJUZPG&YQFSJFODF 8FC 7JEFP.BSL αʔόʔ ࢹௌ৘ใΛ෼ੳ ಈը഑৴αʔϏε΁ΞΫηε )5.-ɺಈըϑΝΠϧ౳Λड৴ ௨৴ࣄۀऀɾ*41 αʔϏεվળ༻ʹ ௐࠪσʔλΛఏڙ ʂ ಈըࢹௌ࣌ͷ 2P&౳Λ֬ೝ ϓϥΠόγʔอޢॲཧ ֶज़ݚڀ༻σʔλ࡞੒ ಗ໊Խॲཧ ಈը഑৴αʔϏεɾ $%/αʔόʔ Web VideoMark

Slide 8

Slide 8 text

Web VideoMark : 8 GCP Internet sodium.js nginx/ uent-d 
 
 sodium.js 
 PC Chrome
 + 
 (Android) 
 endpoint log file SodiumServer AggregateServer QoE 
 
 QoE 
 GeoIP 
 (IP ) QoE Server QoE 
 mongoDB mongoDB 
 QoE 
 
 QoE

Slide 9

Slide 9 text

Web VideoMark Paravi, TVer, YouTube CQ IOT https://vm.webdino.org/opendata Web VideoMark Project by WebDINO Japan http://vm.webdino.org/ 9

Slide 10

Slide 10 text

(58%, sandvine ) ( ) (Chrome) (Web VideoMark) Android (VideoMark Browser) ( ) 10

Slide 11

Slide 11 text

CQ CQ 
 https://www.webdino.org/updates/news/201902051051/ 11

Slide 12

Slide 12 text

Web VideoMark ( ) 
 ( ) : 2019 10 12

Slide 13

Slide 13 text

( ) ( ) QoE 15 QoE ⭐ 13

Slide 14

Slide 14 text

14

Slide 15

Slide 15 text

( MacBook ) Paravi, TVer QoE (3.2) YouTube 3 YouTube QoE 4,9, 3.5-3.6, 3.2 , Paravi TVer QoE 4.5 : 15

Slide 16

Slide 16 text

Web VideoMark QoS QoE 16

Slide 17

Slide 17 text

QoE ISP APN DL CM YouTube ( ) 17

Slide 18

Slide 18 text

note: (QoE) (QoE) (QoE: Quality of Experience) (1 5, MOS) QoE QoE (ITU) Web VideoMark ITU (NTT ) 18

Slide 19

Slide 19 text

: API ( QoE) PC VM URL "stats" 19

Slide 20

Slide 20 text

( ) PC/Android Web VideoMark https://vm.webdino.org/ ( ) (Jetson Nano) LAN OK : https://vm.webdino.org/contact 20

Slide 21

Slide 21 text

https://vm.webdino.org 21

Slide 22

Slide 22 text

Measurement
 & Open Data

Slide 23

Slide 23 text

https://vm.webdino.org/opendata IP URL Android UserAgent UA (a,b,c) https://vm.webdino.org/opendata 23

Slide 24

Slide 24 text

IP URL User Agent User Agent Android ID ( 1 ) 24

Slide 25

Slide 25 text

IP ( ) , , ISP (GeoIP ) APN, PLMN (Android ) URL CDN IP [*] URL [*] [*] ( ) etc... Representation [**] Codec Representation / etc... QoE ( ) (RTT) [*] [*] etc... [*] etc... CM [*] etc... [*] [**] 25

Slide 26

Slide 26 text

SIM ID GDPR IP GPS User Agent URL 26

Slide 27

Slide 27 text

( ) 2019-04-07T06:38:58+00:00 {"date":"2019-04-07T06:38:58.253Z","startTime": 297957833.29999995,"endTime":297962836.1000001,"session":"8b37d168-7343-45a1-a507- f8c230a0ebd4","location":"https://www.youtube.com","userAgent":"Mozilla/5.0 (Linux; Android 9; XXX) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.124 Mobile Safari/537.36","appVersion":"5.0 (Linux; Android 9; XXX) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.124 Mobile Safari/537.36","sequence":58463,"video": [{"property":{"uuid":"fd65101f-a8df-47c6-a804-3ac719e04ca3","id":"","class": ["video-stream","html5-main-video"],"src":"blob:https://www.youtube.com/a1551960- f053-492e-b945-84b075c8598f","domainName":"r7---sn-5n5ip- ioqe.googlevideo.com","width":0,"height":0,"videoWidth":1280,"videoHeight": 720,"mediaSize":82.384398,"defaultPlaybackRate":1,"playbackRate":1,"playStartTime": 1554511611888.0,"playEndTime":1554511694180.0,"currentPlayPos": 82.384398,"currentPlayTime":82.245},"playback_quality":[],"event_play": [],"event_play_delta":[],"event_pause":[],"event_pause_delta":[],"event_seeking": [],"event_seeking_delta":[],"event_seeked":[],"event_seeked_delta": [],"event_ended":[],"event_ended_delta":[],"event_stalled": [],"event_stalled_delta":[],"event_progress":[],"event_progress_delta": [],"event_waiting":[],"event_waiting_delta":[],"event_canplay": [],"event_canplay_delta":[]}],"resource_timing": [],"REMOTE_ADDR":"180.235.59.0","country":"JP","subdivision":"13","isp":"Asahi Net"} 5 27

Slide 28

Slide 28 text

( , mongo DB ) session: session id: uuid video id: uuid service: youtube user agent: XXXXXXX qoe: 4.2 connection: - type: cellular effectiveType: 4g downlink: 1.55 downlinkMax: 100 rtt: 150 apn: NTT DOCOMO plmn: NTT DOCOMO network: # IPͱGeoIP৘ใ - server host: xxx.yyy.com server ip: x.y.z.0 client ip: x.y.z.0 location: tokyo isp: ocn video: video id: xxxxxx duration: 160.3 representations: - resolution: height: 1080 width: 1920 video codec: webm audio codec: opus video bitrate: 12345 audio bitrate: 123 play history: start time: 1234567890 end time: 1234567890 throughput history: # DL࣌ࠁͱ଎౓ɾRTT - dl time: 1234567890 throughput: 123456 rtt: 1234 event history: - event type: seek time: 12.3 representation history: # ৄࡉׂѪ frame drop history: # ৄࡉׂѪ buffer history: # ৄࡉׂѪ cm history: - duration: 12:3 start time: 1234567890 end time: 1234567890 yml 28

Slide 29

Slide 29 text

https://vm.webdino.org 29

Slide 30

Slide 30 text

Initial Data Analysis

Slide 31

Slide 31 text

3/1 4/30 (2 ) 23645 10 12K 1000 ISP ISP : 31

Slide 32

Slide 32 text

QoE ISP QoE 32

Slide 33

Slide 33 text

( ) PC/Mobile ISP 33

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

( ) 5 3 9 ( - ) / 1.0 1.0 CM 35

Slide 36

Slide 36 text

( ) X 0.93-1.0 CM (CM ) Y, Z 1.0 ( ) Y 0-0.03 CM 36 Y X Z Y

Slide 37

Slide 37 text

1.0 ( - ) / CM 1.0 Y CM 1.0-1.15 Z CM 1.0-1.15 Y, Z 1.5 37 X Z Y

Slide 38

Slide 38 text

PC PC 0.0 CM 38 PC Mobile

Slide 39

Slide 39 text

QoE ( ) (QoE) 
 ( ) CM 39

Slide 40

Slide 40 text

/ 20 ( 130 ) 5, 10 5 5 5,10,15... 5,10,15... ( ) 15 / 40

Slide 41

Slide 41 text

QoE

Slide 42

Slide 42 text

( ) (bps) ( ) 貢 ( ) 42

Slide 43

Slide 43 text

( ) QoE QoE QoE 43

Slide 44

Slide 44 text

B ( ) C ( ) 
 ( ) 44 A B C A B C

Slide 45

Slide 45 text

B ( ) C ( ) QoE 45 A B C

Slide 46

Slide 46 text

ISP OCN ( ) Asahi Net ( ) Asahi Net ( ) WiFi OCN ( ) ( ) 46 ISP X ISP Y ISP Z ISP X ISP Y ISP Z

Slide 47

Slide 47 text

ISP ARTERIA ( ), Asahi Net ( ) PC vs Mobile QoE ( ) ISP ( ) 47 ISP X ISP Y ISP Z

Slide 48

Slide 48 text

QoE ( , ) QoE 1.0 5.0 30 貢 1080p, 720p ABR 48 A 720p30 1.5~2Mbsp 480p30 1M 1080p 2~3Mbps 1080p 5Mbps

Slide 49

Slide 49 text

QoE A PC PC QoE 5 1080p30 ( ) PC 3.5 QoE PC 49 A (PC) A (mobile)

Slide 50

Slide 50 text

QoE B, C QoE 4.6-4.73 720p ( ) B QoE 4.73 1080p ( ) 50 B C

Slide 51

Slide 51 text

ISP QoE OCN PC 4K, 8K, 60fps ARTERIA 
 51 OCN ARTERIA

Slide 52

Slide 52 text

ISP QoE ISP X QoE Asahi Net Android 52 Asahi Net ISP X

Slide 53

Slide 53 text

ISP QoE ISP Y QoE 1 , Z QoE 53 ISP Z ISP Y

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

( , ) 2060p30, 1080p30, 720p30, 480p30 ABR (ABR ) 55 A

Slide 56

Slide 56 text

A PC PC 1080p30 FSP PC 1080p30 56 A (PC) A (mobile)

Slide 57

Slide 57 text

B, C B B 720p 2Mbps C 1080p 5Mbps 57 B C

Slide 58

Slide 58 text

ISP OCN PC 貢 58 OCN ARTERIA

Slide 59

Slide 59 text

ISP Asahi Net Android 59 Asahi Net ISP X

Slide 60

Slide 60 text

ISP ( ) 60 ISP Z ISP Y

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

( , ) 62 A

Slide 63

Slide 63 text

A PC PC PC 63 A (PC) A (mobile)

Slide 64

Slide 64 text

B C 64 B C

Slide 65

Slide 65 text

ISP ISP ( ) 65 OCN ARTERIA

Slide 66

Slide 66 text

ISP 66 Asahi Net ISP X

Slide 67

Slide 67 text

ISP 67 ISP Z ISP Y

Slide 68

Slide 68 text

QoE ( ) QoE QoE ( ) 68

Slide 69

Slide 69 text

ISP ( ) PC/Android : OSS 69

Slide 70

Slide 70 text

OSS YouTube ( ) 70

Slide 71

Slide 71 text

QoE QoE QoE QoE : 1080p, 30fps, audio 128kbps ( )
 video 3380kbps => 4.813, 5000kbps => 4.882 QoE ( ) : QoE 71 


Slide 72

Slide 72 text

Web VideoMark ( ) CM ( CM CM ) ISP QoE ISP 72

Slide 73

Slide 73 text

Next Steps

Slide 74

Slide 74 text

: APN PLMN etc.
 : RTT etc. CM : CM etc. CM QoE CM ( ) 74

Slide 75

Slide 75 text

3 10 YouTube + QoE 75

Slide 76

Slide 76 text

( ) 較 Chrome 76

Slide 77

Slide 77 text

mongo DB ( ) 77

Slide 78

Slide 78 text

etc... : https://vm.webdino.org/contact 78

Slide 79

Slide 79 text

( ) ( ) PC/Android Web VideoMark https://vm.webdino.org/ ( ) (Jetson Nano) LAN OK : https://vm.webdino.org/contact 79

Slide 80

Slide 80 text

https://vm.webdino.org 80

Slide 81

Slide 81 text

vol.119 IEICE (IEICE ) © 2019 IEICE , vol.119, no. 298, CQ-2019-92, Nov. 2019 ( ) ® 貢 81