AV1 Patent Issue

WebRTC Meetup Tokyo #21 で使用したスライド (改)
https://atnd.org/events/105581

更新履歴:
・AV1 の商用リアルタイム 4K ハードウェアエンコーダ IP 事例を追記
・ロイヤリティーフリーという用語を正しく使うよう修正
・AoM Patent License の防衛的契約解除の影響範囲に関して修正・大幅追記
・動画配信サービスの体感品質評価に関するプロジェクト情報を拡充

Transcript

1. AV1 Patent issue @WebRTC Meetup Tokyo #21 on 2019.05.28 Tomoya ASAI (CTO, WebDINO Japan)
2. about:me Tomoya ASAI - dynamis webdino.org CTO dynamis @ Community Twitter: @dynamitter facebook.com/dynamis speakerdeck.com/dynamis I LOVE Firefox = Red Panda = Petit Panda = Lesser Panda 2 @
3. about: WebDINO Japan Web, Browser, Open ( : ) (Chromium, Firefox ) ( : Gecko ) Web & Open Web IoT 2017 : Mozilla Japan WebDINO Japan https://www.webdino.org/ 3
4. TL;DR AV1 H.265/HEVC 縄 HW PoC Chrome/Firefox YouTube AV1 AoM or AV1 AV1 2019.03 SISVEL 縄 4
5. Topics AV1 & Patent History Current Implementations Quality of Video Patent Pool by SISVEL AoM Patent License Next... 5
6. disclimer AoM SISVEL Mozilla Firefox CTO 6
7. AV1 & Patent History
8. Google VPx 2009.08 Google On2 VP7 VP9 OSS 2011.02 MPEG/LA VP8 2013.03 Google MPEG/LA VP8 Google VP8 11 VP8,9 AV1 2013.03 Nokia VP8 86 HTC ( VP8 ) IETF Nokia AoM Microsoft 8
9. : MPEG/LA VP8 CIF Licensing LLC France Telecom ( Orange) Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V. Fujitsu Limited Koninklijke Philips Electronics N.V. LG Electronics Inc. Mitsubishi Electric Corporation NTT DOCOMO, INC.1 Panasonic Corporation Samsung Electronics Co., Ltd. Siemens Corporation2 11 https://www.webmproject.org/cross-license/vp8/primary-licensors/ 9
10. : H.264 (MPEG/LA) Apple Inc CableTelevision Laboratories, Inc.* Cisco Systems Canada Co. Cisco Technology, Inc. Dolby International AB Dolby Laboratories Licensing Corporation Electronics and Telecommunications Research Institute Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V. Fujitsu Limited GE Video Compression, LLC Godo Kaisha IP Bridge1 Google LLC HP Inc.* JVC KENWOOD Corporation Koninklijke Philips N.V. Korea Advanced Institute of Science and Technology (KAIST) LG Electronics Inc. Maxell, Ltd. Microsoft Corporation Mitsubishi Electric Corporation NEC Corporation NEWRACOM, Inc.* NTT DOCOMO, INC. Nippon Telegraph and Telephone Corporation Orange SA* Panasonic Corporation Polycom, Inc. Robert Bosch GmbH Samsung Electronics Co., Ltd. Sharp Corporation Siemens AG Sony Corporation Tagivan II LLC Telefonaktiebolaget LM Ericsson The Trustees of Columbia University in the City of New York* Toshiba Corporation Vidyo, Inc. ZTE Corporation* *Up to and through date of last patent expiration https://www.mpegla.com/programs/avc-h-264/licensors/ 10
11. : H.265 (MPEG/LA) Alpha Digitech, Inc. Apple Inc. British Broadcasting Corporation (BBC) Canon Inc. Digital Insights Inc. Electronics and Telecommunicatio ns Research Institute (ETRI) Fujitsu Limited Funai Electric Co., Ltd. Hangzhou Hikvision Digital Technology Co., Ltd. HUMAX Co., Ltd. IBEX PT Holdings IDEAHUB, Inc. Industry Academy Cooperation Foundation of Sejong University Infobridge Pte. Ltd. Intellectual Discovery Co., LTD. JVC KENWOOD Corporation Korea Advanced Institute of Science and Technology (KAIST) Korean Broadcasting System (KBS) KPIC Ltd. KT Corp. Kwangwoon University Industry Academic Collaboration Foundation M&K Holdings Inc. Massachusetts Institute of Technology (MIT) Maxell, Ltd. NEC Corporation Nippon Hoso Kyokai (NHK) Nippon Telegraph and Telephone Corporation (NTT) NTT DOCOMO, INC. Orange SA Samsung Electronics Co., Ltd. Siemens Corp. SK Planet Co., Ltd. SK Telecom Co., Ltd. Sky Media Tech, Inc. SungKyunKwan University Research & Business Foundation Tagivan II LLC The Trustees of Columbia University in the City of New York* University Industry Cooperation Foundation of Korea Aerospace University University Industry Cooperation Group of Kyung Hee University Vidyo, Inc. *Up to and through date of last patent expiration https://www.mpegla.com/programs/hevc/licensors/ 11
12. : H.265 (HEVC Advance) Canon Inc. Dolby Laboratories Licensing Corporation A liate of Dolby Laboratories, Inc. Dolby International AB A liate of Dolby Laboratories, Inc. Electronics and Telecommunications Research Institute, including patents co- owned with/among: Industry-Academy Cooperation Foundation of Sejong University Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung GE Video Compression, LLC A liate of General Electric Company Godo Kaisha IP Bridge 1 HFI Innovation Inc. A liate of MediaTek, Inc HUMAX Co., Ltd. IDEAHUB Inc. Koninklijke Philips N.V. Korea Advanced Institute of Science & Technology Korean Broadcasting System Kwangwoon University, Industry-Academic Collaboration Foundation Mitsubishi Electric Corporation Nippon Telegraph and Telephone Corporation NTT DOCOMO, INC. Samsung Electronics Co., Ltd. University-Industry Cooperation Foundation of Korea Aerospace University University-Industry Cooperation Group of Kyung Hee University Warner Bros. Entertainment, Inc. https://www.hevcadvance.com/pdfnew/LicensorList.pdf 12
13. : H.265 ( ) InterDigital HEVC technicolor InterDigital Velos Media Ericsson Panasonic Qualcomm Sharp Sony Blackberry MPEG/LA AT&T Broadcom Cisco Canon Disney Fraunhofer Huawei Intel KDDI Microsoft NICT Nokia https://qiita.com/yohhoy/items/c2579097a507b1fbdddb 13
14. : H.265 (2018.01 ) http://blog.chiariglione.org/a-crisis-the-causes-and-a-solution/ 14
15. Alliance for Open Media AV1 2015.09 AoM (Alliance for Open Media) Amazon, Cisco, Google, Intel, Microsoft, Mozilla, Net ix Daala, Thor, VPx AV1 2018.03 AV1 1.0 2018.09 YouTube AV1 Beta AV1 AV1 (CM AV1 ) TestTube AV1 15
16. AoM (2019.04 ) IT https://people.xiph.org/~negge/NAB2019.pdf 16 AOM Members - Today (42) 51
17. AV1 https://aomedia.org/ 17 AoM YouTube AV1+Opus
18. Implementations
19. libaom - AoM Dav1d - 2018.06 AoM 2018.07 VideoLAN 2018.11 Firefox Nightly 2019.03 Chrome Canary 2019.04 dav1d libaom 2 5 
 Firefox Beta 11.8% AV1 https://hacks.mozilla.org/2019/05/firefox-brings-you-smooth-video-playback-with-the-worlds-fastest-av1-decoder/ 19
20. dav1d iPad Pro 4K https://people.xiph.org/~negge/NAB2019.pdf 20 dav1d-0.2.1 Release (March 13, 2019) 22 1080p decode speeds in FPS (*) If you are Apple dav1d Can Decode 4k on Mobile(*) 23
21. 2017.11 Firefox https://hacks.mozilla.org/2017/11/dash-playback-of-av1- video/ Firefox, Chrome dav1d https://hacks.mozilla.org/2019/05/ refox-brings-you-smooth- video-playback-with-the-worlds-fastest-av1-decoder/ https://chromium.googlesource.com/chromium/src.git/+/ ede4345734371ac35b4e50cea7ef3332dabd54d5%5E%21/ Edge AV1 Video Extension (Beta) https://www.microsoft.com/en-us/p/av1-video-extension- beta/9mvzqvxjbq9v?activetab=pivot:overviewtab 21
22. libaom AoM x265 3 rav1e Mozilla Xiph Rust SVT-AV1 Intel Net ix Intel SVT Skylake 縄 libaom PSNR 14% 22 2018/07 libaom 2019/03 libaom
23. SVT-AV1 (fps, FullHD, 2019.03) https://www.phoronix.com/scan.php?page=news_item&px=27-CPUs-AV1-SVT-Video-Encode 23 SVT-AV1
 8.67 fps libaom 0.28 fps SVT-HEVC 288 fps
24. AV1 x264 4-5 24
25. AV1 H.265 https://beta.arewecompressedyet.com/ 25 A: x264, B: x265, C: libvpx, D: SVT-AV1
26. SOCIONEXT 2018.06 Amazon EC2 F1 FPGA NGCodec 2019.04 AV1 (4K30p) FPGA PoC NAV Allegro DVT 2019.04 4K IP AoM Intel, NVIDIA, ARM, AMD, Broadcom etc... 26
27. ( ) Two Orioles - EVE for AV1 ATEME Harmonic Bitmovin (cloud encoder) Visionular - Aurora : FPGA (720p30) https://dwango.github.io/articles/av1hwencoder/ 27
28. Quality of Video
29. : PSNR, SSIM PSNR: 
 SSIM: : VMAF (Net ix ) https://github.com/Net ix/vmaf https://medium.com/net ix-techblog/toward-a- practical-perceptual-video-quality-metric- 653f208b9652 29
30. QoS: Quality of Service ( ) etc. QoE (Quality of Experience) (MOS ) QoE ITU (ITU-T P.1201.1 ) 30
31. ISP/MVO/MVNO 31
32. Web VideoMark http://vm.webdino.org/ Paravi, TVer, YouTube ( IOT CQ ) 32 https://urls.jp/vm
33. ֶज़ݚڀ༻σʔλ
 ΍ௐࠪ෼ੳ݁ՌΛ
 Ұൠެ։ ૹ৴ ʲΫϥΠΞϯτ৘ใʳ w ୺຤छผ
 1$
    PS
    ϞόΠϧ
    w ϒϥ΢βόʔδϣϯ
    w 04
    όʔδϣϯ
    ౳
    ஫
    ࢯ໊ɾॅॴ౳ͷݸਓ ಛఆ৘ใ͸ؚ·ͳ͍
 6TFS"HFOU
    ͔Β൑அ ʲωοτϫʔΫ৘ใʳ w ௨৴ࣄۀࣾɾ*41
    w ஍Ҭ
    ౎ಓ෎ݝ
    ౳
    ஫
    ૹ৴ݩ*1ΞυϨεΛ ಗ໊Խ ຤ඌΦΫςοτ ࡟আ ޙʹ(FP*1Ͱਪఆ ʲ࠶ੜ඼࣭ύϥϝʔλʳ w ಈը৘ใ
    ϏοτϨʔτɺ ղ૾౓ɺϑϨʔϜϨʔτ
    ౳
    w ࠶ੜ৘ใ
    ࠶ੜͷதஅɺ όοϑΝϦϯά
    ౳
    w ૢ࡞৘ใ
    ಈըͷγʔΫɺ Ұ࣌ఀࢭɾ࠶։
    ౳ 8FC
    7JEFP.BSL
    ֦ுػೳ΍ϒϥ΢βʔͰࢹௌ৘ใΛऔಘ ࢹௌͨ͠ಈըͷ
    ମײ඼࣭ 2P& Λ dͷ਺஋ͱͯ͠ ϑΟʔυόοΫ ಈըαʔϏεͷ
    ਪఆମײ඼࣭஋ 2P& 2VBMJUZ
    PG
    &YQFSJFODF ಈը഑৴αʔϏεɾ $%/
    αʔόʔ 8FC
    7JEFP.BSL
    
    αʔόʔ ࢹௌ৘ใΛ෼ੳ ௨৴ࣄۀऀɾ*41 αʔϏεվળ༻ʹ
    ௐࠪσʔλΛఏڙ ར༻ऀͷ
 ϒϥ΢βʔ ʂ ಈըࢹௌ࣌ͷ
    2P&
    Λ֬ೝ ϓϥΠόγʔอޢॲཧ ֶज़ݚڀ༻σʔλ࡞੒ ಈը഑৴αʔϏε΁ΞΫηε )5.-ɺಈըϑΝΠϧ౳Λड৴ ಗ໊Խॲཧ
34. (58%) (Chrome) (Web VideoMark) Android (VideoMark Browser) ( ) ( ) 34
35. https://urls.jp/vm 35
36. SISVEL
37. 2019 SISVEL 2019.03 SISVEL JVC , NTT, Orange, Philips, IPR 5 (3/5 ) NAV Show 2019.04 AoM AoM Patent License 1.0 2019.04 SISVEL AV1 Google On2 VP8 OSS 10 ( ) 37
38. SISVEL 2019.03 2019.07 2019.08 2019.10 7-8 38 14 2019 MAR APR MAY JUN JUL AUG SEP OCT CALL FOR PATENTS + THIRD PARTY EVALUATION: ONGOING ACTIVITY LAUNCH OF THE PROGRAM PATENT BROCHURE PUBLICATION APPROACH TO MARKET VP9 AND AV1 TIMELINE
39. SISVEL AV1/VP9 ( ) (STB ) ( ) 39 16 Consumer Display Devices e.g.: • smartphones, tablets, notebooks, computers, convertibles, • TVs, video projectors, • virtual reality and/or augmented reality devices, • photo cameras (with display) and video cameras (with display). Consumer Non‐Display Devices e.g.: • set‐top‐boxes, video and gaming consoles, dongles, • decoders and players, home theatres and streaming media players, • photo cameras (without display), video cameras (without display), • desktop PCs (without display) and graphic cards LICENSE OFFER (II) PRODUCTS 15 LICENSE OFFER (I) COVERAGE • The licenses offered cover any consumer product making use of the VP9 or AV1 specifications. • The licenses offered by Sisvel do not cover: – content on physical digital media storage or non‐physical content distribution; or – components or subassemblies such as, without limitation, chipsets, semiconductor components or embedded modules
40. SISVEL AV1/VP9 ( ) (STB ) ( ) 39 16 Consumer Display Devices e.g.: • smartphones, tablets, notebooks, computers, convertibles, • TVs, video projectors, • virtual reality and/or augmented reality devices, • photo cameras (with display) and video cameras (with display). Consumer Non‐Display Devices e.g.: • set‐top‐boxes, video and gaming consoles, dongles, • decoders and players, home theatres and streaming media players, • photo cameras (without display), video cameras (without display), • desktop PCs (without display) and graphic cards LICENSE OFFER (II) PRODUCTS
41. SISVEL AV1/VP9 25% ( ) 40 20 Confidential - Under NDA Licensee concludes license agreement (Effective Date of license agreement) Compliant Rate to be applied for future sale Standard Royalty Rate to be applied for past sale Period of Licensee’s non-compliance Standard Royalty Rate to be applied for all units including reported units and underreported units. Date of breach cured Compliant Rate to be applied again Found non-compliance Time line Date of breach started COMPLIANT RATE HOW DOES IT WORK 17 VP9 AV1 Standard Rate Compliant Rate Consumer Display Device EUR 0.24 EUR 0.18 Consumer Non‐Display Device EUR 0.08 EUR 0.06 Standard Rate Compliant Rate Consumer Display Device EUR 0.32 EUR 0.24 Consumer Non‐Display Device EUR 0.11 EUR 0.08 LICENSE OFFER (III) ROYALTY RATES – RUNNING For a licensee that is in full compliance with its obligations under the license agreement(s), the royalty rate payable for each licensed product shall be the Compliant Rate.
42. SISVEL VP9/AV1 http://www.sisvel.com/licensing-programs/audio-and-video-coding-decoding/video-coding-platform/license-terms/av1-license-terms 41 19 Consumer Display Devices Minimum Commitment threshold (no. of units) Maximum Commitment threshold (no. of units) Applicable royalty rate per unit 1 100,000 EUR 0.168 100,001 1,000,000 EUR 0.156 1,000,001 25,000,000 EUR 0.144 25,000,001 75,000,000 EUR 0.132 75,000,001 or above unlimited EUR 0.120 Minimum Commitment threshold (no. of units) Maximum Commitment threshold (no. of units) Applicable royalty rate per unit 1 20,000 EUR 0.056 20,001 200,000 EUR 0.052 200,001 5,000,000 EUR 0.048 5,000,001 14,000,000 EUR 0.044 14,000,001 or above unlimited EUR 0.040 Consumer Non‐Display Devices LICENSE OFFER (V) ROYALTY RATES – COMMITTED VOLUMES – AV1 18 Consumer Display Devices Minimum Commitment threshold (no. of units) Maximum Commitment threshold (no. of units) Applicable royalty rate per unit 1 100,000 EUR 0.126 100,001 1,000,000 EUR 0.117 1,000,001 25,000,000 EUR 0.108 25,000,001 75,000,000 EUR 0.099 75,000,001 or above unlimited EUR 0.090 Minimum Commitment threshold (no. of units) Maximum Commitment threshold (no. of units) Applicable royalty rate per unit 1 20,000 EUR 0.042 20,001 200,000 EUR 0.039 200,001 5,000,000 EUR 0.036 5,000,001 14,000,000 EUR 0.033 14,000,001 or above unlimited EUR 0.030 Consumer Non‐Display Devices LICENSE OFFER (IV) ROYALTY RATES – COMMITTED VOLUMES – VP9
43. SISVEL VP9/AV1 VP9 + AV1 0.21 0.56/ (display ) : 2018 15.55 0.21/unit 400 / AV1 0.12 0.24/ AVC by MPEG/LA $0.1 0.2/ HEVC by MPEG/LA $0.2/ 
 HEVC by HEVC Advance $0.325 0.566/ AVC/HEVC by MPEG/LA 10 / AVC by MPEG/LA $9.75M ( 10 )/ HECV 5 ... 42
44. Patent License : @yohhoy
45. AV1 AoM Patent License 1.0 Licensor ( ) AoM ( ...) 44
46. AoM Patent License (RAND ) "Necessary Claims" W3C Essential Claim AoM Licensor / Essential Claim: ( ) https://aomedia.org/license/patent-license/ 1.1. Patent License. 45
47. AoM Patent License : 1.2.1. Availability. ... Licensee must make its Necessary Claims available under this License ... OSS AV1 ( ) Licensor AoM AV1 Licensee 2.9. Licensor. Licensor means (i) any Licensee ... or (ii) a person or entity ... its membership and/or participation in the Alliance for Open Media working group ... AV1 Licensee Licensor https://aomedia.org/license/patent-license/ 1.2.1. Availability 46
48. AoM Patent License : 1.3. Defensive Termination. If any Licensee, ... initiates patent litigation ... against another entity or any person asserting that any Implementation infringes Necessary Claims, any patent licenses granted under this License directly to the Licensee are immediately terminated as of the date of the initiation of action Necessary Claims AoM Patent License https://aomedia.org/license/patent-license/ 1.3. Defensive Termination 47
49. AoM Patent License : If any Licensee, its Affiliates, or its agents initiates patent litigation or files maintains, or voluntarily participates in a lawsuit against another entity or any person asserting that any Implementation infringes Necessary Claims ... "Affiliate" Necessary Claim SISVEL "Affiliate" https://aomedia.org/license/patent-license/ 1.3. Defensive Termination 48
50. AoM Patent License : "A liate" 2.1. Affiliate. Affiliate means an entity that directly or indirectly Controls, is Controlled by, or is under common Control of that party. 2.2. Control. Control means direct or indirect control of more than 50% of the voting power to elect directors ... 100% (subsidairy) affiliate 50% 縄 ( ) https://aomedia.org/license/patent-license/ 2.1. Affiliate, 2.2. Control. 49
51. : NTT : http://www.ntt.co.jp/gnavi/ 50 AV1 WebRTC AV1
52. : IPR & : https://www.toshiba.co.jp/contact/guide/ts_group.htm 51
53. : JVC JVC JVC JVC JVC JVC JVC JVC JVC : https://www.toshiba.co.jp/contact/guide/ts_group.htm 52 AV1 AoM 

54. SISVEL ( ) ( ) ( ) SISVEL SISVEL AV1 = AV1 AV1 53
55. SISVEL AV1 AV1 AV1 = Web Android (Android Google 縄 AoM ) Android AV1 ( ) 54
56. 51 51 AOM Members - Today (42) 51 VS
57. Any Questions? Feel free to contact: dynamis webdino.org
 @dynamitter, facebook.com/dynamis Slide: speakerdeck.com/dynamis 56 @
58. https://urls.jp/vm 57