スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

by FromAtom

Slide 1

Slide 1 text

εΫϦʔϯ഑৴ػೳͷ ࣮૷͕େมͩͬͨͷͰ ஌ݟΛ͓఻͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A pixiv.inc FromAtom

Slide 2

Slide 2 text

• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք΁ • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch / pixiv Sketch LIVE iOSΞϓϦ։ൃ @FromAtom

Slide 3

Slide 3 text

Slide 4

Slide 4 text

͜ͷൃදͷΰʔϧ

Slide 5

Slide 5 text

ϥΠϒ഑৴ྲྀߦͬͯΔ͠ɺ ͏ͪ΋ϥΠϒ഑৴΍Δͧʂʂʂ

Slide 6

Slide 6 text

ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ

Slide 7

Slide 7 text

εΫϦʔϯ഑৴ػೳ

Slide 8

Slide 8 text

εΫϦʔϯ഑৴ػೳͱ͸ εϚʔτϑΥϯͷը໘ΛϥΠϒ഑৴Ͱ͖Δػೳ

Slide 9

Slide 9 text

• εϚϗ୯ମͰϥΠϒ഑৴͕Մೳ • ήʔϜ࣮گ΍͓ֆ͔͖ͷ഑৴ʹར༻͞Ε͍ͯΔ • ࠷ۙ͸VTuberք۾Ͱ΋ར༻͞Ε͍ͯΔ εΫϦʔϯ഑৴ػೳͱ͸

Slide 10

Slide 10 text

ReplayKitͱεΫϦʔϯ഑৴

Slide 11

Slide 11 text

• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩ࿥΍LIVE഑৴͕؆୯ʹͰ͖Δ • ϚΠΫԻ੠΍ΞϓϦ͕ग़ྗ͍ͯ͠ΔԻ΋औಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ ReplayKitͱεΫϦʔϯ഑৴

Slide 12

Slide 12 text

• iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ ‣ ഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕഑৴ɾ࿥ըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘࿥ըͰ͖Δͷͱಉ͡࢓૊Έ ReplayKitͱεΫϦʔϯ഑৴

Slide 13

Slide 13 text

ࠓ͔Β࡞ΔͳΒ ࠷௿Ͱ΋iOS 11+

Slide 14

Slide 14 text

͡Ό͋ReplayKitʹ͍ͭͯௐ΂ͯΈΑ͏

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

5͔݅͠هࣄ͕Ͷ͐ʂ

Slide 18

Slide 18 text

ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ

Slide 19

Slide 19 text

• ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ Χϝϥ΍Push௨஌ͱಉ͡ • iOSඪ४ͷը໘ऩ࿥΍QuickTimeܦ༝Ͱͷը໘࿥ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δ௚લʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢Δ೔ʑΛա͝͠·ͨ͠ ͪΐͬͱͨ͠஫ҙ఺

Slide 20

Slide 20 text

ΞϓϦͷը໘Λ࿥ըͰ͖Δ iOS 9+

Slide 21

Slide 21 text

Slide 22

Slide 22 text

࿥ը։࢝

Slide 23

Slide 23 text

࿥ըऴྃ

Slide 24

Slide 24 text

iOS 10+ ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ

Slide 25

Slide 25 text

• Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • ഑৴Λ͍ͨ͠ΞϓϦͱ഑৴͢ΔΞϓϦͷ૒ํͰ࣮૷͕ඞཁ iOS10Ͱͷը໘഑৴

Slide 26

Slide 26 text

Slide 27

Slide 27 text

഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

Slide 28

Slide 28 text

഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

Slide 29

Slide 29 text

഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

Slide 30

Slide 30 text

഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

Slide 31

Slide 31 text

Slide 32

Slide 32 text

഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷ -഑৴ऴྃ-

Slide 33

Slide 33 text

഑৴͢ΔΞϓϦͰͷ࣮૷ • Broadcast Upload ExtensionΛ࣮૷͢Δ • ৄ͘͠͸͜ͷޙ iOS11+ ͷ࿩ͱҰॹʹ͠·͢

Slide 34

Slide 34 text

iOS 11+ Ͳͷը໘΋഑৴ɾ࿥ըͰ͖Δ

Slide 35

Slide 35 text

• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴ iOS11+Ͱͷը໘഑৴

Slide 36

Slide 36 text

• ࣮૷ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBuﬀerΛऔಘͯ͠࿥ըɾ഑৴͕Մೳʹ • ϚΠΫԻ੠΍ΞϓϦԻ੠΋औಘՄೳ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ

Slide 37

Slide 37 text

࣮૷

Slide 38

Slide 38 text

• ίϯτϩʔϧηϯλʔͷ˕͔Βબ୒Ͱ͖Δ • iOS9+ͱ͸ҧ͍ը໘શମ͕ऩ࿥Ͱ͖Δ ‣ ௨஌΋ऩ࿥͞ΕΔͷͰ஫ҙ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴

Slide 39

Slide 39 text

Slide 40

Slide 40 text

iOS 11+ ΍΍͍͜͠ͷͰҰ୴੔ཧ

Slide 41

Slide 41 text

• iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ͕഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩ࿥ɾ࿥ը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴ ReplayKitͱεΫϦʔϯ഑৴

Slide 42

Slide 42 text

iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌׬ྃ

Slide 43

Slide 43 text

iOS 11+ Broadcast Upload Extension

Slide 44

Slide 44 text

• ࠓճ͸͜ͷExtensionͷ࿩ • Լهͷ2ͭʹ֘౰ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ ‣ iOS11+ - ίϯτϩʔϧηϯλʔܦ༝Ͱ഑৴Ͱ͖Δ Broadcast Upload Extension

Slide 45

Slide 45 text

࣮૷खॱ

Slide 46

Slide 46 text

࣮૷खॱ

Slide 47

Slide 47 text

࣮૷खॱ

Slide 48

Slide 48 text

࣮૷खॱ

Slide 49

Slide 49 text

iOS 11+ SetupUI Extensionͬͯͳʹʁʁ

Slide 50

Slide 50 text

SetupUI Extensionͱ͸

Slide 51

Slide 51 text

SetupUI Extensionͱ͸

Slide 52

Slide 52 text

• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ৔߹͸Target͔Β࡟আ͢Ε͹Α͍ SetupUI Extensionͱ͸

Slide 53

Slide 53 text

࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ • StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏

Slide 54

Slide 54 text

࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ • StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏

Slide 55

Slide 55 text

࣮૷खॱ • TARGETS͔ΒSetupUI ExtensionΛબ୒ • General಺ͷDeployment Info͔ΒMain InterfaceΛมߋ

Slide 56

Slide 56 text

࣮૷खॱ • InfoλϒΛ։͍ͯ "NSExtension" ಺ͷ "NSExtensionPrincipalClass" Λফ͢ • "NSExtension" ಺Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ

Slide 57

Slide 57 text

࣮૷खॱ • ޙ͸ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮૷ • ഑৴༻ͷॲཧ͸Broadcast Upload ExtensionଆͰ࣮૷ • SetupUI ExtensionͰ͸LIVE഑৴ʹඞཁͳ৘ใΛೖྗͯ͠΋Β͏ ‣ e.g. ഑৴ͷλΠτϧɺίϝϯτɺެ։ൣғ

Slide 58

Slide 58 text

iOS 11+ Ұ୴੔ཧ

Slide 59

Slide 59 text

iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension

Slide 60

Slide 60 text

iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ഑৴͍ͨ͠

Slide 61

Slide 61 text

iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension SetupUI ഑৴͍ͨ͠

Slide 62

Slide 62 text

iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuﬀer

Slide 63

Slide 63 text

iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuﬀer ੈք

Slide 64

Slide 64 text

iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ

Slide 65

Slide 65 text

iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ ഑৴͍ͨ͠

Slide 66

Slide 66 text

iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuﬀer ഑৴͍ͨ͠

Slide 67

Slide 67 text

iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuﬀer ഑৴͍ͨ͠ ੈք

Slide 68

Slide 68 text

iOS 11+ Broadcast Upload Extension

Slide 69

Slide 69 text

• ॳظঢ়ଶͰ͸ϑΝΠϧ͸2ͭ • "SampleHandler.swift" ʹ࣮૷Λॻ͍͍ͯ͘ ࣮૷खॱ

Slide 70

Slide 70 text

iOS 11+ SampleHandler.swift ʹ ࣮૷Λॻ͍͍ͯ͘

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

• ഑৴։࢝࣌ʹݺ͹ΕΔ • ϩάΠϯॲཧ • ഑৴༻ͷԼॲཧ

Slide 73

Slide 73 text

• ഑৴͕ϙʔζɾ࠶։͞ΕͨΒݺ͹ΕΔ

Slide 74

Slide 74 text

• broadcastFinishedɿ഑৴ऴྃ࣌ʹݺ͹ΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺ͹ΕΔ

Slide 75

Slide 75 text

• ͜͜ͰCMSampleBuﬀerΛड͚औΔ • ͋ͱ͸αʔόαΠυʹૹ৴͢Ε͹Α͍

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճ͸WebRTCͷ࿩ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻ ‣ Sora iOS SDK͕͋ΔͷͰiOS͸ͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios CMSampleBuﬀerΛૹΔ

Slide 78

Slide 78 text

࣮૷खॱ

Slide 79

Slide 79 text

໰୊ • Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ͸50MB·Ͱ • ௒͑Δͱ໰౴ແ༻ͰKill͞ΕΔ • Sora iOS SDK͕CMSampleBuﬀerΛVP8Τϯίʔυ͢Δࡍʹ௒͑Δ ‣ VP8͸CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ

Slide 80

Slide 80 text

ղܾํ๏ • CMSampleBuﬀerΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuﬀer

Slide 81

Slide 81 text

ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer

Slide 82

Slide 82 text

ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage

Slide 83

Slide 83 text

ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)

Slide 84

Slide 84 text

ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer CIImage transformed(by:)

Slide 85

Slide 85 text

ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer CIImage transformed(by:) SDK΁

Slide 86

Slide 86 text

ϦαΠζͨ͠ΒૹΕͨ

Slide 87

Slide 87 text

• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛ໨ࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ୒͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎Ͱ΋ϝϞϦ͸࢖͏ͷͰ஫ҙ ੍ݶΛ௒͑ͳ͍αΠζΛ୳Δ

Slide 88

Slide 88 text

iOS 11+ Ի੠

Slide 89

Slide 89 text

• ϚΠΫԻ੠ͱΞϓϦԻ੠ • ͳʹ΋͠ͳ͍ͱϚΠΫԻ੠͸औಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚͹औಘͰ͖Δ Ի੠

Slide 90

Slide 90 text

iOS 11+ Sora iOS SDK͸libwebrtcͷ࢓্༷ Extension্ͰԻ੠͕ͱΕͳ͍

Slide 91

Slide 91 text

• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠૷ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ଄͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத • RTMP(S) or HLSͰHaishinKitΛར༻͢Ε͹औಘՄೳ Sora iOS SDK with libwebrtc

Slide 92

Slide 92 text

To Be Continued

Slide 93

Slide 93 text

iOS 11+ ͔͜͜Β͸ࡉ͔ͳϊ΢ϋ΢ूͰ͢

Slide 94

Slide 94 text

• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuﬀer͕ৼͬͯ͘Δ • ղܾࡦ͸2छྨ ‣ Ի੠ͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ഑৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ - ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ੒͢Δํ๏ - ΫϥΠΞϯταΠυͰ͸APIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛ΋ͱʹੜ੒͢Δ ը໘ʹมԽ͕ແ͍ͱ͖

Slide 95

Slide 95 text

• ը໘ऩ࿥͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔൑ผ͢Δ • ը໘ऩ࿥ͷ։࢝ɾऴྃΛݕ஌͢Δ ը໘ऩ࿥Λݕ஌͍ͨ͠ͱ͖

Slide 96

Slide 96 text

• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘

Slide 97

Slide 97 text

• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏

Slide 98

Slide 98 text

• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍৘ใ • ը໘ऩ࿥தʹGPUΛ࢖͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰ΋ࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍

Slide 99

Slide 99 text

• εΫϦʔϯ഑৴ػೳΛ࡞Γ͍ͨͱ͖͸ReplayKitΛ࢖͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮૷͠·͠ΐ͏ • ϝϞϦ͕50MB͔͠࢖͑ͳ͍ͷͰ஫ҙ͠·͠ΐ͏ ·ͱΊ