ۭؒϏσΦϑΥʔϚοτ MV-HEVC͕ࣸ͠ࡱΔੈք ߦ໦ ઍय़ʢͳΊ͖ ͪ͸ΔʣGitHub, X @Ridwy Streaming Client Team, גࣜձࣾ AbemaTV iOSDC JAPAN 2024, day1 (8/23)

גࣜձࣾϐΫηϥ Mac, iOS޲͚TVνϡʔφʔ, ಈըฤूΞϓϦ nana music ΦʔσΟΦΤϯδϯ, ΤϑΣΫλ։ൃ גࣜձࣾ AbemaTV Streaming Client Team ABEMA͕ରԠ͢ΔશͯͷσόΠεͷ ϓϨΠϠʔͷ࣮૷ͱ࠶ੜ඼࣭ͷ୲อ ܦྺ 3DΧϝϥ΍3DςϨϏͱग़ձ͏ ഑৴ܗࣜͳͲΛৄ͘͠஌Γ͍ͨʂ

Vision Pro Ͱ Immersive VideoΛମݧ͠·͔ͨ͠ʁ Apple 2024/07/18 ϓϨεϦϦʔεΑΓ

͋Δ͍͸ ࣗ෼ͰࡱӨۭͨؒ͠ϏσΦΛ Apple 2023/12/11 ϓϨεϦϦʔεΑΓ

ྟ৔ײΛੜΈग़͢ཁҼ ໨ͷલʹେ͖͘޿͕Δදࣔ + ࠨӈͷ໨ͷݟ͑ํͷҧ͍ʢࢹࠩʣͷ࠶ݱʹΑΔ Ԟߦ͖ײ

ΞδΣϯμ • ࢹࠩʹΑΔཱମࢹ • ه࿥ίʔσοΫ MV-HEVC • Apple͕ఏڙ͢Δ̏ͭͷۭؒϝσΟΞࢹௌମݧ • 3DϏσΦ • Apple Immersive Video • ۭؒϏσΦ • ۭؒϏσΦΛऔΓר͘ঢ়گ

ࢹࠩͱ͸... ໨Λยํͣͭดͯۙ͘͡ͷ෺ΛݟΔͱɺਫฏํ޲ͷҐஔ͕গ͚ͩ͠ҟͳΔ ӈ໨ ࠨ໨

ࠨӈͷ໨ʹͪΐͬͱζϨͨө૾Λ༻ҙ͢Ε͹ ྑ͍ײ͡ͷମݧʹͳΔʁ NO ࣗવͳମݧͷͨΊʹ͸഑ྀ΍޻෉͕ඞཁ

ਓ͸Ԟߦ͖ͷ೺ѲʹࢹࠩҎ֎΋࢖͏ ੜମػೳ • ϐϯτௐઅΛ͢Δے೑͔ΒͷϑΟʔυόοΫ • ӡಈࢹࠩʢମ͕ಈ͚͹ݟ͑ํ΋มΘΔʣ ൃୡաఔͰֶशͨ͠಺༰ • র໌ͷӄӨ΍൓ࣹ • ःṭ • ෺ͷେ͖͞ etc ࢹ͚ࠩͩ࠶ݱͯ͠΋ଞͱໃ६͕͋Δͱ νϥ͖ͭ΍؟ͷർ࿑͕ىͬͯ͜͠·͏ ણࡉŦƄżʂ

ࢹཱࠩମࢹͷ໰୊ ᫔᫓ͱௐઅͷෆҰக

྆໨Ͱ෺ΛݟΔ࣌͸ɺ؟ٿΛͦͪΒʹ޲͚Δ ͜ͷے೑ͷಈ͖΋Ԟߦ͖Λײ͡ΔͨΊͷॏཁͳख͕͔Γ ᫔᫓ (Convergence) 🍏 ᫔᫓֯ ŧŔŕŪدΓ໨

ย໨͝ͱʹਫথମͷް͞Λௐઅ 🍏 ϐϯτௐઅ

🍏 ย໨ͷػೳΏ͑ʹ ϐϯτ͸σΟεϓϨΠʹ ߹Θ͍ͤͨ ໨ઢ͸ͬͪ͜ʹ߹͏ Convergence Point ᫔᫓ͷมԽʹௐઅ͸ҾͬுΒΕΔ͕ গ͢͠Δͱௐ੔͸ຊདྷͷҐஔʹ໭͍ͬͯ͘ ͜ͷ2͕ͭҰக͠ͳ͍ͱෆշײ͕ੜ͡Δ

ࢹ͕ࠩখ͚͞Ε͹᫔᫓ͱௐઅͷෆҰகʹΑΔෆշײ͸ײ͡ʹ͍͘ • ࢹ͕ࠩ͋·Γʹ΋େ͖͍ͱ྆໨ͷ૾Λ༥߹Ͱ͖ͳ͍ • ࢹࠩ2౓ҎԼ → ཱମײΛ͔ͬ͠Γײ͡Δ͕ർΕΔ • ࢹࠩ1౓ҎԼ → ཱମײ͸ऑ͍͕ɺ௕࣌ؒݟͯ΋շద • ݁ہεΫϦʔϯ໘෇͕ۙҰ൪ݟ΍͍͢ 😇 ର৅෺ͷڑ཭ʹ஫ҙͯ͠ө૾Λ੍࡞͢Δ ͕͜͜໨҆ʂ

ཱମࢹྗͷ෼཭ᮢ͸௨ৗࢹྗͷ30ഒʂ ߴਫ਼ࡉσΟεϓϨΠ͕ඞཁ

௨ৗࢹྗ = 2఺Λ෼཭Ͱ͖Δ ෼཭ᮢ: 1 ෼֯ʢ1/60౓ʣ 2DͷσΟεϓϨΠ΍ө૾ૉࢠ͸ɺ͜ΕΛຬ଍ͤ͞Δ͜ͱΛ໨ඪʹ࡞ΒΕ͍ͯΔ ཱମࢹྗ = Ԟߦ͖ͷࠩΛײ͡Δ͜ͱ͕Ͱ͖Δ ෼཭ᮢ: 2 ඵ֯ ௨ৗࢹྗͷ30ഒʂ ͬͪ͜΋ຬ଍͍ͤͨ͞ʂ ※ ࢀߟ: Vision Pro ղ૾౓ ย໨ 3660 x 3200 → 90°/3660 ≒ 1.5෼֯

ͦΜͳVision ProͰࢹࠩಈըΛָ͠ΉͨΊʹ͸ ࠨӈͷ໨ʹ޲͚ͨߴղ૾౓ͷө૾͕ඞཁ ө૾ίʔσοΫ MV-HEVC

HEVC (H.265) ͸ AVC (H.264) ΑΓߴղ૾౓ͷө૾Λޮ཰Α͘ѹॖͰ͖Δ HEVC ( High Ef f iciency Video Coding ) "7$ ) $6Y )&7$ ) $6Մม

MV-HEVC ( Multiview HEVC) ࣌ؒతʹ͍ۙϑϨʔϜ͕ࣅ͍ͯΔ͜ͱΛར༻ͯ͠ѹॖ ෳ਺ͷViewΛѻ͑ΔΑ͏ʹʹ֦ுͨ͠HEVC ISO/IEC 23008-2 (ITU H.265), 2014 I B B B B B B B P ࠨ໨༻ͷView Layer 0 P B B B B B B B P Viewؒͷ ྨࣅ΋ར༻ ͯ͠ѹॖ ӈ໨༻ͷView Layer 1 ͜ͷ֦ுʹඇରԠͷHEVCσίʔμͰ͸ϕʔεͷLayer 0͚͕ͩग़ྗ͞ΕΔ

MV-HEVC Ͱࢹࠩө૾Λӡ༻͢ΔͨΊͷPro f ile͕ఏҊ͞Ε͍ͯΔ ෳ਺ViewΛѻ͏ө૾ίʔσοΫͰ͋Δ ʮMV-HEVCΛࢹࠩө૾Ͱ࢖͏࣌͸͜͏͠Α͏Ͷʯ ͱ͍͏໿ଋࣄ 2023/6/21 Version 0.9 (Beta)

ࢹࠩө૾ʹؔ͢Δ৘ใͷอଘϑΥʔϚοτ΋ެ։͞Εͨ 2023/6/21 Version 0.9 (Beta) vexu eyes stri hero Video Extended Usage Box Stereo View Information Box Hero Stereo Eye DescriptionBox

MV-HEVC Τϯίʔμʔ/σίʔμʔ Vision Proͷొ৔Ҏ߱ɺ࣮૷͕ग़࢝Ί͍ͯΔ • Apple APIఏڙ͋Γ AVAssetWriterInputTaggedPixelBufferGroupAdaptor, iOS 17+ AVPlayerVideoOutput, iOS 17.2+ • MainConcept @ NAB Show 2024 • Ateme ffmpegͳͲͷπʔϧͷରԠ͸ࠓޙʹظ଴ Τϯίʔυ σίʔυ

ࢹࠩಈըͷετϦʔϛϯά Stereoscopic Video on HTTP Live Streaming ετϦʔϜͷө૾ϨΠΞ΢τΛࣔͨ͢Ίͷଐੑ REQ-VIDEO-LAYOUT • CH-MONO ैདྷͷ2Dө૾ • CH-STEREO ࢹࠩө૾ #EXTM3U #EXT-X-VERSION:12 #EXT-X-STREAM-INF:BANDWIDTH=910558, ...(ུ)... ,REQ-VIDEO-LAYOUT="CH-STEREO" prog_index.m3u8

Apple͕ఏڙ͢Δ 3ͭͷۭؒϝσΟΞࢹௌମݧ • 3DϏσΦ • Apple Immersive Video • ۭؒϏσΦ ࢹཱࠩମࢹʹΑΔಈըମݧΛ"QQMF͸ͲͷΑ͏ʹϓϩμΫτԽͨ͠ͷ͔ʁ શͯʹ.7)&7$͕ར༻͞Ε͍ͯΔ

3DϏσΦ 3Dөը࡞඼ͳͲ • өըؗ΍ϦϏϯάͰେը໘Ͱ࡞඼Λָ͠ΉΠϝʔδ • ެࣜͷ Apple TVΞϓϦଞɺAVPlayer΍SwiftUIͷVideoPlayerͰ࠶ੜͰ͖Δ • QuickLookͰ͸2D࠶ੜ өըͷ࡞தͰཱମײ͸ҰఆͰ͸ͳ͘ɺUIʹೃછ·ͤΔͷ͸೉͍ͨ͠Ί

Apple Immersive Video ɹࢹ໺͍ͬͺ͍ʹ޿͕Δ຅ೖײ͋Δϓϩίϯςϯπ • Apple TVΞϓϦͰͷࢹௌ • ίϯςϯπ͸·ͩগͳ͍͕ɺଓʑͱAppleͰ੍࡞͕ਐΜͰ͍Δ • 180౓ͷࢹ໺֯ • ย໨8K 3DϏσΦ • ۭؒΦʔσΟΦ ϑΥʔϚοτͷৄࡉ͸ෆ໌ ഑৴͞Ε͍ͯΔετϦʔϜΛΩϟϓνϟͯ͠ݟΔͱ... MV-HEVC ย໨4320×4320, 90fps, HDR10, Fish-eye෩ͷProjectionʢ౤Өํ๏ʣ

੍࡞ϫʔΫϑϩʔ 2024೥ޙ൒ʹBlackmagicͱApple͔ΒแׅతͳπʔϧΩοτ͕ఏڙ͞ΕΔ༧ఆ Blackmagic URSA Cine Immersive DaVinci Resolve Studio Apple Compressor Vision ProͰ֬ೝ͠ͳ͕Βฤू

Apple Immersive Video͕ݟΒΕΔͷ͸ Apple TVΞϓϦ͚ͩ...ʁ • ͨͿΜͦ͏ ࠓͷॴɺ࠶ੜAPI͸ఏڙ͞Ε͍ͯͳ͍ • Reality Composer ProͰࢹքΛ෴͏Α͏ʹͯ͠ಈըΛ࠶ੜ͢Ε͹ ྨࣅͷମݧ͸ఏڙͰ͖Δ

ۭؒϏσΦ • ࣮෺େ + Ԟߦ͖ײͰࢥ͍ग़ͷ৔໘ʹ໭͔ͬͨͷΑ͏ͳମݧ • ࢹքશͯΛ෴͑ΔΘ͚Ͱ͸ͳ͘ɺ90౓ఔ౓·Ͱ • ϑνͷ෦෼͸΅͔͞ΕͨදࣔʹͳΔ • ެࣜͷࣸਅΞϓϦɺQuickLook framework ͷ PreviewApplication APIͰ࠶ੜՄೳ • Final Cut ProͰฤूՄೳʹͳΔ༧ఆ iPhone 15 Pro ΍ Vision Pro ͰϢʔβʔ͕ࡱӨՄೳʂ

ۭؒϏσΦ͕ࡱӨͰ͖ΔσόΠε iPhone 15 Pro, iPhone 15 Pro Max • ެࣜͷΧϝϥΞϓϦ, iOS 17.2+ • AVCaptureSession, iOS 18+ Vision Pro • ެࣜͷΧϝϥΞϓϦ, visionOS 1.0+ ޿֯ ௒޿֯ ๬ԕ ۙ͘ΛࡱΔͷʹ޲͍͍ͯΔϨϯζ2ͭ ޿֯ͱ௒޿֯Λԣฒͼʹͯ͠ར༻ ※ iPhone 14 ProͰ͸Ϩϯζ഑ஔ͕ҟͳΔͨΊࡱӨͰ͖ͳ͍

ʮPoint-and-shoot ͰࡱΕΔʯ ΧϝϥΛඃࣸମʹ޲͚ͯγϟολʔΛԡ͚ͩ͢ʂ ࣮͸޻෉͕͍ͬͺ͍ʂ

3Dөը੍࡞Ͱ͸ εςϨΦάϥϑΝʔͱ͍͏ઐ໳Ո͕׆༂͍ͯ͠Δ ݱ৔Ͱ͸Χϝϥؒڑ཭ͳͲͷௐ੔Λ୲౰ ࢹࠩޮՌͰ࡞඼ΛΑΓັྗ͋Δ΋ͷʹ͢Δ໾ׂ

ࠨӈͷΧϝϥͷઃఆΛಉ͡ʹ͢Δ ҉͔ͬͨΓඃࣸମ͕ۙ͗ͨ͢ΓͰζϨ͕ੜͯ͡͠·͏࣌͸ܯࠂΛදࣔ

ਫฏΛอͭ ਫฏ͕Θ͔ΔUIΛදࣔ

खϒϨରࡦΛ͢Δʢେը໘ͳͷͰखϒϨͷෆշײ͕ڧ͍ʣ ڧྗͳखͿΕิਖ਼

https://support.apple.com/ja-jp/guide/apple-vision-pro/dev7068c3c93/visionos ᫔᫓-ௐઅͷෆҰகʹΑΔർ࿑Λආ͚ΔͨΊ ඈͼग़͗͢͠ͳ͍Α͏ʹ͢Δ ʮ͖ۙͮ͗͢ͳ͍ʯ ϕετϓϥΫςΟεͱͯ͠ఏࣔ ͜Ε͚ͩͰେৎ෉...ʁ 🤔 https://support.apple.com/ja-jp/guide/iphone/iph6e3a6d4fe/ios

͍͍ͩͨେৎ෉

ࢹ͕ࠩখ͚͞Ε͹᫔᫓ͱௐઅͷෆҰகʹΑΔෆշײ͸ײ͡ʹ͍͘ • ࢹ͕ࠩ͋·Γʹ΋େ͖͍ͱ྆໨ͷ૾Λ༥߹Ͱ͖ͳ͍ • ࢹࠩ2౓ҎԼ → ཱମײΛ͔ͬ͠Γײ͡Δ͕ർΕΔ • ࢹࠩ1౓ҎԼ → ཱମײ͸ऑ͍͕ɺ௕࣌ؒݟͯ΋շద • ݁ہεΫϦʔϯ໘෇͕ۙҰ൪ݟ΍͍͢ 😇 ର৅෺ͷڑ཭ʹ஫ҙͯ͠ө૾Λ੍࡞͢Δ ͕͜͜໨҆ʂ ࠶ܝ

ͲΕ͚ͩ཭ΕΕ͹ྑ͍ͷ͔ʁ ࢹࠩ1౓ҎԼʹͳΔ࠷୹ͷࡱӨڑ཭͸ɺϨϯζִؒʹൺྫ σόΠε Baseline ʢϨϯζִؒʣ ࠷୹ڑ཭ Appleਪ঑ͷࡱӨڑ཭ iPhone 15 Pro iPhone 15 Pro Max 19.24mm 55cm 90cm ʙ 240cm Vision Pro 63.76mm 1.8m - iPhone͸ख࣋ͪɺVision Pro͸ΰʔάϧܕͳͷͰɺͪΐ͏Ͳྑ͍ڑ཭

Apple 2023/12/11 ϓϨεϦϦʔεΑΓ ө૾ͷϑνʹखલʹඈͼग़͢΋ͷ͕͋Δͱݟʹ͍͘ ࠶ੜ࣌ʹϑνΛ΅͔͢ ※ 3Dөը੍࡞Ͱ͸͜͏ͨ͠ը໘ߏ੒Λආ͚Δ ࢹཱࠩମࢹʹ͸ʮճΓࠐΊͳ͍ʯͱ͍͏ऑ఺͕͋Δ͕ ͦΕ͕ࣗવʹࢥ͑ΔޮՌ΋

ۭؒϏσΦ = MV-HEVC + ۭؒϝλσʔλ • Projection ౤Өํ๏ • Baseline Ϩϯζִؒ • Field of View ࢹ໺֯ • Disparity Adjustment ࢹࠩௐ੔ ࡱӨͷઃఆ "࣮෺େ" Λ࠶ݱ͢ΔͨΊʹ ར༻͞Ε͍ͯͦ͏ UI্Ͱඃࣸମ͕ Windowͷ໘ͷഎޙʹདྷΔΑ͏ʹ͢Δ

↑ ۭؒϏσΦ͸ίϨ Projection ౤Өํ๏ ੈքΛฏ໘ʹϚοϐϯά͢Δํ๏ Rectilinear ϐϯϗʔϧΧϝϥ Equirectangular ਖ਼ڑԁ౵ਤ๏ Fisheye

Baseline Ϩϯζִؒ • ਓؒͷ໨ͷִؒʢ͓Αͦ 64mmʣͱಉ͡ʹ͢Δͱ ࠷΋஧࣮ʹԞߦ͖͕࠶ݱͰ͖Δ • ڱ͍ͱཱମײΛڧ͘ײ͡ɺ޿͍ͱฏ໘తʹײ͡Δ Baseline

Field of View ࢹ໺֯

Rectilinear ProjectionͰ͸ࢹ໺֯Λେ͖͘͢Δͱ࢛۱͕࿪ΜͰ֯౓ͷ෼ղೳ͕Լ͕Δ ۭؒϏσΦͰ͸90౓Ҏ্ͷࢹ໺֯͸ਪ঑͞Ε͍ͯͳ͍

Disparity Adjustment ࢹࠩௐ੔ • Vision ProͰදࣔ͢Δͱ͖ʹଞͷUIͱίϯϑϦΫτ͠ͳ͍Α͏ʹ Window ද໘͔Βඃࣸମ͕ඈͼग़͞ͳ͍Α͏ʹ͢Δ • ࠨӈͷө૾Λਫฏʹγϑτ͢Δͱඈͼग़͠/Ԟߦ͖Λௐ੔Ͱ͖Δ ө૾ͷ෯ͷ2%ఔ౓͕ྑ͍ Apple 2023/12/11 ϓϨεϦϦʔεΑΓ

AVAssetWriterInput(mediaType: .video, outputSettings: settings) "77JEFP$PNQSFTTJPO1SPQFSUJFT,FZ L75$PNQSFTTJPO1SPQFSUZ,FZ@1SPKFDUJPO,JOE L75$PNQSFTTJPO1SPQFSUZ,FZ@4UFSFP$BNFSB#BTFMJOF L75$PNQSFTTJPO1SPQFSUZ,FZ@)PSJ[POUBM'JFME0G7JFX L75$PNQSFTTJPO1SPQFSUZ,FZ@)PSJ[POUBM%JTQBSJUZ"EKVTUNFOU

ۭؒϝλσʔλ͸Ͳ͜ʹॻ͔Ε͍ͯΔͷ͔ʁ Video Extended Usage Box಺ & ۙ͘ ະهࡌ ಺༰ΛݟΔͷʹMP4box͕ศར

σόΠε Baseline Ϩϯζִؒ Field of View ࢹ໺֯ Disparity Adjustment ࢹࠩௐ੔ iPhone 15 Pro 19.24mm 63.4౓ 2% Vision Pro 63.76mm 71.59౓ 2.93% ۭؒϝλσʔλͷྫ

ۭؒࣸਅ Vision Pro ͰࡱӨՄೳʢۭؒϏσΦͷࣸਅ൛ʣ • εςϨΦ HEIC + ۭؒϝλσʔλ • ެࣜͷࣸਅΞϓϦɺ QuickLook framework ͷ PreviewApplication APIɺ JavaScriptͷElement FullScreen API ͰݟΔ͜ͱ͕Ͱ͖Δ • ಡΈࠐΈɾॻ͖ग़͠͸ Core Graphics API ͰՄೳ UIImageViewͰ͸ ཱମʹͳΒͳ͍

ۭؒϏσΦΛऔΓר͘ঢ়گ • ࡱӨɾ࠶ੜ • Τϯίʔυɾσίʔυ • ϑΥʔϚοτ࢓༷ APIఏڙ APIఏڙ Pro f ileͳͲ͕ެ։ʢЌʣ ΧϝϥΞϓϦ ฤूΞϓϦ VRήʔϜͷۭؒεΫγϣ ARάϥεͳͲͰݟΔ Χϝϥ΍ϑΥτϑϨʔϜ ARσόΠεͳͲ͕ରԠ͢ΔՄೳੑ Ͱ͖ͦ͏ͳ͜ͱ

Canon EOS R7޲͚ ۭؒϏσΦࡱӨ༻Ϩϯζ RF-S7.8mm F4 STM DUAL lens 2024೥ޙ൒ൃച༧ఆ VITURE One, VITURE Pro ެࣜ SpaceWalker ΞϓϦͰ ۭؒϏσΦ࠶ੜՄೳ ۭؒϏσΦͷΞοϓϩʔυ, ࠶ੜɾγΣΞΛαϙʔτ༧ఆ Vimeo 3rd PartyͷରԠɺଓʑ

ެࣜ Sample Code͕ॆ࣮ Converting side-by-side 3D video to multiview HEVC and spatial video https://developer.apple.com/documentation/avfoundation/media_reading_and_writing/converting_side-by- side_3d_video_to_multiview_hevc_and_spatial_video macOS 15.0+ αΠυόΠαΠυͷಈը → CMTaggedBuffer → MV-HEVC, ۭؒϏσΦ Reading multiview 3D video files https://developer.apple.com/documentation/avfoundation/media_reading_and_writing/reading_multiview_3d_video_ f iles macOS 14.0+ MV-HEVC → CVPixelBuffer Writing spatial photos https://developer.apple.com/documentation/imageio/writing-spatial-photos macOS 14.0+ 2ຕͷը૾ → ۭؒࣸਅ WWDC24, Build compelling spatial photo and video experiences 6:19 - Spatial video capture on iPhone 15 Pro iOS 18+ Beta Beta

͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠