iOSではじめるフォトグラメトリ #iOSDC

iOSDC 2023での発表スライドです。

プロポーザル： https://fortee.jp/iosdc-japan-2023/proposal/31781f59-2e99-48f0-ba0e-db04590c24c4

Transcript

1. iOSͰ͸͡ΊΔ ϑΥτάϥϝτϦ అ मҰ @shu223 iOSDC 2023
2. ࣗݾ঺հ • అ मҰ • @shu223 (GitHub, Qiita, Zenn, note, , YouTube, Podcast, etc...) • ॻ੶ʢ঎ۀग़൛4࡭ɺݸਓग़൛ଟ਺ @BOOTHʣ:
3. • 1೥ؒݚڀ։ൃ෦ͷϚωʔδϟʔʹνϟϨϯδ • 12೥ؒiOS͚ͩ΍͖ͬͯͨΤϯδχΞͷΩϟϦΞνΣϯδ • 2023೥6݄͔ΒΤϯδχΞʹ෮ؼ • Ϛωʔδϟʔ͔ΒΤϯδχΞʹ໭Γ·͢ • ͍·΍ͬͯΔ͜ͱ • BLEΛ༻໊͍ͨࢗަ׵ػೳΛϓϩμΫτ޲͚ʹ࣮૷த
4. աڈͷiOSDCొஃಈը΋ͥͻ ʢ͓͢͢Ίॱʣ 1. ඈͼಓ۩Ͱ͸ͳ͍Metalʢ2017ʣ 2. Depth in Depthʢ2018ʣ 3. ւ֎ΧϯϑΝϨϯεʹొஃ͢Δʢ2016ʣ 4. ػցֶशͷϒϧʔΦʔγϟϯ Core MLʢ2020ʣ 5. ࠓͦ͜watchOSʢ2019ʣ
5. ΞδΣϯμ • ϑΥτάϥϝτϦͱ͸ • ϑΥτάϥϝτϦͷ࣮૷ • ϑΥτάϥϝτϦ࣮ફฤ
6. ΞδΣϯμ • ϑΥτάϥϝτϦͱ͸ • ϑΥτάϥϝτϦͷ࣮૷ • ϑΥτάϥϝτϦ࣮ફฤ
7. ࣸਅ͔Β3DϞσϧΛੜ੒͢Δٕज़
8. σϞ
11. ϑΥτάϥϝτϦͷݪཧ Φʔόʔϥοϓ͢ΔΑ͏࿈ଓతʹࡱӨ
12. ॏͳΓ߹͏ಛ௃͔ΒࣸਅҐஔਪఆ → ఺܈ੜ੒ → ϝογϡੜ੒
13. ʮ3DϞσϧʁήʔϜ੍࡞༻Ͱ͠ΐʁʯ
14. ✋
15. Ϣʔεέʔε: ैདྷͷࣸਅͷஔ͖׵͑ ΧϝϥͰͷࡱӨػೳʗը૾දࣔػೳΛ࣋ͭαʔϏε͢΂͕ͯ ϢʔεέʔεʢͱͳΔՄೳੑʣ • ঎඼ࣸਅ • Ϩετϥϯͷళ಺ࣸਅ • ϢʔβʔΞΠίϯʢc.f. Spatial Computingʣ
16. ࿩͞ͳ͍͜ͱ • APIͷʮ࢖͍ํʯ͸࠷খݶʹ • هࣄͰิ׬ • ϓϩ༻ͷࡱӨػࡐΛ࢖ͬͨຊ֨తͳϑΥτάϥϝτϦͷ࿩ • ͳΔ΂͘iOS/macOSͷٕज़͚ͩΛ࢖͏ • طଘΞϓϦΛ࢖ͬͨϑΥτάϥϝτϦͷ࿩
17. ຊ೔ͷΰʔϧ վળͷࢼߦࡨޡΛ௨ͯ͡ɺຊ࣭తɾ൚༻తͳʮϑΥτάϥϝτ ϦͷצॴʯΛཧղ͢Δ
18. ΞδΣϯμ • ϑΥτάϥϝτϦͱ͸ • ϑΥτάϥϝτϦͷ࣮૷ • ϑΥτάϥϝτϦ࣮ફฤ
19. Object Capture • Apple͸3DεΩϟϯͷ͜ͱΛ͜͏ݺΜͰ͍Δ • Image Capture API ʢࡱӨʣ + Reconstruction APIʢ3D෮ ݩʣ
20. Image Capture API • iOS 17ͷ৽API • ૉਓͰ΋؆୯ʹߴ඼࣭ʹࡱӨͰ͖Δ UI/UX
21. Reconstruction API • ϑΥτάϥϝτϦʹΑΓࣸਅ͔Β3D ϞσϧΛ෮ݩ • WWDC21ͰmacOS޲͚APIͱͯ͠ొ ৔ɺWWDC23ͰiOS΋αϙʔτ
22. Object Capture APIΛ༻͍ͨϑΥτάϥϝτϦͷ࣮૷ let session = try PhotogrammetrySession(input: inputFolderUrl) try session.process(requests: [.modelFile(url: outputFile)]) • ࠷খ࣮૷͸ͨͬͨͷ2ߦʂ • PhotogrammetrySession ͷॳظԽ • process(requests:) ϝιουΛݺͿ
23. Reality Composer Pro • Object CaptureͷReconstruction API ΛΞϓϦԽͨ͠΋ͷ • Xcodeʹಉࠝ • ը૾ϑΥϧμΛυϥοάˍυϩοϓ → ϘλϯΆͪͰOK
24. ΞδΣϯμ • ϑΥτάϥϝτϦͱ͸ • ϑΥτάϥϝτϦͷ࣮૷ • ϑΥτάϥϝτϦ࣮ફฤ
25. ͜Ε Λ΍Γ͍ͨ
26. • ʮΦϒδΣΫτʯͰ͸ͳ͘ʮۭؒʯ ͷ3DϞσϧԽ • iPhoneͰࡱͬͨಈըΛݩʹ͍ͯ͠Δ Β͍͠ʢຊདྷ͸ߴՁͳϓϩ༻ࡱӨػ ࡐΛ࢖͏ʣ • 3DF ZephyrʢWindowsͷΈɾ༗ྉʣ ͱ͍͏ιϑτΛ࢖༻͍ͯ͠ΔΒ͍͠ → Object Capture Ͱಉ༷ͷϑΥτά ϥϝτϦ͸Մೳ͔ʁ
27. ϑΥτάϥϝτϦ with Object Captureͷ՝୊ • ಈը͔ΒͷೖྗΛαϙʔτ͍ͯ͠ͳ͍ • ʮΦϒδΣΫτʯΩϟϓνϟʔͱᨳ͓ͬͯΓɺʮۭؒʯ͸෮ ݩͰ͖ͳ͍Մೳੑ • 3DF Zephry, Reality Capture, Metashapeͱ͍ͬͨஶ໊ͳ༗ྉ ιϑτͱൺֱͯ͠ք۾Ͱͷར༻࣮੷͕΄΅ͳ͍
28. ಈը͔ΒͷϑΥτάϥϝτϦ Object Capture API΋Reality Composer Pro΋ಈըΛೖྗͱ͢Δ ͜ͱ͸Ͱ͖ͳ͍ → ಈըͷϑϨʔϜը૾Λநग़͢ΔΞϓϦΛॻ͍ͨ let generator = AVAssetImageGenerator(asset: asset) generator.generateCGImagesAsynchronously(forTimes: times) { requestedTime, cgImage, actualTime, result, error in ...
29. ࡱӨͨ͠ಈը • iPhoneยख࣋ͪͰࡱӨ • ௨ৗϞʔυ • ޿֯ʢ0.5ʣ
30. ϑϨʔϜநग़݁Ռ • Ұఆִ࣌ؒؒͰநग़
31. ϑΥτάϥϝτϦ࣮ߦ
33. ςΫενϟ͕Ԛ͍
34. ໰୊ɿ நग़ͨ͠ϑϨʔϜ͕ϒϨϒϨ • ςΫενϟͷ඼࣭͕Լ͕Δ • ϝογϡ඼࣭ʹ΋Өڹ͕ͰΔ
35. ϒϨͷগͳ͍ϑϨʔϜΛબΜͰநग़͢Δ ϥϓϥγΞϯϑΟϧλΛ༻͍ͨϒϨ౓߹͍ͷ൑ఆ 1. MPSImageLaplacian ͰϥϓϥγΞϯϑΟϧλΛը૾ʹద ༻͠ɺΤοδ෦෼ͷίϯτϥετΛڧௐ 2. MPSImageStatisticsMeanAndVariance Ͱը૾ͷ෼ࢄ Λܭࢉ • ϒϨ͍ͯΔը૾͸2ͷ஋͕௿͘ͳΔ 2 ͜ΕΛߟྀ͠ɺ୯७ͳ͖͍͠஋ॲཧͰͳ͘ɺલޙϑϨʔϜͷ஋ͱൺֱͯ͠࠾༻ෆ࠾༻Λܾఆ͢ΔϩδοΫʹͨ͠ɻ
36. ϒϨ౓߹͍൑ఆ݁Ռ ෼ࢄ஋ɿ 80ʢ௿͍ʣ
37. ϒϨ౓߹͍൑ఆ݁Ռ ෼ࢄ஋ɿ 365ʢߴ͍ʣ
38. ϒϨݕग़͸͏·͍ͬͯͦ͘͏
39. ϑΥτάϥϝτϦ࣮ߦ
40. ݁ՌʢσϞʣ
41. ςΫενϟ΋ʢ΍΍ʣ͖Ε͍ʹͳͬͨ ϝογϡͷ෮ݩ౓΋վળ
42. ݩಈըͷվળ
43. iPhoneͰϑΥτάϥϝτϦ༻ͷಈըΛࡱΔίπ • ԣ޲͖ • ޿֯ • ΞΫγϣϯϞʔυ • ໌Δ͞ͷ੍ݶ͕ݫ͍͠
44. ࡱӨͨ͠ಈը • iPhoneยख࣋ͪࡱӨ • ޿֯ • ԣ޲͖ • ΞΫγϣϯϞʔυON
45. ϑΥτάϥϝτϦ࣮ߦ
46. ݁ՌʢσϞʣ
48. • ⭕ น΍஍໘ͷ࠶ݱ౓͸͔ͳΓྑ͍ • ❌ ʮۭʯʹϝογϡ͕ੜ੒͞Εɺς Ϋενϟ͕షΒΕ͍ͯΔ
49. ʮۭʯͷ໰୊ ۭͷ෦෼ʹϝογϡ͕࡞ΒΕͳ͍Α͏ɺ আ֎͍ͨ͠
50. ϑΥτάϥϝτϦͱϚεΫ 3D෮ݩॲཧʹ࢖༻͠ͳ͍ϐΫηϧΛ ʮϚεΫը૾ʯΛ༩͑Δ͜ͱͰࢦఆͰ͖ Δ
51. Core MLΛར༻ͯ͠ʮۭϚεΫʯΛੜ੒ ηάϝϯςʔγϣϯ݁Ռ 3 ͔ΒۭϚεΫΛੜ੒ 3 Ϟσϧ͸ DANet Λར༻ɻʢϞσϧબఆͷৄࡉɿ Core ML ηάϝϯςʔγϣϯϞσϧͷൺֱ ʣ
52. PhotogrammetrySession ʹϚεΫΛ౉࣮͢૷ 1/2 PhotogrammetrySample Λར༻͢Δ // PhotogrammetrySampleͷॳظԽ let pixelBuffer = createPixelBuffer(from: inputImage) var sample = PhotogrammetrySample(id: index, image: pixelBuffer) // ϚεΫը૾ΛobjectMaskϓϩύςΟʹηοτ let maskPixelBuffer = createMaskPixelBuffer(from: maskImage) sample.objectMask = maskPixelBuffer
53. PhotogrammetrySession ʹϚεΫΛ౉࣮͢૷ 2/2 PhotogrammetrySample Λར༻͢Δ // PhotogrammetrySamples ͔Β PhotogrammetrySession ΛॳظԽ session = try PhotogrammetrySession(input: samples, configuration: configuration)
54. ϚεΫը૾ͷϑΥʔϚοτ • ݩը૾ͱಉ͡෯ɾߴ͞ • ϐΫηϧϑΥʔϚοτ͸ kCVPixelFormatType_OneComponent8
55. ϑΥτάϥϝτϦ݁ՌʢσϞʣ
56. ۭ෦෼ͷෆཁͳϝογϡ͸আڈͰ͖ͨ
57. ΞϓϦέʔγϣϯ΁ͷ૊ΈࠐΈ • 3DϞσϧͷதΛา͘Α͏ͳΞχϝʔγϣϯʢSceneKitʣ
58. ͞ΒʹվળͰ͖ΔϙΠϯτ • ۭϚεΫੜ੒ʹ͓͍ͯηάϝϯςʔ γϣϯޙʹmattingॲཧ 4 • ϑϨʔϜநग़ʹ͓͍ͯΧϝϥͷҠಈ ྔΛߟྀ͢Δ • Geometryͷࢦఆ 4 ը૾͸ https://twitter.com/nobelchoco/status/1137308632914255874 ΑΓɻ
59. ͦͷଞݕূͨ͜͠ͱʢQ&AͰ࿩͠·͠ΐ͏ʣ • Object Capture͸ը૾ຕ਺ΛͲ͜·ͰݮΒͤΔ͔ • Object CaptureͰࡱӨ͞ΕΔը૾ʹؚ·ΕΔ৘ใʢؔ࿈ه ࣄʣ • Object CaptureϑΥτάϥϝτϦͰ͸Ͳͷ৘ใΛԿʹ࢖͏͔ ʢؔ࿈هࣄʣ • MetashapeΛ࢖ͬͨϑΥτάϥϝτϦ • Metashape vs Object Capture
60. ʢ͓·͚ʣvisionOS΁ͷ૊ΈࠐΈ • USDZΛRealityKit Content PackageʹೖΕͯ Model3D ϏϡʔͰදࣔ͢Δ͚ͩʂ
61. ·ͱΊ • 3DϞσϧͷϢʔεέʔε͸ແݶେ • ࣸਅͷ୅ସʹͳΓಘΔ • Object CaptureΛ༻͍ͨϑΥτάϥϝτϦͷجૅͱ࣮ફ • ʮΦϒδΣΫτʯͷΩϟϓνϟʔੑೳ͸ߴ͍ • ʮۭؒʯ΋࣮͸͍͚ͦͦ͜͜Δ • ϒϨ౓߹͍ͷ൑ఆ/ࡱӨͷίπ/ۭϚεΫ
62. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ