飛び道具ではないMetal #iOSDC

iOSDC 2017での登壇資料です。

[概要]

WWDC17でもMetalのセッション会場は閑古鳥が鳴いてました。Metalって3Dゲームとかカメラアプリとかのトリッキーなエフェクト用でしょ、うちには関係ないなぁ・・・と思われている方も多いかもしれません。しかし、UIKitの下回りもMetalですし、色んな所で実はMetalは暗躍しています。そんなMetalにもっと多くの開発者に興味を持ってもらうため、非ゲーム・非カメラアプリ開発者にも関係しそうな切り口からMetalやGPUについて紹介してみたいと思います。

Transcript

1. iOSDC 2017 Shuichi Tsutsumi @shu223 ඈͼಓ۩Ͱ͸ͳ͍
3. WWDC 2014
4. ͦ͏ʮΞϨʯ͕ൃද͞Εͨ೥
7. SwiftͱMetal͸ಉ͍೥!
8. WWDC2014ͰͷMetalͷσϞ
9. WWDC2014ͰͷSwiftൃද
10. ! !
11. Metalʢ࠶ʣ
12. " "
13. ʰMetalɾɾɾ͏ͪʹ͸ؔ܎ͳ͍ͳ͊ʱ
14. γΣʔμͰ͜Μͳʮදݱʯ΋Ͱ͖·͢ʂ .FUBMγΣʔμ͜ͱ͸͡Ί
    
    8FC(-(-4-ͷ๛෋ͳαϯϓϧΛࢀߟʹ͢Δ
    
    2JJUB ※ಈը΍ը૾Ͱ͸ͳ͘ɺ͢΂ͯϓϩάϥϜͰੜ੒͍ͯ͠·͢
15. ʢઈରϓϩμΫτͰ͸΍Βͳ͍ɾɾɾʣ
16. ✋
18. Metal GPU • Core ML • Vision • Core Image • SceneKit • SpriteKit • MapKit
19. Metal GPU • Core ML • Vision • Core Image • SceneKit • SpriteKit • MapKit • Core Animation • UIKit
20. 143,000 MetalͷAPIΛ௚઀ར༻͍ͯ͠ΔΞϓϦͷ਺
21. 143,000 17,000,000 MetalͷAPIΛ௚઀ར༻͍ͯ͠ΔΞϓϦͷ਺ MetalͷAPIΛ҉໧తʹར༻͍ͯ͠ΔΞϓϦͷ਺
22. ͋ͳͨͷΞϓϦͷԼͰ΋ Metal͕҉༂͍ͯ͠Δʂ
23. ඈͼಓ۩Ͱ͸ͳ͍.FUBM
24. ఻͍͑ͨ͜ͱ ✓Metal͸ؔ܎ͳ͘ͳ͍ ✓Metalͷجૅ ✓Metalͷ࣮૷ํ๏
25. imageView.image = image
27. Կ͕ى͖͍ͯΔͷ͔ʁ
28. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ
29. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ ը໘෼ͷ
    ϐΫηϧσʔλ
30. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ ը໘෼ͷ
    ϐΫηϧσʔλ ϐΫηϧσʔλ
 දࣔ
31. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ ը໘෼ͷ
    ϐΫηϧσʔλ ඵؒʹճ
    ॻ͖ࠐΉ ϐΫηϧσʔλ
 දࣔ
32. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ ը໘෼ͷ
    ϐΫηϧσʔλ ඵؒʹճ
    ॻ͖ࠐΉ ϐΫηϧσʔλ
 දࣔ ղ૾౓͕ߴ͘ͳΔ΄Ͳେม
33. CPUͱGPUͷੑ࣭ͷҧ͍ CPU͸εϙʔπΧʔ • ͱʹ͔͘଎͍ • Ұ౓ʹଟ͘ͷਓΛӡ΂ͳ͍ GPU͸όε • ಉ͡໨త஍ͷਓΛҰ౓ʹଟ͘ӡ΂Δ • ଎౓ࣗମ͸CPUʹ͸ٴ͹ͳ͍
34. • CPU͸଎ͯ͘ɺԿͰ΋Ͱ͖Δʢ൚༻ϓϩηοαʣ - ͔ͱ͍ͬͯԿͰ΋΍ΒͤΔͱɺ͙͢ʹՔಇ཰100%ʹ • GPU͕ಘҙͳ࢓ࣄʢʹฒྻԋࢉʣ͸GPUʹ΍ΒͤΔ - ಉ͡ܭࢉΛߦ͏ॲཧ - ௖఺ͷܭࢉʢ࠲ඪม׵ʣ - ϥΠςΟϯά - ͦͷଞϐΫηϧ͝ͱͷॲཧ
35. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ 1ඵؒʹ60ճ ॻ͖ࠐΉ • CPU͸৭Μͳ͜ͱͰ๩͍͠ • ϐΫηϧ͝ͱͷܭࢉɺϑϨʔϜόοϑΝ ΁ͷॻ͖ࠐΈ͸ฒྻॲཧOK ղ૾౓͕ߴ͘ͳΔ΄Ͳେม 1ը໘෼ͷϐΫηϧσʔλ
36. σΟεϓϨΠ ϓϩηοα ϑϨʔϜ
 όοϑΝ 1ඵؒʹ60ճ ॻ͖ࠐΉ • CPU͸৭Μͳ͜ͱͰ๩͍͠ • ϐΫηϧ͝ͱͷܭࢉɺϑϨʔϜόοϑΝ ΁ͷॻ͖ࠐΈ͸ฒྻॲཧOK GPU ղ૾౓͕ߴ͘ͳΔ΄Ͳେม 1ը໘෼ͷϐΫηϧσʔλ
37. మଇ: GPU͕ಘҙͳ࢓ࣄ͸GPUʹ
38. GPUΛૢΔ
40. GPU΁ͷΞΫηεΛఏڙ GPU Your app ???
41. OpenGLͱͷҧ͍
42. OpenGL • Ҏલ͔Β͋ΔάϥϑΟοΫεAPI • ֤ࣾͷϋʔυ΢ΣΞɾϓϥοτϑΥʔϜʹରԠ • Appleͷϋʔυ΢ΣΞ͚ͩʹ࠷దԽ͞Ε͍ͯΔΘ͚Ͱ͸ ͳ͍
43. Metal • Apple͕ಠࣗʹࡦఆ • Appleͷϋʔυ΢ΣΞʹ࠷దԽ • OpenGLΑΓ࠷େ10ഒߴ଎ • ࠓ͸OpenGL࣮૷ͷԼճΓ΋Metal
44. ͷ࣮૷
45. imageView.image = image
47. Metal APIΛʮ໌ࣔతʹʯୟ͍ͯ ը૾Λඳը͢Δ৔߹ͷ࠷খ࣮૷
48. func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalError()} 
 guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fatalError()} let targetW = min(texture.width, drawable.texture.width) let targetH = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(targetW, targetH, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } private let device = MTLCreateSystemDefaultDevice()! 
 private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture(name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat }
49. ʮ࠷খݶʯͷ࣮૷Ͱొ৔͢ΔΫϥε • MTLDevice • MTLCommandQueue • MTLCommandBuffer • MTLCommandEncoder • MTLTexture • MTKView • MTKTextureLoader • etc…
51. $ ͔͕ͨը૾Λදࣔ͢Δ͙Β͍Ͱͳͥ͜Μͳʹɻɻ
52. ཧղͷखҾ͖
54. $16͸ɺ(16΁ͷ໋ྩΛ
 ίϚϯυόοϑΝͱ͍͏ܗͰ࡞੒
55. $16͸ɺ(16΁ͷ໋ྩΛ
 ίϚϯυόοϑΝͱ͍͏ܗͰ࡞੒ (16ʹ౉͢
56. $16͸ɺ(16΁ͷ໋ྩΛ
 ίϚϯυόοϑΝͱ͍͏ܗͰ࡞੒ (16͸౉͞ΕͨίϚ ϯυΛ࣮ߦ͢Δ (16ʹ౉͢
57. $16͸ɺ(16΁ͷ໋ྩΛ
 ίϚϯυόοϑΝͱ͍͏ܗͰ࡞੒ $16ͱ(16͸྆ํ͔ΒΞΫη εͰ͖ΔϝϞϦྖҬΛ࣋ͭ (16͸౉͞ΕͨίϚ ϯυΛ࣮ߦ͢Δ (16ʹ౉͢
58. MTLCommandBuffer MTLTexture MTLDevice
59. MTLCommandBuffer MTLTexture MTLDevice MTLCommandEncoder
60. MTLCommandBuffer MTLTexture MTLDevice MTLCommandQueue MTLCommandEncoder
61. MTLCommandBuffer MTLTexture MTLDevice MTLCommandQueue MTKTextureLoader MTLCommandEncoder
62. ۩ମతʹݟ͍͖ͯ·͠ΐ͏
63. MTLDevice let device = MTLCreateSystemDefaultDevice()! ※Τϥʔॲཧ౳͸লུ͍ͯ͠·͢
64. MTLTexture, MTKTextureLoader // ϩʔμʔΛॳظԽ loader = MTKTextureLoader(device: device) // ςΫενϟΛΞηοτΧλϩά͔Βڞ༗ϝϞϦྖҬʹϩʔυ texture = try! loader.newTexture(
 name: “hoge", 
 scaleFactor: view.contentScaleFactor, 
 bundle: nil) ※Τϥʔॲཧ౳͸লུ͍ͯ͠·͢
65. MTLCommandQueue, MTLCommandBuffer // ίϚϯυΩϡʔ͔ΒίϚϯυόοϑΝΛ࡞੒ let commandBuffer = commandQueue.makeCommandBuffer()! // … ίϚϯυόοϑΝʹίϚϯυ௥Ճ // ίϚϯυόοϑΝΛίϛοτʢʹΤϯΩϡʔ͞ΕΔʣ commandBuffer.commit() ※Τϥʔॲཧ౳͸লུ͍ͯ͠·͢ // ίϚϯυΩϡʔΛॳظԽ commandQueue = device.makeCommandQueue()
66. MTLCommandEncoder • ίϚϯυόοϑΝʹίϚϯυΛ௥ՃʢΤϯίʔυʣ͢Δ MTLRenderCommandEncoder άϥϑΟοΫεϨϯμϦϯάίϚϯυΛΤϯίʔυ͢Δ MTLComputeCommandEncoder ฒྻԋࢉॲཧΛΤϯίʔυ͢Δ MTLBlitCommandEncode όοϑΝɾςΫενϟؒͷίϐʔॲཧΛΤϯίʔυ͢Δ
67. MTLCommandEncoder (MTLBlitCommandEncoder) let blitEncoder = commandBuffer.makeBlitCommandEncoder()! blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: srcOrigin, sourceSize: srcSize, to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin:dstOrigin) blitEncoder.endEncoding() ※Τϥʔॲཧ౳͸লུ͍ͯ͠·͢
68. MTLCommandEncoder (MTLBlitCommandEncoder) let blitEncoder = commandBuffer.makeBlitCommandEncoder()! blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: srcOrigin, sourceSize: srcSize, to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin:dstOrigin) blitEncoder.endEncoding() ※Τϥʔॲཧ౳͸লུ͍ͯ͠·͢ ίϐʔݩ ίϐʔઌ
69. // ίϚϯυΩϡʔ͔ΒίϚϯυόοϑΝΛ࡞੒ let commandBuffer = commandQueue.makeCommandBuffer()! // ίϚϯυΤϯίʔμΛ࡞੒ let blitEncoder = commandBuffer.makeBlitCommandEncoder()! blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: srcOrigin, sourceSize: srcSize, to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin:dstOrigin) // ίϚϯυ௥Ճ׬ྃ blitEncoder.endEncoding() // ίϚϯυόοϑΝΛίϛοτʢʹΤϯΩϡʔ͞ΕΔʣ commandBuffer.commit() ※Τϥʔॲཧ౳͸লུ͍ͯ͠·͢
70. private let device = MTLCreateSystemDefaultDevice()! private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture(name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat } func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalError()} 
 guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fatalError()} let targetW = min(texture.width, drawable.texture.width) let targetH = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(targetW, targetH, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() }
71. private let device = MTLCreateSystemDefaultDevice()! private func setup() { commandQueue = device.makeCommandQueue() let textureLoader = MTKTextureLoader(device: device) texture = try! textureLoader.newTexture(name: "highsierra", scaleFactor: view.contentScaleFactor, bundle: nil) mtkView.device = device mtkView.delegate = self mtkView.colorPixelFormat = texture.pixelFormat } ίϚϯυόοϑΝ࡞੒ → ΤϯΩϡʔ →ʢGPU΁ૹ৴ʣ →ʢGPUͰίϚϯυ࣮ߦʣ →ʢඳըʣ func draw(in view: MTKView) { guard let drawable = view.currentDrawable else {return} guard let commandBuffer = commandQueue.makeCommandBuffer() else {fatalError()} 
 guard let blitEncoder = commandBuffer.makeBlitCommandEncoder() else {fatalError()} let targetW = min(texture.width, drawable.texture.width) let targetH = min(texture.height, drawable.texture.height) blitEncoder.copy(from: texture, sourceSlice: 0, sourceLevel: 0, sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0), sourceSize: MTLSizeMake(targetW, targetH, texture.depth), to: drawable.texture, destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0)) blitEncoder.endEncoding() commandBuffer.present(drawable) commandBuffer.commit() commandBuffer.waitUntilCompleted() } ɾɾɾίϚϯυΩϡʔੜ੒ ςΫενϟΛϩʔυ ϏϡʔͷηοτΞοϓ ɾɾɾσόΠεੜ੒
72. % എܠʹ͋Δ֓೦Λ౿·͑Δͱҙ֎ͱ؆୯
73. Powered by
74. GPUΛૢΕΔΑ͏ʹʂ GPU Your app ???
75. ૿͑ଓ͚ΔMetal׆༻ͷ৔ • ARKitͷΧελϜϨϯμϦϯά • MPS CNN / RNNΛ༻͍ͨػցֶश • Core ImageͷΧελϜϑΟϧλ
76. ࢀߟॻ੶ GPU Metal
77. ·ͱΊ • Metal͸ؔ܎ͳ͘ͳ͍ - UIKitͷԼճΓͱͯ͠΋҉༂ • GPU͕ಘҙͳ࢓ࣄ͸GPUʹ೚ͤΑ͏ • Metalͷجૅ - Metalͱ͸ / OpenGLͱͷൺֱ • Metalͷ࣮૷ - എܠʹ͋Δ֓೦Λཧղ͢Δͱ؆୯
78. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ https://github.com/shu223