Metal in Swift

Hardcore Basics of 3D Graphics on iOS

Transcript

1. Metal in Swift Hardcore Basics of 3D Graphics on iOS Warren Moore 4 Nov 2014 Realm
2. Slides and Code Code: tiny.cc/swiftmetal Slides: tiny.cc/swiftslides
3. Who am I? Formerly  Freelance iOS engineer Now writing metalbyexample.com
6. Disclaimers I'm new to Swift I can't possibly cover everything in <1 hour
7. Poll
8. Rendering Basics Part One
9. What is 3D Rendering?
10. What is 3D Rendering?
11. What is 3D Rendering?
12. What is 3D Rendering?
15. The "Fixed-Function" Pipeline Configurable but not programmable Often refers to consumer graphics hardware c. 2004 Used in contrast to "programmable" pipeline
16. The Programmable Pipeline Stages of the pipeline run small programs called shaders Grants enormous power and flexibility Requires greater responsibility from developers
17. The Pipeline: Triangles to Pixels draw call vertex shader rasterize fragment shader
18. Vertex Data
19. Vertex Data [x1 y1 z1 ] [r1 g1 b1 ] [x2 y2 z2 ] [r2 g2 b2 ] [x0 y0 z0 ] [r0 g0 b0 ]
20. Transformations Translate
21. Transformations Translate
22. Transformations Translate Rotate
23. Transformations Translate Rotate
24. Transformations Translate Rotate Scale
25. Transformations Translate Rotate Scale
26. The View Frustum
27. Perspective Projection
28. Moving Between Coordinate Spaces Points start out relative to the objects they belong to "Model space" We need all objects in a consistent coordinate space "World space" Situate everything relative to the virtual camera "Eye space Project all points onto the near plane "Screen space"
29. Matrix Concatenation v' = Mproj * Meye * Mworld * Mmodel * v
30. Scan-line Rasterization
31. The Pipeline: Triangles to Pixels
32. Working with Metal Part Two
33. The Context of Metal GPU SpriteKit SceneKit CoreAnimation CoreGraphics OpenGL ES Metal
34. Metal API Conventions Most objects are known only by protocol • Concrete types are hardware-dependent • More obvious in Objective-C than Swift Descriptors are bags of properties • Used to create immutable instances
35. Devices Abstraction of a GPU Conform to MTLDevice Used to create many other kinds of Metal objects
36. Creating a Device let device = MTLCreateSystemDefaultDevice()
37. Talking to UIKit let metalLayer = CAMetalLayer() metalLayer.device = device metalLayer.pixelFormat = .BGRA8Unorm
38. Drawables Vended by CAMetalLayer layer.nextDrawable() Vends a MTLTexture drawable.texture Abstracts the "swap chain" draw copy display
39. Render Pass Descriptors Attaches the current drawable texture to the framebuffer Describes how to treat framebuffer attachments • Clear, Store, Don't Care
40. Configuring a Render Pass let passDescriptor = MTLRenderPassDescriptor() passDescriptor.colorAttachments[0].texture = drawable.texture passDescriptor.colorAttachments[0].loadAction = .Clear passDescriptor.colorAttachments[0].storeAction = .Store passDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0.8, 0.0, 0.0, 1.0)
41. Command Submission Flow Command Queue Command Buffer Command Buffer Command Buffer Command Encoder Command Encoder Command Encoder GPU
42. Command Queues Conform to MTLCommandQueue protocol Used to submit encoded render instructions Command Queue Command Bu!er Command Bu!er Command Bu!er GPU
43. Creating a Command Queue commandQueue = device.newCommandQueue()
44. Command Encoders Command Bu!er Command Encoder Write commands into command buffers Various types encode different ops • Render, Compute, Blit Where "draw calls" happen
45. Creating a Render Command Encoder commandBuffer.renderCommandEncoderWithDescriptor(passDescriptor)!
46. Issuing Draw Calls // … fixed-function configuration … commandEncoder.drawPrimitives(.Triangle, vertexStart:0, vertexCount:3)
47. Presenting and Committing // … draw calls … commandEncoder.endEncoding() commandBuffer.presentDrawable(drawable) commandBuffer.commit()
48. Demo Clearing the Screen
49. The Render Pipeline in Metal A pre-compiled set of graphics state One pipeline per pair of vertex and fragment functions Pipelines are expensive to create but cheap to swap
50. Vertex Descriptors Describe how vertices are laid out in memory
51. Example Vertex Layout x y z w r g b a position color
52. Creating a Vertex Descriptor let vertexDescriptor = MTLVertexDescriptor() vertexDescriptor.attributes[0].offset = 0; vertexDescriptor.attributes[0].format = .Float4 vertexDescriptor.attributes[0].bufferIndex = 0 vertexDescriptor.attributes[1].offset = sizeof(Float32) * 4 vertexDescriptor.attributes[1].format = .Float4 vertexDescriptor.attributes[1].bufferIndex = 0 vertexDescriptor.layouts[0].stepFunction = .PerVertex vertexDescriptor.layouts[0].stride = sizeof(Float32) * 8
53. Creating a Pipeline Descriptor let pipelineDescriptor = MTLRenderPipelineDescriptor() pipelineDescriptor.vertexDescriptor = vertexDescriptor pipelineDescriptor.vertexFunction = vertexFunction pipelineDescriptor.fragmentFunction = fragmentFunction pipelineDescriptor.colorAttachments[0].pixelFormat = .BGRA8Unorm
54. Libraries and Functions A library is a collection of functions • The default library consists of all functions compiled into your app A function is a programmable piece of the pipeline • What we've been calling a "shader"
55. Creating Libraries and Functions let library = device.newDefaultLibrary()! let vertexFunction = library.newFunctionWithName("vertex_func") let fragmentFunction = library.newFunctionWithName("fragment_func")
56. A Simple Vertex Shader vertex OutVertex vertex_func(device InVertex *vert [[buffer(0)]], constant Uniforms &uniforms [[buffer(1)]], uint vid [[vertex_id]]) { OutVertex outVertex; outVertex.position = uniforms.rotation_matrix * vert[vid].position; outVertex.color = vert[vid].color; return outVertex; }
57. A Simple Fragment Shader fragment half4 fragment_func(OutVertex vert [[stage_in]]) { return half4(vert.color); }
58. Creating a Pipeline State pipeline = device.newRenderPipelineStateWithDescriptor(pipelineDescriptor, error:error)
59. Draw Call Redux commandEncoder.setRenderPipelineState(pipeline) commandEncoder.setVertexBuffer(vertexBuffer, offset:0, atIndex:0) commandEncoder.setVertexBuffer(uniformBuffer, offset:0, atIndex:1) commandEncoder.drawPrimitives(.Triangle, vertexStart:0, vertexCount:3) commandEncoder.endEncoding()
60. Demo Rendering and Animation in 2D
61. Moving to 3D Add a depth buffer Add a normal direction to each vertex Introduce a perspective projection matrix Introduce a fragment shader for lighting
62. The Depth Buffer Allows rendering triangles in any order Tracks the camera-relative depth of each fragment Depth ranges from 0..1 Cleared to the furthest distance before each frame
63. Generating Geometry We need more than a single triangle Let's generate a sphere! The SphereGenerator calculates: • Vertex position • Normal • Texture coordinates (for later)
64. Basics of Lighting Lighting is a complicated subject We'll approximate the most important term: diffuse Light that re-radiates from a surface in all directions Contrasts with specular, which reflects in a particular direction Dependent only on surface normal and light direction intensity = normal · lightDirection
65. A Vertex Shader for Lighting vertex OutVertex light_vertex(device InVertex *vert [[buffer(0)]], constant Uniforms &uniforms [[buffer(1)]], uint vid [[vertex_id]]) { OutVertex outVertex; outVertex.position = uniforms.projectionMatrix * uniforms.modelViewMatrix * vert[vid].position; outVertex.normal = uniforms.normalMatrix * vert[vid].normal; return outVertex; }
66. A Per-Pixel Lighting Shader fragment half4 light_fragment(OutVertex vert [[stage_in]]) { float intensity = saturate(dot(normalize(vert.normal), lightDirection)); return half4(intensity, intensity, intensity, 1); }
67. Demo Lighting and Rendering in 3D
68. Texturing Associate each vertex with a 2D texture coordinate Replaces the vertex color with a per-pixel diffuse color
69. Unwrapping a Cow
70. Loading Texture Data from a UIImage Leverage CG to draw a UIImage into a bitmap context Copy pixel data into MTLTexture
71. Loading Texture Data from a UIImage let textureDescriptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat(.RGBA8Unorm, width: Int(width), height: Int(height), mipmapped: true) let texture = device.newTextureWithDescriptor(textureDescriptor) let region = MTLRegionMake2D(0, 0, Int(width), Int(height)) texture.replaceRegion(region, mipmapLevel: 0, withBytes: rawData, bytesPerRow: Int(bytesPerRow))
72. Sampling Abstracts the notion of reading pixel data Can be configured to interpolate between "texels"
73. Creating a Sampler State let samplerDescriptor = MTLSamplerDescriptor() samplerDescriptor.minFilter = .Nearest samplerDescriptor.magFilter = .Linear samplerState = device.newSamplerStateWithDescriptor(samplerDescriptor)
74. A Texturing Fragment Shader fragment half4 tex_fragment(OutVertex vert [[stage_in]], texture2d<float> texture [[texture(0)]], sampler samp [[sampler(0)]]) { float4 diffuseColor = texture.sample(samp, vert.texCoords); return half4(diffuseColor.r, diffuseColor.g, diffuseColor.b, 1); }
76. Demo Texturing
77. Takeaways Slimmer and more consistent API than OpenGL Requires more effort than high-level libraries Provides unprecedented access to hardware
78. Resources / Questions? developer.apple.com • Metal Shading Language Guide • Metal Programming Guide • WWDC 2014 Videos raywenderlich.com • Metal Tutorial with Swift: Getting Started metalbyexample.com