Slide 1
Slide 1 text
Metal in Swift
Hardcore Basics of 3D Graphics on iOS
Warren Moore
4 Nov 2014
Realm
Slide 2
Slide 2 text
Slides and Code
Code: tiny.cc/swiftmetal
Slides: tiny.cc/swiftslides
Slide 3
Slide 3 text
Who am I?
Formerly
Freelance iOS engineer
Now writing metalbyexample.com
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Disclaimers
I'm new to Swift
I can't possibly cover everything in <1 hour
Slide 7
Slide 7 text
Poll
Slide 8
Slide 8 text
Rendering Basics
Part One
Slide 9
Slide 9 text
What is 3D Rendering?
Slide 10
Slide 10 text
What is 3D Rendering?
Slide 11
Slide 11 text
What is 3D Rendering?
Slide 12
Slide 12 text
What is 3D Rendering?
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
The "Fixed-Function" Pipeline
Configurable but not programmable
Often refers to consumer graphics hardware c. 2004
Used in contrast to "programmable" pipeline
Slide 16
Slide 16 text
The Programmable Pipeline
Stages of the pipeline run small programs called shaders
Grants enormous power and flexibility
Requires greater responsibility from developers
Slide 17
Slide 17 text
The Pipeline: Triangles to Pixels
draw call vertex
shader
rasterize fragment
shader
Slide 18
Slide 18 text
Vertex Data
Slide 19
Slide 19 text
Vertex Data
[x1
y1
z1
]
[r1
g1
b1
]
[x2
y2
z2
]
[r2
g2
b2
]
[x0
y0
z0
]
[r0
g0
b0
]
Slide 20
Slide 20 text
Transformations
Translate
Slide 21
Slide 21 text
Transformations
Translate
Slide 22
Slide 22 text
Transformations
Translate Rotate
Slide 23
Slide 23 text
Transformations
Translate Rotate
Slide 24
Slide 24 text
Transformations
Translate Rotate Scale
Slide 25
Slide 25 text
Transformations
Translate Rotate Scale
Slide 26
Slide 26 text
The View Frustum
Slide 27
Slide 27 text
Perspective Projection
Slide 28
Slide 28 text
Moving Between Coordinate Spaces
Points start out relative to the objects they belong to
"Model space"
We need all objects in a consistent coordinate space
"World space"
Situate everything relative to the virtual camera
"Eye space
Project all points onto the near plane
"Screen space"
Slide 29
Slide 29 text
Matrix Concatenation
v' = Mproj * Meye * Mworld * Mmodel * v
Slide 30
Slide 30 text
Scan-line Rasterization
Slide 31
Slide 31 text
The Pipeline: Triangles to Pixels
Slide 32
Slide 32 text
Working with Metal
Part Two
Slide 33
Slide 33 text
The Context of Metal
GPU
SpriteKit
SceneKit
CoreAnimation
CoreGraphics
OpenGL ES
Metal
Slide 34
Slide 34 text
Metal API Conventions
Most objects are known only by protocol
• Concrete types are hardware-dependent
• More obvious in Objective-C than Swift
Descriptors are bags of properties
• Used to create immutable instances
Slide 35
Slide 35 text
Devices
Abstraction of a GPU
Conform to MTLDevice
Used to create many other kinds of Metal objects
Slide 36
Slide 36 text
Creating a Device
let device = MTLCreateSystemDefaultDevice()
Slide 37
Slide 37 text
Talking to UIKit
let metalLayer = CAMetalLayer()
metalLayer.device = device
metalLayer.pixelFormat = .BGRA8Unorm
Slide 38
Slide 38 text
Drawables
Vended by CAMetalLayer
layer.nextDrawable()
Vends a MTLTexture
drawable.texture
Abstracts the "swap chain"
draw
copy
display
Slide 39
Slide 39 text
Render Pass Descriptors
Attaches the current drawable texture to the framebuffer
Describes how to treat framebuffer attachments
• Clear, Store, Don't Care
Slide 40
Slide 40 text
Configuring a Render Pass
let passDescriptor = MTLRenderPassDescriptor()
passDescriptor.colorAttachments[0].texture = drawable.texture
passDescriptor.colorAttachments[0].loadAction = .Clear
passDescriptor.colorAttachments[0].storeAction = .Store
passDescriptor.colorAttachments[0].clearColor =
MTLClearColorMake(0.8, 0.0, 0.0, 1.0)
Slide 41
Slide 41 text
Command Submission Flow
Command Queue
Command
Buffer
Command
Buffer
Command
Buffer
Command
Encoder
Command
Encoder
Command
Encoder
GPU
Slide 42
Slide 42 text
Command Queues
Conform to MTLCommandQueue protocol
Used to submit encoded render instructions
Command Queue
Command
Bu!er
Command
Bu!er
Command
Bu!er
GPU
Slide 43
Slide 43 text
Creating a Command Queue
commandQueue = device.newCommandQueue()
Slide 44
Slide 44 text
Command Encoders
Command
Bu!er
Command
Encoder
Write commands into command buffers
Various types encode different ops
• Render, Compute, Blit
Where "draw calls" happen
Slide 45
Slide 45 text
Creating a Render Command Encoder
commandBuffer.renderCommandEncoderWithDescriptor(passDescriptor)!
Slide 46
Slide 46 text
Issuing Draw Calls
// … fixed-function configuration …
commandEncoder.drawPrimitives(.Triangle, vertexStart:0, vertexCount:3)
Slide 47
Slide 47 text
Presenting and Committing
// … draw calls …
commandEncoder.endEncoding()
commandBuffer.presentDrawable(drawable)
commandBuffer.commit()
Slide 48
Slide 48 text
Demo
Clearing the Screen
Slide 49
Slide 49 text
The Render Pipeline in Metal
A pre-compiled set of graphics state
One pipeline per pair of vertex and fragment functions
Pipelines are expensive to create but cheap to swap
Slide 50
Slide 50 text
Vertex Descriptors
Describe how vertices are laid out in memory
Slide 51
Slide 51 text
Example Vertex Layout
x y z w r g b a
position color
Slide 52
Slide 52 text
Creating a Vertex Descriptor
let vertexDescriptor = MTLVertexDescriptor()
vertexDescriptor.attributes[0].offset = 0;
vertexDescriptor.attributes[0].format = .Float4
vertexDescriptor.attributes[0].bufferIndex = 0
vertexDescriptor.attributes[1].offset = sizeof(Float32) * 4
vertexDescriptor.attributes[1].format = .Float4
vertexDescriptor.attributes[1].bufferIndex = 0
vertexDescriptor.layouts[0].stepFunction = .PerVertex
vertexDescriptor.layouts[0].stride = sizeof(Float32) * 8
Slide 53
Slide 53 text
Creating a Pipeline Descriptor
let pipelineDescriptor = MTLRenderPipelineDescriptor()
pipelineDescriptor.vertexDescriptor = vertexDescriptor
pipelineDescriptor.vertexFunction = vertexFunction
pipelineDescriptor.fragmentFunction = fragmentFunction
pipelineDescriptor.colorAttachments[0].pixelFormat = .BGRA8Unorm
Slide 54
Slide 54 text
Libraries and Functions
A library is a collection of functions
• The default library consists of all functions compiled into your app
A function is a programmable piece of the pipeline
• What we've been calling a "shader"
Slide 55
Slide 55 text
Creating Libraries and Functions
let library = device.newDefaultLibrary()!
let vertexFunction = library.newFunctionWithName("vertex_func")
let fragmentFunction = library.newFunctionWithName("fragment_func")
Slide 56
Slide 56 text
A Simple Vertex Shader
vertex OutVertex vertex_func(device InVertex *vert [[buffer(0)]],
constant Uniforms &uniforms [[buffer(1)]],
uint vid [[vertex_id]])
{
OutVertex outVertex;
outVertex.position = uniforms.rotation_matrix * vert[vid].position;
outVertex.color = vert[vid].color;
return outVertex;
}
Slide 57
Slide 57 text
A Simple Fragment Shader
fragment half4 fragment_func(OutVertex vert [[stage_in]])
{
return half4(vert.color);
}
Slide 58
Slide 58 text
Creating a Pipeline State
pipeline = device.newRenderPipelineStateWithDescriptor(pipelineDescriptor,
error:error)
Slide 59
Slide 59 text
Draw Call Redux
commandEncoder.setRenderPipelineState(pipeline)
commandEncoder.setVertexBuffer(vertexBuffer, offset:0, atIndex:0)
commandEncoder.setVertexBuffer(uniformBuffer, offset:0, atIndex:1)
commandEncoder.drawPrimitives(.Triangle, vertexStart:0, vertexCount:3)
commandEncoder.endEncoding()
Slide 60
Slide 60 text
Demo
Rendering and Animation in 2D
Slide 61
Slide 61 text
Moving to 3D
Add a depth buffer
Add a normal direction to each vertex
Introduce a perspective projection matrix
Introduce a fragment shader for lighting
Slide 62
Slide 62 text
The Depth Buffer
Allows rendering triangles in any order
Tracks the camera-relative depth of each fragment
Depth ranges from 0..1
Cleared to the furthest distance before each frame
Slide 63
Slide 63 text
Generating Geometry
We need more than a single triangle
Let's generate a sphere!
The SphereGenerator calculates:
• Vertex position
• Normal
• Texture coordinates (for later)
Slide 64
Slide 64 text
Basics of Lighting
Lighting is a complicated subject
We'll approximate the most important term: diffuse
Light that re-radiates from a surface in all directions
Contrasts with specular, which reflects in a particular direction
Dependent only on surface normal and light direction
intensity = normal · lightDirection
Slide 65
Slide 65 text
A Vertex Shader for Lighting
vertex OutVertex light_vertex(device InVertex *vert [[buffer(0)]],
constant Uniforms &uniforms [[buffer(1)]],
uint vid [[vertex_id]])
{
OutVertex outVertex;
outVertex.position = uniforms.projectionMatrix *
uniforms.modelViewMatrix *
vert[vid].position;
outVertex.normal = uniforms.normalMatrix * vert[vid].normal;
return outVertex;
}
Slide 66
Slide 66 text
A Per-Pixel Lighting Shader
fragment half4 light_fragment(OutVertex vert [[stage_in]])
{
float intensity = saturate(dot(normalize(vert.normal), lightDirection));
return half4(intensity, intensity, intensity, 1);
}
Slide 67
Slide 67 text
Demo
Lighting and Rendering in 3D
Slide 68
Slide 68 text
Texturing
Associate each vertex with a 2D texture coordinate
Replaces the vertex color with a per-pixel diffuse color
Slide 69
Slide 69 text
Unwrapping a Cow
Slide 70
Slide 70 text
Loading Texture Data from a UIImage
Leverage CG to draw a UIImage into a bitmap context
Copy pixel data into MTLTexture
Slide 71
Slide 71 text
Loading Texture Data from a UIImage
let textureDescriptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat(.RGBA8Unorm,
width: Int(width),
height: Int(height),
mipmapped: true)
let texture = device.newTextureWithDescriptor(textureDescriptor)
let region = MTLRegionMake2D(0, 0, Int(width), Int(height))
texture.replaceRegion(region,
mipmapLevel: 0,
withBytes: rawData,
bytesPerRow: Int(bytesPerRow))
Slide 72
Slide 72 text
Sampling
Abstracts the notion of reading pixel data
Can be configured to interpolate between "texels"
Slide 73
Slide 73 text
Creating a Sampler State
let samplerDescriptor = MTLSamplerDescriptor()
samplerDescriptor.minFilter = .Nearest
samplerDescriptor.magFilter = .Linear
samplerState = device.newSamplerStateWithDescriptor(samplerDescriptor)
Slide 74
Slide 74 text
A Texturing Fragment Shader
fragment half4 tex_fragment(OutVertex vert [[stage_in]],
texture2d texture [[texture(0)]],
sampler samp [[sampler(0)]])
{
float4 diffuseColor = texture.sample(samp, vert.texCoords);
return half4(diffuseColor.r, diffuseColor.g, diffuseColor.b, 1);
}
Slide 75
Slide 75 text
No content
Slide 76
Slide 76 text
Demo
Texturing
Slide 77
Slide 77 text
Takeaways
Slimmer and more consistent API than OpenGL
Requires more effort than high-level libraries
Provides unprecedented access to hardware
Slide 78
Slide 78 text
Resources / Questions?
developer.apple.com
• Metal Shading Language Guide
• Metal Programming Guide
• WWDC 2014 Videos
raywenderlich.com
• Metal Tutorial with Swift: Getting Started
metalbyexample.com