Image Processing for iOS

An overview of three image processing frameworks for iOS: Core Image, Accelerate / vImage and Metal (including Metal Performance Shaders).

Transcript

1. IMAGE PROCESSING FOR IOS Simon Gladman for ProgSCon April 2016
2. MOBILE IMAGE EDITING Image capture & manipulation a primary use case for iOS devices Apps from vendors such as Pixelmator & Adobe Desktop functionality and performance on iOS devices
3. IMAGE PROCESSING FRAMEWORKS Three main frameworks: Core Image Accelerate / vImage Metal High performance & low energy Cross platform - both iOS and OS X
4. CORE IMAGE Image processing framework Introduced to iOS in 2011 with 16 built-in filters Over 170 built-in filters Support to write custom filter kernels Feature recognition for faces, rectangles, text, QR codes Near parity with OS X
5. ACCELERATE / VIMAGE Accelerate is Apple’s vector processing framework for iOS SIMD: Single Instruction Multiple Data Includes support for digital signal processing, math and linear algebra and image processing (vImage) vImage introduced for iOS in 2011 Updated in 2013 with better Core Graphics interoperability Core Video support added in 2014
6. METAL Apple’s framework for lowest overhead GPU access Introduced in 2014 Compute kernels offer data-parallel computation C++ style language Precompiled shaders Metal Performance Shaders released in 2015 (iOS only)
7. COMPANION PROJECT https://github.com/FlexMonkey/ ProgSConCompanion
8. CORE IMAGE EXPLORED
9. CORE IMAGE KERNEL A CIKernel contains the image processing algorithm that’s executed once for every pixel in a destination image
10. CORE IMAGE FILTER A CIFilter wraps up one or more kernels into lightweight, mutable object that generates an output image Most accept a range of parameters Input image Numeric parameters
11. CORE IMAGE IMAGE A CIImage contains the “recipe” to create a final image from one or more Core Image filters It’s only when a CIImage is converted to a renderable format that the filters are actually executed Core Image filters use CIImage instances as inputs and outputs
12. CORE IMAGE CONTEXT A CIContext is the fundamental class for rendering Core Image generated content Represents the drawing destination - either CPU or GPU Expensive to instantiate
13. FILTER CATEGORIES 173 filters in 21 filter categories Distortion Effect Geometry Adjustment Composite Operation Halftone Effect Color Adjustment Color Effect Transition Tile Effect Generator Reduction Gradient Stylize Sharpen Blur Video Still Image Interlaced Non Square Pixels
14. BLUR For example Gaussian Blur Box & Tent Zoom & Motion Blur
15. COLOR ADJUSTMENT For example Hue adjustment Gamma & Exposure Tone Curves Color Temperature
16. COLOR EFFECTS For example False Color Photo Effects Posterisation Color Cube & Polynomial
17. DISTORTION For example Droste Pinch & Bump Torus Lens Twirl & Vortex
18. GENERATOR For example Starshine & Lenticular Halo Stripes & Checkerboards Solid Color Random Noise
19. STYLIZE For example Convolution Depth of Field Pixellate & Crystalize
20. FILTERING IN PRACTICE Convert a UIImage to a CIImage let bruges = UIImage( named: “bruges.jpg")! let image = CIImage( image: bruges)!
21. FILTERING IN PRACTICE let noise = CIFilter(name: "CIRandomGenerator")?.outputImage? .imageByCroppingToRect(image.extent) let filteredImage = image .imageByApplyingFilter( "CIVignette", withInputParameters: [kCIInputIntensityKey: 4]) .imageByApplyingFilter( "CIDarkenBlendMode", withInputParameters: [kCIInputBackgroundImageKey: noise!]) .imageByApplyingFilter( "CIColorControls", withInputParameters: [kCIInputSaturationKey: 0.25, kCIInputContrastKey: 1.15]) .imageByApplyingFilter( "CISepiaTone", withInputParameters: nil)
22. FILTERING IN PRACTICE Render output to CGImage and display with UIImageView let context = CIContext() let finalImage = context.createCGImage( filteredImage, fromRect: filteredImage.extent) imageView.image = UIImage( CGImage: finalImage)
23. KERNEL CONCATENATION Noise Vignette Darken Blend Sepia Color Controls
24. KERNEL CONCATENATION Noise Vignette Darken Blend Sepia Single Kernel
25. CUSTOM CORE IMAGE KERNELS Three types of kernel General Color - can only change color information Warp - can only change where the destination pixel is sampled from Written in Core Image Kernel Language A dialect of GLSL
26. COLOR KERNEL let shadedTileKernel = CIColorKernel( string: "kernel vec4 shadedTile(__sample pixel)" + "{" + " vec2 coord = samplerCoord(pixel);" + " float brightness = mod(coord.y, 80.0) / 80.0;" + " brightness *= 1.0 - mod(coord.x, 80.0) / 80.0;" + " return vec4(sqrt(brightness) * pixel.rgb, pixel.a); " + "}")
27. COLOR KERNEL Execute the kernel to create a CIImage let final = shadedTileKernel.applyWithExtent( extent, arguments: arguments)
28. WARP KERNEL let carnivalMirrorKernel = CIWarpKernel(string: "kernel vec2 carnivalMirror(float xWavelength, float xAmount," + "float yWavelength, float yAmount)" + "{" + " float y = destCoord().y + sin(destCoord().y / yWavelength) * yAmount; " + " float x = destCoord().x + sin(destCoord().x / xWavelength) * xAmount; " + " return vec2(x, y); " + "}")
29. WARP KERNEL Execute the kernel to create a CIImage let final = kernel.applyWithExtent( extent, roiCallback: { (index, rect) in return rect }, inputImage: inputImage, arguments: arguments)
30. GENERAL KERNEL let maskedVariableBlur = CIKernel(string: "kernel vec4 lumaVariableBlur(sampler image, sampler blurImage, float blurRadius) " + "{ " + " vec2 d = destCoord(); " + " vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb; " + " float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722)); " + " float n = 0.0; " + " int radius = int(blurAmount * blurRadius); " + " vec3 accumulator = vec3(0.0, 0.0, 0.0); " + " for (int x = -radius; x <= radius; x++) " + " { " + " for (int y = -radius; y <= radius; y++) " + " { " + " vec2 workingSpaceCoordinate = d + vec2(x,y); " + " vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate); " + " vec3 color = sample(image, imageSpaceCoordinate).rgb; " + " accumulator += color; " + " n += 1.0; " + " } " + " } " + " accumulator /= n; " + " return vec4(accumulator, 1.0); " + "} " )
31. GENERAL KERNEL • applyWithExtent() has same signature as a warp kernel
32. CUSTOM KERNELS
33. CUSTOM KERNELS
34. CUSTOM KERNELS
35. CUSTOM KERNELS
36. VIMAGE EXPLORED
37. VIMAGE BUFFERS Converting a UIImage to a vImage Buffer var format = vImage_CGImageFormat( bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.Last.rawValue), version: 0, decode: nil, renderingIntent: .RenderingIntentDefault) let cgImage = UIImage(named: "landscape.jpg")!.CGImage! var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, cgImage, UInt32(kvImageNoFlags))
38. VIMAGE BUFFERS Converting a vImage buffer to a UIImage extension UIImage { convenience init?(fromvImageOutBuffer outBuffer:vImage_Buffer) { var mutableBuffer = outBuffer var error = vImage_Error() let cgImage = vImageCreateCGImageFromBuffer( &mutableBuffer, &format, nil, nil, UInt32(kvImageNoFlags), &error) self.init(CGImage: cgImage.takeRetainedValue()) } }
39. VIMAGE HISTOGRAM Histograms contain the frequency of color values in an image Equalisation: increase global contrast by making an image’s histogram uniform Calculation: returns the histogram of an image. Specification: apply a histogram (e.g. from a source image) to a target image Contrast Stretch: Stretches the contrast range of a source image to the full range of supported values (i.e. normalises).
40. VIMAGE EQUALIZATION let pixelBuffer = malloc( CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) vImageEqualization_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer)
41. VIMAGE EQUALIZATION
42. VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage) -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }
43. VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage) -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }
44. VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage, histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }
45. VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage, histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }
46. let monalisa = UIImage(named: "monalisa.jpg")! let bluesky = UIImage(named: "bluesky.jpg")! let histogram = histogramCalculation(bluesky.CGImage!) let colored = histogramSpecification(monalisa.CGImage!, histogram: histogram) VIMAGE SPECIFICATION
47. VIMAGE CONTRAST STRETCH vImageContrastStretch_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags))
48. VIMAGE CONTRAST STRETCH
49. VIMAGE CONVOLUTION Convolution Different kernel for each color channel Deconvolution (Richardson-Lucy) More edging options than Core Image Supports larger kernels than Core Image
50. DECONVOLUTION vImageRichardsonLucyDeConvolve_ARGB8888( &inBuffer, &outBuffer, nil, 0, 0, kernel, nil, kernelSide, kernelSide, 0, 0, divisor, 0, [0,0,0,0], iterationCount, UInt32(kvImageNoFlags))
51. DECONVOLUTION Source image blurred
52. DECONVOLUTION Destination image deconvolved using Richardson-Lucy
53. VIMAGE MORPHOLOGY Dilate & Erode With custom kernel Max & Min Fast versions of dilate & erode with rectangular kernels
54. VIMAGE DILATION let kernel: [UInt8] = [ 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255] vImageDilate_ARGB8888( &inBuffer, &outBuffer, 0, 0, kernel, UInt(kernelSide), UInt(kernelSide), UInt32(kvImageNoFlags))
55. DILATION Original Image
56. DILATION Dilation applied to thresholded version and composited over original
57. VIMAGE CONTINUED Conversions - lots! Floating point to integer color with dithering YUV to RGB Interleaved (RGB, RGB, RGB) to planar (RRR, GGG, BBB)
58. VIMAGE CONTINUED Geometry - high quality transforms Scale Rotate Shear
59. VIMAGE CONTINUED Transform Gamma Hue, saturation, brightness Color matrix Color polynomial
60. METAL EXPLORED
61. METAL PERFORMANCE SHADERS Framework of data-parallel image processing algorithms for GPU Similar set of functionality to vImage Histogram functions including equalisation & specification Convolution, Gaussian blur Morphology: min, max, erode and dilate Resampling: scale and transform Thresholding & Integral
62. MPS GAUSSIAN BLUR Different implementations of Gaussian blur optimised for Image size Blur radius Blur sigma
63. MPS GAUSSIAN BLUR Create a Metal texture from a UIImage let sourceImage = UIImage(named: “telescope.jpg")! let imageTexture: MTLTexture = { let textureLoader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!) let imageTexture:MTLTexture do { imageTexture = try textureLoader.newTextureWithCGImage( sourceImage.CGImage!, options: nil) } catch { fatalError("unable to create texture from image") } return imageTexture }()
64. MPS GAUSSIAN BLUR Textures based on UIImage are upside-down let intermediateTextureDesciptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat( MTLPixelFormat.RGBA8Unorm, width: imageTexture.width, height: imageTexture.height, mipmapped: false) let intermediateTexture = device.newTextureWithDescriptor(intermediateTextureDesciptor) let blur = MPSImageGaussianBlur( device: device, sigma: abs(sin(value)) * 200) let scale = MPSImageLanczosScale(device: device) var tx = MPSScaleTransform( scaleX: 1, scaleY: -1, translateX: 0, translateY: Double(-imageTexture.height)) withUnsafePointer(&tx) { scale.scaleTransform = $0 }
65. MPS GAUSSIAN BLUR Encode both shaders to command buffer let imageView = MTKView() let commandQueue = device.newCommandQueue() let commandBuffer = commandQueue.commandBuffer() scale.encodeToCommandBuffer( commandBuffer, sourceTexture: imageTexture, destinationTexture: intermediateTexture) blur.encodeToCommandBuffer( commandBuffer, sourceTexture: intermediateTexture, destinationTexture: currentDrawable.texture) commandBuffer.presentDrawable(imageView.currentDrawable!) commandBuffer.commit();
66. MPS GAUSSIAN BLUR
67. METAL COMPUTE SHADERS Lowest level - developer is responsible for setting up Metal classes: Device - interface to GPU Library - repository of functions Command Queue - queues & submits commands Function - a Metal shader Pipeline State - compiles function Command Buffer - stores encoded commands Command Encoder - encodes resources to byte code
68. METAL COMPUTE SHADERS Advantages over a Core Image kernel Improved tooling Support for arrays Write to multiple pixels Write to multiple targets Disadvantages A lot more code! CIKL is automatically translated on the fly to Metal shader language
69. METAL COMPUTE SHADERS kernel void pixellate( texture2d<float, access::read> inTexture [[texture(0)]], texture2d<float, access::write> outTexture [[texture(1)]], constant float &pixelWidth [[ buffer(0) ]], constant float &pixelHeight [[ buffer(1) ]], uint2 gid [[thread_position_in_grid]]) { uint width = uint(pixelWidth); uint height = uint(pixelHeight); const uint2 pixellatedGid = uint2( (gid.x / width) * width, (gid.y / height) * height); const float4 colorAtPixel = inTexture.read(pixellatedGid); outTexture.write(colorAtPixel, gid); }
70. METAL PIXELLATE FILTER
71. RESOURCES Twitter: @FlexMonkey Core Image for Swift on Apple’s iBooks Store & Gumroad https://github.com/FlexMonkey/ Filterpedia https://github.com/FlexMonkey/ ProgSConCompanion http://flexmonkey.blogspot.co.uk