IMAGE PROCESSING FOR IOS Simon Gladman for ProgSCon April 2016

MOBILE IMAGE EDITING Image capture & manipulation a primary use case for iOS devices Apps from vendors such as Pixelmator & Adobe Desktop functionality and performance on iOS devices

IMAGE PROCESSING FRAMEWORKS Three main frameworks: Core Image Accelerate / vImage Metal High performance & low energy Cross platform - both iOS and OS X

CORE IMAGE Image processing framework Introduced to iOS in 2011 with 16 built-in filters Over 170 built-in filters Support to write custom filter kernels Feature recognition for faces, rectangles, text, QR codes Near parity with OS X

ACCELERATE / VIMAGE Accelerate is Apple’s vector processing framework for iOS SIMD: Single Instruction Multiple Data Includes support for digital signal processing, math and linear algebra and image processing (vImage) vImage introduced for iOS in 2011 Updated in 2013 with better Core Graphics interoperability Core Video support added in 2014

METAL Apple’s framework for lowest overhead GPU access Introduced in 2014 Compute kernels offer data-parallel computation C++ style language Precompiled shaders Metal Performance Shaders released in 2015 (iOS only)

COMPANION PROJECT https://github.com/FlexMonkey/ ProgSConCompanion

CORE IMAGE EXPLORED

CORE IMAGE KERNEL A CIKernel contains the image processing algorithm that’s executed once for every pixel in a destination image

CORE IMAGE FILTER A CIFilter wraps up one or more kernels into lightweight, mutable object that generates an output image Most accept a range of parameters Input image Numeric parameters

CORE IMAGE IMAGE A CIImage contains the “recipe” to create a final image from one or more Core Image filters It’s only when a CIImage is converted to a renderable format that the filters are actually executed Core Image filters use CIImage instances as inputs and outputs

CORE IMAGE CONTEXT A CIContext is the fundamental class for rendering Core Image generated content Represents the drawing destination - either CPU or GPU Expensive to instantiate

FILTER CATEGORIES 173 filters in 21 filter categories Distortion Effect Geometry Adjustment Composite Operation Halftone Effect Color Adjustment Color Effect Transition Tile Effect Generator Reduction Gradient Stylize Sharpen Blur Video Still Image Interlaced Non Square Pixels

BLUR For example Gaussian Blur Box & Tent Zoom & Motion Blur

COLOR ADJUSTMENT For example Hue adjustment Gamma & Exposure Tone Curves Color Temperature

COLOR EFFECTS For example False Color Photo Effects Posterisation Color Cube & Polynomial

DISTORTION For example Droste Pinch & Bump Torus Lens Twirl & Vortex

GENERATOR For example Starshine & Lenticular Halo Stripes & Checkerboards Solid Color Random Noise

STYLIZE For example Convolution Depth of Field Pixellate & Crystalize

FILTERING IN PRACTICE Convert a UIImage to a CIImage let bruges = UIImage( named: “bruges.jpg")! let image = CIImage( image: bruges)!

FILTERING IN PRACTICE let noise = CIFilter(name: "CIRandomGenerator")?.outputImage? .imageByCroppingToRect(image.extent) let filteredImage = image .imageByApplyingFilter( "CIVignette", withInputParameters: [kCIInputIntensityKey: 4]) .imageByApplyingFilter( "CIDarkenBlendMode", withInputParameters: [kCIInputBackgroundImageKey: noise!]) .imageByApplyingFilter( "CIColorControls", withInputParameters: [kCIInputSaturationKey: 0.25, kCIInputContrastKey: 1.15]) .imageByApplyingFilter( "CISepiaTone", withInputParameters: nil)

FILTERING IN PRACTICE Render output to CGImage and display with UIImageView let context = CIContext() let finalImage = context.createCGImage( filteredImage, fromRect: filteredImage.extent) imageView.image = UIImage( CGImage: finalImage)

KERNEL CONCATENATION Noise Vignette Darken Blend Sepia Color Controls

KERNEL CONCATENATION Noise Vignette Darken Blend Sepia Single Kernel

CUSTOM CORE IMAGE KERNELS Three types of kernel General Color - can only change color information Warp - can only change where the destination pixel is sampled from Written in Core Image Kernel Language A dialect of GLSL

COLOR KERNEL let shadedTileKernel = CIColorKernel( string: "kernel vec4 shadedTile(__sample pixel)" + "{" + " vec2 coord = samplerCoord(pixel);" + " float brightness = mod(coord.y, 80.0) / 80.0;" + " brightness *= 1.0 - mod(coord.x, 80.0) / 80.0;" + " return vec4(sqrt(brightness) * pixel.rgb, pixel.a); " + "}")

COLOR KERNEL Execute the kernel to create a CIImage let final = shadedTileKernel.applyWithExtent( extent, arguments: arguments)

WARP KERNEL let carnivalMirrorKernel = CIWarpKernel(string: "kernel vec2 carnivalMirror(float xWavelength, float xAmount," + "float yWavelength, float yAmount)" + "{" + " float y = destCoord().y + sin(destCoord().y / yWavelength) * yAmount; " + " float x = destCoord().x + sin(destCoord().x / xWavelength) * xAmount; " + " return vec2(x, y); " + "}")

WARP KERNEL Execute the kernel to create a CIImage let final = kernel.applyWithExtent( extent, roiCallback: { (index, rect) in return rect }, inputImage: inputImage, arguments: arguments)

GENERAL KERNEL let maskedVariableBlur = CIKernel(string: "kernel vec4 lumaVariableBlur(sampler image, sampler blurImage, float blurRadius) " + "{ " + " vec2 d = destCoord(); " + " vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb; " + " float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722)); " + " float n = 0.0; " + " int radius = int(blurAmount * blurRadius); " + " vec3 accumulator = vec3(0.0, 0.0, 0.0); " + " for (int x = -radius; x <= radius; x++) " + " { " + " for (int y = -radius; y <= radius; y++) " + " { " + " vec2 workingSpaceCoordinate = d + vec2(x,y); " + " vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate); " + " vec3 color = sample(image, imageSpaceCoordinate).rgb; " + " accumulator += color; " + " n += 1.0; " + " } " + " } " + " accumulator /= n; " + " return vec4(accumulator, 1.0); " + "} " )

GENERAL KERNEL • applyWithExtent() has same signature as a warp kernel

CUSTOM KERNELS

CUSTOM KERNELS

CUSTOM KERNELS

CUSTOM KERNELS

VIMAGE EXPLORED

VIMAGE BUFFERS Converting a UIImage to a vImage Buffer var format = vImage_CGImageFormat( bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.Last.rawValue), version: 0, decode: nil, renderingIntent: .RenderingIntentDefault) let cgImage = UIImage(named: "landscape.jpg")!.CGImage! var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, cgImage, UInt32(kvImageNoFlags))

VIMAGE BUFFERS Converting a vImage buffer to a UIImage extension UIImage { convenience init?(fromvImageOutBuffer outBuffer:vImage_Buffer) { var mutableBuffer = outBuffer var error = vImage_Error() let cgImage = vImageCreateCGImageFromBuffer( &mutableBuffer, &format, nil, nil, UInt32(kvImageNoFlags), &error) self.init(CGImage: cgImage.takeRetainedValue()) } }

VIMAGE HISTOGRAM Histograms contain the frequency of color values in an image Equalisation: increase global contrast by making an image’s histogram uniform Calculation: returns the histogram of an image. Specification: apply a histogram (e.g. from a source image) to a target image Contrast Stretch: Stretches the contrast range of a source image to the full range of supported values (i.e. normalises).

VIMAGE EQUALIZATION let pixelBuffer = malloc( CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) vImageEqualization_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer)

VIMAGE EQUALIZATION

VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage) -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer(alpha) let redPtr = UnsafeMutablePointer(red) let greenPtr = UnsafeMutablePointer(green) let bluePtr = UnsafeMutablePointer(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }

VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage) -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer(alpha) let redPtr = UnsafeMutablePointer(red) let greenPtr = UnsafeMutablePointer(green) let bluePtr = UnsafeMutablePointer(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }

VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage, histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer(histogram.alpha) let redPtr = UnsafePointer(histogram.red) let greenPtr = UnsafePointer(histogram.green) let bluePtr = UnsafePointer(histogram.blue) let rgba = UnsafeMutablePointer>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }

VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage, histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer(histogram.alpha) let redPtr = UnsafePointer(histogram.red) let greenPtr = UnsafePointer(histogram.green) let bluePtr = UnsafePointer(histogram.blue) let rgba = UnsafeMutablePointer>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }

let monalisa = UIImage(named: "monalisa.jpg")! let bluesky = UIImage(named: "bluesky.jpg")! let histogram = histogramCalculation(bluesky.CGImage!) let colored = histogramSpecification(monalisa.CGImage!, histogram: histogram) VIMAGE SPECIFICATION

VIMAGE CONTRAST STRETCH vImageContrastStretch_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags))

VIMAGE CONTRAST STRETCH

VIMAGE CONVOLUTION Convolution Different kernel for each color channel Deconvolution (Richardson-Lucy) More edging options than Core Image Supports larger kernels than Core Image

DECONVOLUTION vImageRichardsonLucyDeConvolve_ARGB8888( &inBuffer, &outBuffer, nil, 0, 0, kernel, nil, kernelSide, kernelSide, 0, 0, divisor, 0, [0,0,0,0], iterationCount, UInt32(kvImageNoFlags))

DECONVOLUTION Source image blurred

DECONVOLUTION Destination image deconvolved using Richardson-Lucy

VIMAGE MORPHOLOGY Dilate & Erode With custom kernel Max & Min Fast versions of dilate & erode with rectangular kernels

VIMAGE DILATION let kernel: [UInt8] = [ 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255] vImageDilate_ARGB8888( &inBuffer, &outBuffer, 0, 0, kernel, UInt(kernelSide), UInt(kernelSide), UInt32(kvImageNoFlags))

DILATION Original Image

DILATION Dilation applied to thresholded version and composited over original

VIMAGE CONTINUED Conversions - lots! Floating point to integer color with dithering YUV to RGB Interleaved (RGB, RGB, RGB) to planar (RRR, GGG, BBB)

VIMAGE CONTINUED Geometry - high quality transforms Scale Rotate Shear

VIMAGE CONTINUED Transform Gamma Hue, saturation, brightness Color matrix Color polynomial

METAL EXPLORED

METAL PERFORMANCE SHADERS Framework of data-parallel image processing algorithms for GPU Similar set of functionality to vImage Histogram functions including equalisation & specification Convolution, Gaussian blur Morphology: min, max, erode and dilate Resampling: scale and transform Thresholding & Integral

MPS GAUSSIAN BLUR Different implementations of Gaussian blur optimised for Image size Blur radius Blur sigma

MPS GAUSSIAN BLUR Create a Metal texture from a UIImage let sourceImage = UIImage(named: “telescope.jpg")! let imageTexture: MTLTexture = { let textureLoader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!) let imageTexture:MTLTexture do { imageTexture = try textureLoader.newTextureWithCGImage( sourceImage.CGImage!, options: nil) } catch { fatalError("unable to create texture from image") } return imageTexture }()

MPS GAUSSIAN BLUR Textures based on UIImage are upside-down let intermediateTextureDesciptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat( MTLPixelFormat.RGBA8Unorm, width: imageTexture.width, height: imageTexture.height, mipmapped: false) let intermediateTexture = device.newTextureWithDescriptor(intermediateTextureDesciptor) let blur = MPSImageGaussianBlur( device: device, sigma: abs(sin(value)) * 200) let scale = MPSImageLanczosScale(device: device) var tx = MPSScaleTransform( scaleX: 1, scaleY: -1, translateX: 0, translateY: Double(-imageTexture.height)) withUnsafePointer(&tx) { scale.scaleTransform = $0 }

MPS GAUSSIAN BLUR Encode both shaders to command buffer let imageView = MTKView() let commandQueue = device.newCommandQueue() let commandBuffer = commandQueue.commandBuffer() scale.encodeToCommandBuffer( commandBuffer, sourceTexture: imageTexture, destinationTexture: intermediateTexture) blur.encodeToCommandBuffer( commandBuffer, sourceTexture: intermediateTexture, destinationTexture: currentDrawable.texture) commandBuffer.presentDrawable(imageView.currentDrawable!) commandBuffer.commit();

MPS GAUSSIAN BLUR

METAL COMPUTE SHADERS Lowest level - developer is responsible for setting up Metal classes: Device - interface to GPU Library - repository of functions Command Queue - queues & submits commands Function - a Metal shader Pipeline State - compiles function Command Buffer - stores encoded commands Command Encoder - encodes resources to byte code

METAL COMPUTE SHADERS Advantages over a Core Image kernel Improved tooling Support for arrays Write to multiple pixels Write to multiple targets Disadvantages A lot more code! CIKL is automatically translated on the fly to Metal shader language

METAL COMPUTE SHADERS kernel void pixellate( texture2d inTexture [[texture(0)]], texture2d outTexture [[texture(1)]], constant float &pixelWidth [[ buffer(0) ]], constant float &pixelHeight [[ buffer(1) ]], uint2 gid [[thread_position_in_grid]]) { uint width = uint(pixelWidth); uint height = uint(pixelHeight); const uint2 pixellatedGid = uint2( (gid.x / width) * width, (gid.y / height) * height); const float4 colorAtPixel = inTexture.read(pixellatedGid); outTexture.write(colorAtPixel, gid); }

METAL PIXELLATE FILTER

RESOURCES Twitter: @FlexMonkey Core Image for Swift on Apple’s iBooks Store & Gumroad https://github.com/FlexMonkey/ Filterpedia https://github.com/FlexMonkey/ ProgSConCompanion http://flexmonkey.blogspot.co.uk