Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Image Processing for iOS

Image Processing for iOS

An overview of three image processing frameworks for iOS: Core Image, Accelerate / vImage and Metal (including Metal Performance Shaders).

Cf40be92f231a15e81512b93a3cd0136?s=128

simon gladman

April 22, 2016
Tweet

Transcript

  1. IMAGE PROCESSING FOR IOS Simon Gladman for ProgSCon April 2016

  2. MOBILE IMAGE EDITING Image capture & manipulation a primary use

    case for iOS devices Apps from vendors such as Pixelmator & Adobe Desktop functionality and performance on iOS devices
  3. IMAGE PROCESSING FRAMEWORKS Three main frameworks: Core Image Accelerate /

    vImage Metal High performance & low energy Cross platform - both iOS and OS X
  4. CORE IMAGE Image processing framework Introduced to iOS in 2011

    with 16 built-in filters Over 170 built-in filters Support to write custom filter kernels Feature recognition for faces, rectangles, text, QR codes Near parity with OS X
  5. ACCELERATE / VIMAGE Accelerate is Apple’s vector processing framework for

    iOS SIMD: Single Instruction Multiple Data Includes support for digital signal processing, math and linear algebra and image processing (vImage) vImage introduced for iOS in 2011 Updated in 2013 with better Core Graphics interoperability Core Video support added in 2014
  6. METAL Apple’s framework for lowest overhead GPU access Introduced in

    2014 Compute kernels offer data-parallel computation C++ style language Precompiled shaders Metal Performance Shaders released in 2015 (iOS only)
  7. COMPANION PROJECT https://github.com/FlexMonkey/ ProgSConCompanion

  8. CORE IMAGE EXPLORED

  9. CORE IMAGE KERNEL A CIKernel contains the image processing algorithm

    that’s executed once for every pixel in a destination image
  10. CORE IMAGE FILTER A CIFilter wraps up one or more

    kernels into lightweight, mutable object that generates an output image Most accept a range of parameters Input image Numeric parameters
  11. CORE IMAGE IMAGE A CIImage contains the “recipe” to create

    a final image from one or more Core Image filters It’s only when a CIImage is converted to a renderable format that the filters are actually executed Core Image filters use CIImage instances as inputs and outputs
  12. CORE IMAGE CONTEXT A CIContext is the fundamental class for

    rendering Core Image generated content Represents the drawing destination - either CPU or GPU Expensive to instantiate
  13. FILTER CATEGORIES 173 filters in 21 filter categories Distortion Effect

    Geometry Adjustment Composite Operation Halftone Effect Color Adjustment Color Effect Transition Tile Effect Generator Reduction Gradient Stylize Sharpen Blur Video Still Image Interlaced Non Square Pixels
  14. BLUR For example Gaussian Blur Box & Tent Zoom &

    Motion Blur
  15. COLOR ADJUSTMENT For example Hue adjustment Gamma & Exposure Tone

    Curves Color Temperature
  16. COLOR EFFECTS For example False Color Photo Effects Posterisation Color

    Cube & Polynomial
  17. DISTORTION For example Droste Pinch & Bump Torus Lens Twirl

    & Vortex
  18. GENERATOR For example Starshine & Lenticular Halo Stripes & Checkerboards

    Solid Color Random Noise
  19. STYLIZE For example Convolution Depth of Field Pixellate & Crystalize

  20. FILTERING IN PRACTICE Convert a UIImage to a CIImage let

    bruges = UIImage( named: “bruges.jpg")! let image = CIImage( image: bruges)!
  21. FILTERING IN PRACTICE let noise = CIFilter(name: "CIRandomGenerator")?.outputImage? .imageByCroppingToRect(image.extent) let

    filteredImage = image .imageByApplyingFilter( "CIVignette", withInputParameters: [kCIInputIntensityKey: 4]) .imageByApplyingFilter( "CIDarkenBlendMode", withInputParameters: [kCIInputBackgroundImageKey: noise!]) .imageByApplyingFilter( "CIColorControls", withInputParameters: [kCIInputSaturationKey: 0.25, kCIInputContrastKey: 1.15]) .imageByApplyingFilter( "CISepiaTone", withInputParameters: nil)
  22. FILTERING IN PRACTICE Render output to CGImage and display with

    UIImageView let context = CIContext() let finalImage = context.createCGImage( filteredImage, fromRect: filteredImage.extent) imageView.image = UIImage( CGImage: finalImage)
  23. KERNEL CONCATENATION Noise Vignette Darken Blend Sepia Color Controls

  24. KERNEL CONCATENATION Noise Vignette Darken Blend Sepia Single Kernel

  25. CUSTOM CORE IMAGE KERNELS Three types of kernel General Color

    - can only change color information Warp - can only change where the destination pixel is sampled from Written in Core Image Kernel Language A dialect of GLSL
  26. COLOR KERNEL let shadedTileKernel = CIColorKernel( string: "kernel vec4 shadedTile(__sample

    pixel)" + "{" + " vec2 coord = samplerCoord(pixel);" + " float brightness = mod(coord.y, 80.0) / 80.0;" + " brightness *= 1.0 - mod(coord.x, 80.0) / 80.0;" + " return vec4(sqrt(brightness) * pixel.rgb, pixel.a); " + "}")
  27. COLOR KERNEL Execute the kernel to create a CIImage let

    final = shadedTileKernel.applyWithExtent( extent, arguments: arguments)
  28. WARP KERNEL let carnivalMirrorKernel = CIWarpKernel(string: "kernel vec2 carnivalMirror(float xWavelength,

    float xAmount," + "float yWavelength, float yAmount)" + "{" + " float y = destCoord().y + sin(destCoord().y / yWavelength) * yAmount; " + " float x = destCoord().x + sin(destCoord().x / xWavelength) * xAmount; " + " return vec2(x, y); " + "}")
  29. WARP KERNEL Execute the kernel to create a CIImage let

    final = kernel.applyWithExtent( extent, roiCallback: { (index, rect) in return rect }, inputImage: inputImage, arguments: arguments)
  30. GENERAL KERNEL let maskedVariableBlur = CIKernel(string: "kernel vec4 lumaVariableBlur(sampler image,

    sampler blurImage, float blurRadius) " + "{ " + " vec2 d = destCoord(); " + " vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb; " + " float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722)); " + " float n = 0.0; " + " int radius = int(blurAmount * blurRadius); " + " vec3 accumulator = vec3(0.0, 0.0, 0.0); " + " for (int x = -radius; x <= radius; x++) " + " { " + " for (int y = -radius; y <= radius; y++) " + " { " + " vec2 workingSpaceCoordinate = d + vec2(x,y); " + " vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate); " + " vec3 color = sample(image, imageSpaceCoordinate).rgb; " + " accumulator += color; " + " n += 1.0; " + " } " + " } " + " accumulator /= n; " + " return vec4(accumulator, 1.0); " + "} " )
  31. GENERAL KERNEL • applyWithExtent() has same signature as a warp

    kernel
  32. CUSTOM KERNELS

  33. CUSTOM KERNELS

  34. CUSTOM KERNELS

  35. CUSTOM KERNELS

  36. VIMAGE EXPLORED

  37. VIMAGE BUFFERS Converting a UIImage to a vImage Buffer var

    format = vImage_CGImageFormat( bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.Last.rawValue), version: 0, decode: nil, renderingIntent: .RenderingIntentDefault) let cgImage = UIImage(named: "landscape.jpg")!.CGImage! var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, cgImage, UInt32(kvImageNoFlags))
  38. VIMAGE BUFFERS Converting a vImage buffer to a UIImage extension

    UIImage { convenience init?(fromvImageOutBuffer outBuffer:vImage_Buffer) { var mutableBuffer = outBuffer var error = vImage_Error() let cgImage = vImageCreateCGImageFromBuffer( &mutableBuffer, &format, nil, nil, UInt32(kvImageNoFlags), &error) self.init(CGImage: cgImage.takeRetainedValue()) } }
  39. VIMAGE HISTOGRAM Histograms contain the frequency of color values in

    an image Equalisation: increase global contrast by making an image’s histogram uniform Calculation: returns the histogram of an image. Specification: apply a histogram (e.g. from a source image) to a target image Contrast Stretch: Stretches the contrast range of a source image to the full range of supported values (i.e. normalises).
  40. VIMAGE EQUALIZATION let pixelBuffer = malloc( CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var

    outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) vImageEqualization_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer)
  41. VIMAGE EQUALIZATION

  42. VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage)

    -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }
  43. VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage)

    -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }
  44. VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage,

    histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }
  45. VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage,

    histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }
  46. let monalisa = UIImage(named: "monalisa.jpg")! let bluesky = UIImage(named: "bluesky.jpg")!

    let histogram = histogramCalculation(bluesky.CGImage!) let colored = histogramSpecification(monalisa.CGImage!, histogram: histogram) VIMAGE SPECIFICATION
  47. VIMAGE CONTRAST STRETCH vImageContrastStretch_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags))

  48. VIMAGE CONTRAST STRETCH

  49. VIMAGE CONVOLUTION Convolution Different kernel for each color channel Deconvolution

    (Richardson-Lucy) More edging options than Core Image Supports larger kernels than Core Image
  50. DECONVOLUTION vImageRichardsonLucyDeConvolve_ARGB8888( &inBuffer, &outBuffer, nil, 0, 0, kernel, nil, kernelSide,

    kernelSide, 0, 0, divisor, 0, [0,0,0,0], iterationCount, UInt32(kvImageNoFlags))
  51. DECONVOLUTION Source image blurred

  52. DECONVOLUTION Destination image deconvolved using Richardson-Lucy

  53. VIMAGE MORPHOLOGY Dilate & Erode With custom kernel Max &

    Min Fast versions of dilate & erode with rectangular kernels
  54. VIMAGE DILATION let kernel: [UInt8] = [ 255, 255, 255,

    255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255] vImageDilate_ARGB8888( &inBuffer, &outBuffer, 0, 0, kernel, UInt(kernelSide), UInt(kernelSide), UInt32(kvImageNoFlags))
  55. DILATION Original Image

  56. DILATION Dilation applied to thresholded version and composited over original

  57. VIMAGE CONTINUED Conversions - lots! Floating point to integer color

    with dithering YUV to RGB Interleaved (RGB, RGB, RGB) to planar (RRR, GGG, BBB)
  58. VIMAGE CONTINUED Geometry - high quality transforms Scale Rotate Shear

  59. VIMAGE CONTINUED Transform Gamma Hue, saturation, brightness Color matrix Color

    polynomial
  60. METAL EXPLORED

  61. METAL PERFORMANCE SHADERS Framework of data-parallel image processing algorithms for

    GPU Similar set of functionality to vImage Histogram functions including equalisation & specification Convolution, Gaussian blur Morphology: min, max, erode and dilate Resampling: scale and transform Thresholding & Integral
  62. MPS GAUSSIAN BLUR Different implementations of Gaussian blur optimised for

    Image size Blur radius Blur sigma
  63. MPS GAUSSIAN BLUR Create a Metal texture from a UIImage

    let sourceImage = UIImage(named: “telescope.jpg")! let imageTexture: MTLTexture = { let textureLoader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!) let imageTexture:MTLTexture do { imageTexture = try textureLoader.newTextureWithCGImage( sourceImage.CGImage!, options: nil) } catch { fatalError("unable to create texture from image") } return imageTexture }()
  64. MPS GAUSSIAN BLUR Textures based on UIImage are upside-down let

    intermediateTextureDesciptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat( MTLPixelFormat.RGBA8Unorm, width: imageTexture.width, height: imageTexture.height, mipmapped: false) let intermediateTexture = device.newTextureWithDescriptor(intermediateTextureDesciptor) let blur = MPSImageGaussianBlur( device: device, sigma: abs(sin(value)) * 200) let scale = MPSImageLanczosScale(device: device) var tx = MPSScaleTransform( scaleX: 1, scaleY: -1, translateX: 0, translateY: Double(-imageTexture.height)) withUnsafePointer(&tx) { scale.scaleTransform = $0 }
  65. MPS GAUSSIAN BLUR Encode both shaders to command buffer let

    imageView = MTKView() let commandQueue = device.newCommandQueue() let commandBuffer = commandQueue.commandBuffer() scale.encodeToCommandBuffer( commandBuffer, sourceTexture: imageTexture, destinationTexture: intermediateTexture) blur.encodeToCommandBuffer( commandBuffer, sourceTexture: intermediateTexture, destinationTexture: currentDrawable.texture) commandBuffer.presentDrawable(imageView.currentDrawable!) commandBuffer.commit();
  66. MPS GAUSSIAN BLUR

  67. METAL COMPUTE SHADERS Lowest level - developer is responsible for

    setting up Metal classes: Device - interface to GPU Library - repository of functions Command Queue - queues & submits commands Function - a Metal shader Pipeline State - compiles function Command Buffer - stores encoded commands Command Encoder - encodes resources to byte code
  68. METAL COMPUTE SHADERS Advantages over a Core Image kernel Improved

    tooling Support for arrays Write to multiple pixels Write to multiple targets Disadvantages A lot more code! CIKL is automatically translated on the fly to Metal shader language
  69. METAL COMPUTE SHADERS kernel void pixellate( texture2d<float, access::read> inTexture [[texture(0)]],

    texture2d<float, access::write> outTexture [[texture(1)]], constant float &pixelWidth [[ buffer(0) ]], constant float &pixelHeight [[ buffer(1) ]], uint2 gid [[thread_position_in_grid]]) { uint width = uint(pixelWidth); uint height = uint(pixelHeight); const uint2 pixellatedGid = uint2( (gid.x / width) * width, (gid.y / height) * height); const float4 colorAtPixel = inTexture.read(pixellatedGid); outTexture.write(colorAtPixel, gid); }
  70. METAL PIXELLATE FILTER

  71. RESOURCES Twitter: @FlexMonkey Core Image for Swift on Apple’s iBooks

    Store & Gumroad https://github.com/FlexMonkey/ Filterpedia https://github.com/FlexMonkey/ ProgSConCompanion http://flexmonkey.blogspot.co.uk