Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Image Processing for iOS

Image Processing for iOS

An overview of three image processing frameworks for iOS: Core Image, Accelerate / vImage and Metal (including Metal Performance Shaders).

simon gladman

April 22, 2016
Tweet

More Decks by simon gladman

Other Decks in Programming

Transcript

  1. MOBILE IMAGE EDITING Image capture & manipulation a primary use

    case for iOS devices Apps from vendors such as Pixelmator & Adobe Desktop functionality and performance on iOS devices
  2. IMAGE PROCESSING FRAMEWORKS Three main frameworks: Core Image Accelerate /

    vImage Metal High performance & low energy Cross platform - both iOS and OS X
  3. CORE IMAGE Image processing framework Introduced to iOS in 2011

    with 16 built-in filters Over 170 built-in filters Support to write custom filter kernels Feature recognition for faces, rectangles, text, QR codes Near parity with OS X
  4. ACCELERATE / VIMAGE Accelerate is Apple’s vector processing framework for

    iOS SIMD: Single Instruction Multiple Data Includes support for digital signal processing, math and linear algebra and image processing (vImage) vImage introduced for iOS in 2011 Updated in 2013 with better Core Graphics interoperability Core Video support added in 2014
  5. METAL Apple’s framework for lowest overhead GPU access Introduced in

    2014 Compute kernels offer data-parallel computation C++ style language Precompiled shaders Metal Performance Shaders released in 2015 (iOS only)
  6. CORE IMAGE KERNEL A CIKernel contains the image processing algorithm

    that’s executed once for every pixel in a destination image
  7. CORE IMAGE FILTER A CIFilter wraps up one or more

    kernels into lightweight, mutable object that generates an output image Most accept a range of parameters Input image Numeric parameters
  8. CORE IMAGE IMAGE A CIImage contains the “recipe” to create

    a final image from one or more Core Image filters It’s only when a CIImage is converted to a renderable format that the filters are actually executed Core Image filters use CIImage instances as inputs and outputs
  9. CORE IMAGE CONTEXT A CIContext is the fundamental class for

    rendering Core Image generated content Represents the drawing destination - either CPU or GPU Expensive to instantiate
  10. FILTER CATEGORIES 173 filters in 21 filter categories Distortion Effect

    Geometry Adjustment Composite Operation Halftone Effect Color Adjustment Color Effect Transition Tile Effect Generator Reduction Gradient Stylize Sharpen Blur Video Still Image Interlaced Non Square Pixels
  11. FILTERING IN PRACTICE Convert a UIImage to a CIImage let

    bruges = UIImage( named: “bruges.jpg")! let image = CIImage( image: bruges)!
  12. FILTERING IN PRACTICE let noise = CIFilter(name: "CIRandomGenerator")?.outputImage? .imageByCroppingToRect(image.extent) let

    filteredImage = image .imageByApplyingFilter( "CIVignette", withInputParameters: [kCIInputIntensityKey: 4]) .imageByApplyingFilter( "CIDarkenBlendMode", withInputParameters: [kCIInputBackgroundImageKey: noise!]) .imageByApplyingFilter( "CIColorControls", withInputParameters: [kCIInputSaturationKey: 0.25, kCIInputContrastKey: 1.15]) .imageByApplyingFilter( "CISepiaTone", withInputParameters: nil)
  13. FILTERING IN PRACTICE Render output to CGImage and display with

    UIImageView let context = CIContext() let finalImage = context.createCGImage( filteredImage, fromRect: filteredImage.extent) imageView.image = UIImage( CGImage: finalImage)
  14. CUSTOM CORE IMAGE KERNELS Three types of kernel General Color

    - can only change color information Warp - can only change where the destination pixel is sampled from Written in Core Image Kernel Language A dialect of GLSL
  15. COLOR KERNEL let shadedTileKernel = CIColorKernel( string: "kernel vec4 shadedTile(__sample

    pixel)" + "{" + " vec2 coord = samplerCoord(pixel);" + " float brightness = mod(coord.y, 80.0) / 80.0;" + " brightness *= 1.0 - mod(coord.x, 80.0) / 80.0;" + " return vec4(sqrt(brightness) * pixel.rgb, pixel.a); " + "}")
  16. COLOR KERNEL Execute the kernel to create a CIImage let

    final = shadedTileKernel.applyWithExtent( extent, arguments: arguments)
  17. WARP KERNEL let carnivalMirrorKernel = CIWarpKernel(string: "kernel vec2 carnivalMirror(float xWavelength,

    float xAmount," + "float yWavelength, float yAmount)" + "{" + " float y = destCoord().y + sin(destCoord().y / yWavelength) * yAmount; " + " float x = destCoord().x + sin(destCoord().x / xWavelength) * xAmount; " + " return vec2(x, y); " + "}")
  18. WARP KERNEL Execute the kernel to create a CIImage let

    final = kernel.applyWithExtent( extent, roiCallback: { (index, rect) in return rect }, inputImage: inputImage, arguments: arguments)
  19. GENERAL KERNEL let maskedVariableBlur = CIKernel(string: "kernel vec4 lumaVariableBlur(sampler image,

    sampler blurImage, float blurRadius) " + "{ " + " vec2 d = destCoord(); " + " vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb; " + " float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722)); " + " float n = 0.0; " + " int radius = int(blurAmount * blurRadius); " + " vec3 accumulator = vec3(0.0, 0.0, 0.0); " + " for (int x = -radius; x <= radius; x++) " + " { " + " for (int y = -radius; y <= radius; y++) " + " { " + " vec2 workingSpaceCoordinate = d + vec2(x,y); " + " vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate); " + " vec3 color = sample(image, imageSpaceCoordinate).rgb; " + " accumulator += color; " + " n += 1.0; " + " } " + " } " + " accumulator /= n; " + " return vec4(accumulator, 1.0); " + "} " )
  20. VIMAGE BUFFERS Converting a UIImage to a vImage Buffer var

    format = vImage_CGImageFormat( bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.Last.rawValue), version: 0, decode: nil, renderingIntent: .RenderingIntentDefault) let cgImage = UIImage(named: "landscape.jpg")!.CGImage! var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, cgImage, UInt32(kvImageNoFlags))
  21. VIMAGE BUFFERS Converting a vImage buffer to a UIImage extension

    UIImage { convenience init?(fromvImageOutBuffer outBuffer:vImage_Buffer) { var mutableBuffer = outBuffer var error = vImage_Error() let cgImage = vImageCreateCGImageFromBuffer( &mutableBuffer, &format, nil, nil, UInt32(kvImageNoFlags), &error) self.init(CGImage: cgImage.takeRetainedValue()) } }
  22. VIMAGE HISTOGRAM Histograms contain the frequency of color values in

    an image Equalisation: increase global contrast by making an image’s histogram uniform Calculation: returns the histogram of an image. Specification: apply a histogram (e.g. from a source image) to a target image Contrast Stretch: Stretches the contrast range of a source image to the full range of supported values (i.e. normalises).
  23. VIMAGE EQUALIZATION let pixelBuffer = malloc( CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var

    outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) vImageEqualization_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer)
  24. VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage)

    -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }
  25. VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage)

    -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) }
  26. VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage,

    histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }
  27. VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage,

    histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! }
  28. let monalisa = UIImage(named: "monalisa.jpg")! let bluesky = UIImage(named: "bluesky.jpg")!

    let histogram = histogramCalculation(bluesky.CGImage!) let colored = histogramSpecification(monalisa.CGImage!, histogram: histogram) VIMAGE SPECIFICATION
  29. VIMAGE CONVOLUTION Convolution Different kernel for each color channel Deconvolution

    (Richardson-Lucy) More edging options than Core Image Supports larger kernels than Core Image
  30. DECONVOLUTION vImageRichardsonLucyDeConvolve_ARGB8888( &inBuffer, &outBuffer, nil, 0, 0, kernel, nil, kernelSide,

    kernelSide, 0, 0, divisor, 0, [0,0,0,0], iterationCount, UInt32(kvImageNoFlags))
  31. VIMAGE MORPHOLOGY Dilate & Erode With custom kernel Max &

    Min Fast versions of dilate & erode with rectangular kernels
  32. VIMAGE DILATION let kernel: [UInt8] = [ 255, 255, 255,

    255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255] vImageDilate_ARGB8888( &inBuffer, &outBuffer, 0, 0, kernel, UInt(kernelSide), UInt(kernelSide), UInt32(kvImageNoFlags))
  33. VIMAGE CONTINUED Conversions - lots! Floating point to integer color

    with dithering YUV to RGB Interleaved (RGB, RGB, RGB) to planar (RRR, GGG, BBB)
  34. METAL PERFORMANCE SHADERS Framework of data-parallel image processing algorithms for

    GPU Similar set of functionality to vImage Histogram functions including equalisation & specification Convolution, Gaussian blur Morphology: min, max, erode and dilate Resampling: scale and transform Thresholding & Integral
  35. MPS GAUSSIAN BLUR Create a Metal texture from a UIImage

    let sourceImage = UIImage(named: “telescope.jpg")! let imageTexture: MTLTexture = { let textureLoader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!) let imageTexture:MTLTexture do { imageTexture = try textureLoader.newTextureWithCGImage( sourceImage.CGImage!, options: nil) } catch { fatalError("unable to create texture from image") } return imageTexture }()
  36. MPS GAUSSIAN BLUR Textures based on UIImage are upside-down let

    intermediateTextureDesciptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat( MTLPixelFormat.RGBA8Unorm, width: imageTexture.width, height: imageTexture.height, mipmapped: false) let intermediateTexture = device.newTextureWithDescriptor(intermediateTextureDesciptor) let blur = MPSImageGaussianBlur( device: device, sigma: abs(sin(value)) * 200) let scale = MPSImageLanczosScale(device: device) var tx = MPSScaleTransform( scaleX: 1, scaleY: -1, translateX: 0, translateY: Double(-imageTexture.height)) withUnsafePointer(&tx) { scale.scaleTransform = $0 }
  37. MPS GAUSSIAN BLUR Encode both shaders to command buffer let

    imageView = MTKView() let commandQueue = device.newCommandQueue() let commandBuffer = commandQueue.commandBuffer() scale.encodeToCommandBuffer( commandBuffer, sourceTexture: imageTexture, destinationTexture: intermediateTexture) blur.encodeToCommandBuffer( commandBuffer, sourceTexture: intermediateTexture, destinationTexture: currentDrawable.texture) commandBuffer.presentDrawable(imageView.currentDrawable!) commandBuffer.commit();
  38. METAL COMPUTE SHADERS Lowest level - developer is responsible for

    setting up Metal classes: Device - interface to GPU Library - repository of functions Command Queue - queues & submits commands Function - a Metal shader Pipeline State - compiles function Command Buffer - stores encoded commands Command Encoder - encodes resources to byte code
  39. METAL COMPUTE SHADERS Advantages over a Core Image kernel Improved

    tooling Support for arrays Write to multiple pixels Write to multiple targets Disadvantages A lot more code! CIKL is automatically translated on the fly to Metal shader language
  40. METAL COMPUTE SHADERS kernel void pixellate( texture2d<float, access::read> inTexture [[texture(0)]],

    texture2d<float, access::write> outTexture [[texture(1)]], constant float &pixelWidth [[ buffer(0) ]], constant float &pixelHeight [[ buffer(1) ]], uint2 gid [[thread_position_in_grid]]) { uint width = uint(pixelWidth); uint height = uint(pixelHeight); const uint2 pixellatedGid = uint2( (gid.x / width) * width, (gid.y / height) * height); const float4 colorAtPixel = inTexture.read(pixellatedGid); outTexture.write(colorAtPixel, gid); }
  41. RESOURCES Twitter: @FlexMonkey Core Image for Swift on Apple’s iBooks

    Store & Gumroad https://github.com/FlexMonkey/ Filterpedia https://github.com/FlexMonkey/ ProgSConCompanion http://flexmonkey.blogspot.co.uk