Share
Embed
Image Processing for iOS
An overview of three image processing frameworks for iOS: Core Image, Accelerate / vImage and Metal (including Metal Performance Shaders).
simon gladman
April 22, 2016
Transcript
71 slides
-
MOBILE IMAGE EDITING Image capture & manipulation a primary use
case for iOS devices Apps from vendors such as Pixelmator & Adobe Desktop functionality and performance on iOS devices -
IMAGE PROCESSING FRAMEWORKS Three main frameworks: Core Image Accelerate /
vImage Metal High performance & low energy Cross platform - both iOS and OS X -
CORE IMAGE Image processing framework Introduced to iOS in 2011
with 16 built-in filters Over 170 built-in filters Support to write custom filter kernels Feature recognition for faces, rectangles, text, QR codes Near parity with OS X -
ACCELERATE / VIMAGE Accelerate is Apple’s vector processing framework for
iOS SIMD: Single Instruction Multiple Data Includes support for digital signal processing, math and linear algebra and image processing (vImage) vImage introduced for iOS in 2011 Updated in 2013 with better Core Graphics interoperability Core Video support added in 2014 -
METAL Apple’s framework for lowest overhead GPU access Introduced in
2014 Compute kernels offer data-parallel computation C++ style language Precompiled shaders Metal Performance Shaders released in 2015 (iOS only) -
CORE IMAGE KERNEL A CIKernel contains the image processing algorithm
that’s executed once for every pixel in a destination image -
CORE IMAGE FILTER A CIFilter wraps up one or more
kernels into lightweight, mutable object that generates an output image Most accept a range of parameters Input image Numeric parameters -
CORE IMAGE IMAGE A CIImage contains the “recipe” to create
a final image from one or more Core Image filters It’s only when a CIImage is converted to a renderable format that the filters are actually executed Core Image filters use CIImage instances as inputs and outputs -
CORE IMAGE CONTEXT A CIContext is the fundamental class for
rendering Core Image generated content Represents the drawing destination - either CPU or GPU Expensive to instantiate -
FILTER CATEGORIES 173 filters in 21 filter categories Distortion Effect
Geometry Adjustment Composite Operation Halftone Effect Color Adjustment Color Effect Transition Tile Effect Generator Reduction Gradient Stylize Sharpen Blur Video Still Image Interlaced Non Square Pixels -
BLUR For example Gaussian Blur Box & Tent Zoom &
Motion Blur -
COLOR ADJUSTMENT For example Hue adjustment Gamma & Exposure Tone
Curves Color Temperature -
COLOR EFFECTS For example False Color Photo Effects Posterisation Color
Cube & Polynomial -
GENERATOR For example Starshine & Lenticular Halo Stripes & Checkerboards
Solid Color Random Noise -
FILTERING IN PRACTICE Convert a UIImage to a CIImage let
bruges = UIImage( named: “bruges.jpg")! let image = CIImage( image: bruges)! -
FILTERING IN PRACTICE let noise = CIFilter(name: "CIRandomGenerator")?.outputImage? .imageByCroppingToRect(image.extent) let
filteredImage = image .imageByApplyingFilter( "CIVignette", withInputParameters: [kCIInputIntensityKey: 4]) .imageByApplyingFilter( "CIDarkenBlendMode", withInputParameters: [kCIInputBackgroundImageKey: noise!]) .imageByApplyingFilter( "CIColorControls", withInputParameters: [kCIInputSaturationKey: 0.25, kCIInputContrastKey: 1.15]) .imageByApplyingFilter( "CISepiaTone", withInputParameters: nil) -
FILTERING IN PRACTICE Render output to CGImage and display with
UIImageView let context = CIContext() let finalImage = context.createCGImage( filteredImage, fromRect: filteredImage.extent) imageView.image = UIImage( CGImage: finalImage) -
CUSTOM CORE IMAGE KERNELS Three types of kernel General Color
- can only change color information Warp - can only change where the destination pixel is sampled from Written in Core Image Kernel Language A dialect of GLSL -
COLOR KERNEL let shadedTileKernel = CIColorKernel( string: "kernel vec4 shadedTile(__sample
pixel)" + "{" + " vec2 coord = samplerCoord(pixel);" + " float brightness = mod(coord.y, 80.0) / 80.0;" + " brightness *= 1.0 - mod(coord.x, 80.0) / 80.0;" + " return vec4(sqrt(brightness) * pixel.rgb, pixel.a); " + "}") -
COLOR KERNEL Execute the kernel to create a CIImage let
final = shadedTileKernel.applyWithExtent( extent, arguments: arguments) -
WARP KERNEL let carnivalMirrorKernel = CIWarpKernel(string: "kernel vec2 carnivalMirror(float xWavelength,
float xAmount," + "float yWavelength, float yAmount)" + "{" + " float y = destCoord().y + sin(destCoord().y / yWavelength) * yAmount; " + " float x = destCoord().x + sin(destCoord().x / xWavelength) * xAmount; " + " return vec2(x, y); " + "}") -
WARP KERNEL Execute the kernel to create a CIImage let
final = kernel.applyWithExtent( extent, roiCallback: { (index, rect) in return rect }, inputImage: inputImage, arguments: arguments) -
GENERAL KERNEL let maskedVariableBlur = CIKernel(string: "kernel vec4 lumaVariableBlur(sampler image,
sampler blurImage, float blurRadius) " + "{ " + " vec2 d = destCoord(); " + " vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb; " + " float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722)); " + " float n = 0.0; " + " int radius = int(blurAmount * blurRadius); " + " vec3 accumulator = vec3(0.0, 0.0, 0.0); " + " for (int x = -radius; x <= radius; x++) " + " { " + " for (int y = -radius; y <= radius; y++) " + " { " + " vec2 workingSpaceCoordinate = d + vec2(x,y); " + " vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate); " + " vec3 color = sample(image, imageSpaceCoordinate).rgb; " + " accumulator += color; " + " n += 1.0; " + " } " + " } " + " accumulator /= n; " + " return vec4(accumulator, 1.0); " + "} " ) -
VIMAGE BUFFERS Converting a UIImage to a vImage Buffer var
format = vImage_CGImageFormat( bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.Last.rawValue), version: 0, decode: nil, renderingIntent: .RenderingIntentDefault) let cgImage = UIImage(named: "landscape.jpg")!.CGImage! var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, cgImage, UInt32(kvImageNoFlags)) -
VIMAGE BUFFERS Converting a vImage buffer to a UIImage extension
UIImage { convenience init?(fromvImageOutBuffer outBuffer:vImage_Buffer) { var mutableBuffer = outBuffer var error = vImage_Error() let cgImage = vImageCreateCGImageFromBuffer( &mutableBuffer, &format, nil, nil, UInt32(kvImageNoFlags), &error) self.init(CGImage: cgImage.takeRetainedValue()) } } -
VIMAGE HISTOGRAM Histograms contain the frequency of color values in
an image Equalisation: increase global contrast by making an image’s histogram uniform Calculation: returns the histogram of an image. Specification: apply a histogram (e.g. from a source image) to a target image Contrast Stretch: Stretches the contrast range of a source image to the full range of supported values (i.e. normalises). -
VIMAGE EQUALIZATION let pixelBuffer = malloc( CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var
outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) vImageEqualization_ARGB8888( &inBuffer, &outBuffer, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) -
VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage)
-> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) } -
VIMAGE SPECIFICATION Get histogram of source image func histogramCalculation(imageRef: CGImage)
-> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let alpha = [UInt](count: 256, repeatedValue: 0) let red = [UInt](count: 256, repeatedValue: 0) let green = [UInt](count: 256, repeatedValue: 0) let blue = [UInt](count: 256, repeatedValue: 0) let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha) let redPtr = UnsafeMutablePointer<vImagePixelCount>(red) let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green) let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue) let rgba = [alphaPtr, redPtr, greenPtr, bluePtr] let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba) vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags)) return (alpha, red, green, blue) } -
VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage,
histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! } -
VIMAGE SPECIFICATION Apply histogram to target image func histogramSpecification(imageRef: CGImage,
histogram: (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt])) -> UIImage { var inBuffer = vImage_Buffer() vImageBuffer_InitWithCGImage( &inBuffer, &format, nil, imageRef, UInt32(kvImageNoFlags)) let pixelBuffer = malloc(CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef)) var outBuffer = vImage_Buffer( data: pixelBuffer, height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef)) let alphaPtr = UnsafePointer<vImagePixelCount>(histogram.alpha) let redPtr = UnsafePointer<vImagePixelCount>(histogram.red) let greenPtr = UnsafePointer<vImagePixelCount>(histogram.green) let bluePtr = UnsafePointer<vImagePixelCount>(histogram.blue) let rgba = UnsafeMutablePointer<UnsafePointer<vImagePixelCount>>([alphaPtr, redPtr, greenPtr, bluePtr]) vImageHistogramSpecification_ARGB8888(&inBuffer, &outBuffer, rgba, UInt32(kvImageNoFlags)) let outImage = UIImage(fromvImageOutBuffer: outBuffer) free(pixelBuffer) return outImage! } -
let monalisa = UIImage(named: "monalisa.jpg")! let bluesky = UIImage(named: "bluesky.jpg")!
let histogram = histogramCalculation(bluesky.CGImage!) let colored = histogramSpecification(monalisa.CGImage!, histogram: histogram) VIMAGE SPECIFICATION -
VIMAGE CONVOLUTION Convolution Different kernel for each color channel Deconvolution
(Richardson-Lucy) More edging options than Core Image Supports larger kernels than Core Image -
DECONVOLUTION vImageRichardsonLucyDeConvolve_ARGB8888( &inBuffer, &outBuffer, nil, 0, 0, kernel, nil, kernelSide,
kernelSide, 0, 0, divisor, 0, [0,0,0,0], iterationCount, UInt32(kvImageNoFlags)) -
VIMAGE MORPHOLOGY Dilate & Erode With custom kernel Max &
Min Fast versions of dilate & erode with rectangular kernels -
VIMAGE DILATION let kernel: [UInt8] = [ 255, 255, 255,
255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 255, 255, 255, 255, 255, 000, 000, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 000, 255, 000, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 000, 255, 255, 000, 255, 255, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 000, 255, 255, 255, 255, 255, 255] vImageDilate_ARGB8888( &inBuffer, &outBuffer, 0, 0, kernel, UInt(kernelSide), UInt(kernelSide), UInt32(kvImageNoFlags)) -
VIMAGE CONTINUED Conversions - lots! Floating point to integer color
with dithering YUV to RGB Interleaved (RGB, RGB, RGB) to planar (RRR, GGG, BBB) -
METAL PERFORMANCE SHADERS Framework of data-parallel image processing algorithms for
GPU Similar set of functionality to vImage Histogram functions including equalisation & specification Convolution, Gaussian blur Morphology: min, max, erode and dilate Resampling: scale and transform Thresholding & Integral -
MPS GAUSSIAN BLUR Different implementations of Gaussian blur optimised for
Image size Blur radius Blur sigma -
MPS GAUSSIAN BLUR Create a Metal texture from a UIImage
let sourceImage = UIImage(named: “telescope.jpg")! let imageTexture: MTLTexture = { let textureLoader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!) let imageTexture:MTLTexture do { imageTexture = try textureLoader.newTextureWithCGImage( sourceImage.CGImage!, options: nil) } catch { fatalError("unable to create texture from image") } return imageTexture }() -
MPS GAUSSIAN BLUR Textures based on UIImage are upside-down let
intermediateTextureDesciptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat( MTLPixelFormat.RGBA8Unorm, width: imageTexture.width, height: imageTexture.height, mipmapped: false) let intermediateTexture = device.newTextureWithDescriptor(intermediateTextureDesciptor) let blur = MPSImageGaussianBlur( device: device, sigma: abs(sin(value)) * 200) let scale = MPSImageLanczosScale(device: device) var tx = MPSScaleTransform( scaleX: 1, scaleY: -1, translateX: 0, translateY: Double(-imageTexture.height)) withUnsafePointer(&tx) { scale.scaleTransform = $0 } -
MPS GAUSSIAN BLUR Encode both shaders to command buffer let
imageView = MTKView() let commandQueue = device.newCommandQueue() let commandBuffer = commandQueue.commandBuffer() scale.encodeToCommandBuffer( commandBuffer, sourceTexture: imageTexture, destinationTexture: intermediateTexture) blur.encodeToCommandBuffer( commandBuffer, sourceTexture: intermediateTexture, destinationTexture: currentDrawable.texture) commandBuffer.presentDrawable(imageView.currentDrawable!) commandBuffer.commit(); -
METAL COMPUTE SHADERS Lowest level - developer is responsible for
setting up Metal classes: Device - interface to GPU Library - repository of functions Command Queue - queues & submits commands Function - a Metal shader Pipeline State - compiles function Command Buffer - stores encoded commands Command Encoder - encodes resources to byte code -
METAL COMPUTE SHADERS Advantages over a Core Image kernel Improved
tooling Support for arrays Write to multiple pixels Write to multiple targets Disadvantages A lot more code! CIKL is automatically translated on the fly to Metal shader language -
METAL COMPUTE SHADERS kernel void pixellate( texture2d<float, access::read> inTexture [[texture(0)]],
texture2d<float, access::write> outTexture [[texture(1)]], constant float &pixelWidth [[ buffer(0) ]], constant float &pixelHeight [[ buffer(1) ]], uint2 gid [[thread_position_in_grid]]) { uint width = uint(pixelWidth); uint height = uint(pixelHeight); const uint2 pixellatedGid = uint2( (gid.x / width) * width, (gid.y / height) * height); const float4 colorAtPixel = inTexture.read(pixellatedGid); outTexture.write(colorAtPixel, gid); } -
RESOURCES Twitter: @FlexMonkey Core Image for Swift on Apple’s iBooks
Store & Gumroad https://github.com/FlexMonkey/ Filterpedia https://github.com/FlexMonkey/ ProgSConCompanion http://flexmonkey.blogspot.co.uk