iOS GPUImage introduction

An introduction to GPUImage that is an excellent image processing framework for iOS. The architecture, threading model and how to leverage OpenGL ES are addressed in the slides.

Transcript

1. Aug. 5, 2014 GPUImage Introduction Bruce Tsai
2. Core Image ✤ Included in iOS from 2011! ✤ 100+ built-in filters! ✤ Multithread by CPU
3. Drawbacks ✤ Customization limited! - Subclass CIFilter in iOS! - Create custom filters (shader) only in OSX! ✤ Performance not tuned! - Address tuning tips in WWDC 2012 (session 511) 3
4. GPUImage ✤ Open source (BSD) iOS framework! ✤ GPU-based image and video processing! ✤ From GPU-accelerated video processing project (2010)! ✤ https://github.com/BradLarson/GPUImage 4
5. Advantages ✤ Performance! - High FPS in real-time preview! ✤ Customization! - OpenGL ES shader code 5
6. Buffer Process Flow VideoCamera Sepia View MovieWriter Blend Picture Source Filter Output
7. Code Snippet GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack]; GPUImageFilter *customFilter = [[GPUImageFilter alloc] initWithFragmentShaderFromFile:@“CustomShader”]; GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:CGRectMake(0.0, 0.0, viewWidth, viewHeight)]; [videoCamera addTarget:customFilter]; [customFilter addTarget:filteredVideoView]; [videoCamera startCameraCapture];
8. Class Diagram
9. Procedures 1. Add effect to input buffer ! 2. Save result to output buffer! 3. Set targets’ input buffer attributes! 4. Set targets’ input buffer! 5. Invoke targets’ operation 9
11. Frame Buffer Object (FBO) glBindFramebuffer(GL_FRAMEBUFFER, framebuffer); glBindTexture(GL_TEXTURE_2D, renderTexture); glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, renderTexture, 0); 0QVRTGUGPVVQ 7+KOOGFKCVGN[
12. Buffer Binding in FBO Frame Buffer Texture Pixel Buffer Render destination Actual render target Main memory buffer
13. Conversion to and from Texture CVPixelBufferRef pxBufferRef = CMSampleBufferGetImageBuffer(imageBuffer); CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDe fault, videoTextureCache, pxBufferRef, NULL, GL_TEXTURE_2D, GL_RGBA, width, height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &videoTextureRef); ... // renderTargetRef is corresponding to renderTexture of FBO [writerAdaptor appendPixelBuffer:renderTargetRef withPresentationTime:pts];
14. Threading Model ✤ Grand Central Dispatch (GCD)! - Dispatch queue! ✤ runSynchronousOnVideoProcessingQueue! ✤ runAsynchronousOnVideoProcessingQueue! ✤ runSynchronousOnContextQueue! ✤ runAsynchronousOnContextQueue 14
16. Demo
17. Reference ✤ https://github.com/BradLarson/GPUImage! ✤ http://blog.db-in.com/all-about-opengl-es-2-x-part-2/! ✤ https://developer.apple.com/library/ios/documentation/ 3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/ Introduction/Introduction.html#//apple_ref/doc/uid/ TP40008793! ✤ https://developer.apple.com/library/ios/documentation/ GraphicsImaging/Conceptual/CoreImaging/ci_intro/ ci_intro.html#//apple_ref/doc/uid/TP30001185