Aug. 5, 2014 GPUImage Introduction Bruce Tsai 

Core Image ✤ Included in iOS from 2011! ✤ 100+ built-in filters! ✤ Multithread by CPU 

Drawbacks ✤ Customization limited! - Subclass CIFilter in iOS! - Create custom filters (shader) only in OSX! ✤ Performance not tuned! - Address tuning tips in WWDC 2012 (session 511) 3 

GPUImage ✤ Open source (BSD) iOS framework! ✤ GPU-based image and video processing! ✤ From GPU-accelerated video processing project (2010)! ✤ https://github.com/BradLarson/GPUImage 4 

Advantages ✤ Performance! - High FPS in real-time preview! ✤ Customization! - OpenGL ES shader code 5 

Buffer Process Flow VideoCamera Sepia View MovieWriter Blend Picture Source Filter Output 

Code Snippet GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack]; GPUImageFilter *customFilter = [[GPUImageFilter alloc] initWithFragmentShaderFromFile:@“CustomShader”]; GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:CGRectMake(0.0, 0.0, viewWidth, viewHeight)]; [videoCamera addTarget:customFilter]; [customFilter addTarget:filteredVideoView]; [videoCamera startCameraCapture]; 

Class Diagram 

Procedures 1. Add effect to input buffer ! 2. Save result to output buffer! 3. Set targets’ input buffer attributes! 4. Set targets’ input buffer! 5. Invoke targets’ operation 9 

No content

Frame Buffer Object (FBO) glBindFramebuffer(GL_FRAMEBUFFER, framebuffer); glBindTexture(GL_TEXTURE_2D, renderTexture); glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, renderTexture, 0); 0QVRTGUGPVVQ 7+KOOGFKCVGN[ 

Buffer Binding in FBO Frame Buffer Texture Pixel Buffer Render destination Actual render target Main memory buffer 

Conversion to and from Texture CVPixelBufferRef pxBufferRef = CMSampleBufferGetImageBuffer(imageBuffer); CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDe fault, videoTextureCache, pxBufferRef, NULL, GL_TEXTURE_2D, GL_RGBA, width, height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &videoTextureRef); ... // renderTargetRef is corresponding to renderTexture of FBO [writerAdaptor appendPixelBuffer:renderTargetRef withPresentationTime:pts]; 

Threading Model ✤ Grand Central Dispatch (GCD)! - Dispatch queue! ✤ runSynchronousOnVideoProcessingQueue! ✤ runAsynchronousOnVideoProcessingQueue! ✤ runSynchronousOnContextQueue! ✤ runAsynchronousOnContextQueue 14 

No content

Demo 

Reference ✤ https://github.com/BradLarson/GPUImage! ✤ http://blog.db-in.com/all-about-opengl-es-2-x-part-2/! ✤ https://developer.apple.com/library/ios/documentation/ 3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/ Introduction/Introduction.html#//apple_ref/doc/uid/ TP40008793! ✤ https://developer.apple.com/library/ios/documentation/ GraphicsImaging/Conceptual/CoreImaging/ci_intro/ ci_intro.html#//apple_ref/doc/uid/TP30001185