On Android Rebecca Franks / @riggaroo Google Developer Expert Practical Image Processing

Rebecca Franks @riggaroo riggaroo.co.za Google Developer Expert

No content

Challenges • Process multiple images • Complex image processing • Storing filters • Custom Canvas Sizes • An iOS app that had the filters already

Mistakes we made Rendering project size vs display size ⏳ Loading bitmaps synchronously Many different Image processing tools/libraries

What is a Bitmap? 2D Array of Pixels Pixel = [R, G, B, A] (red, green, blue, alpha) Width Height

Loading a Bitmap

How do I load up a Bitmap? • Manually with BitmapFactory • Glide • Picasso • Fresco • …

val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options) options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap

Glide GlideApp.with(context) .load(“http://goo.gl/gEgYUd") .into(imageView) https://github.com/bumptech/glide

OutOfMemory

OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)

Bitmap too large Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too large(283435200bytes) bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas .java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)

You need to scale your images Height = 1920px Width = 700px 1920px * 700px = 1 344 000 * 4 bytes per pixel = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = 0.0256mb

How do I scale down my image? val options = BitmapFactory.Options() options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) BitmapFactory.decodeResource(res, resId, options)

Glide GlideApp.with(context) .load(file) .into(imageView)

Glide GlideApp.with(context) .asBitmap() .load(file) .submit(size.width, size.height)

Applying Effects

No content

Applying Image Effects • getPixel() & setPixel() • Glide Image Transformations / GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++) • … ⛔ ⛔

getPixel() & setPixel()

setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }

Pros % • Easy to understand • Manipulate per Pixel manually Cons & • Suuuuuper Slow • CPU Execution • No parallel execution

ColorFilter

What is a Color Matrix Filter? Modifies the color of each pixel drawn with a certain Paint.

ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t

ColorMatrix 4 * 5 matrix applied to each pixel R’ = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]

ColorMatrix Identity Matrix - same image in as out 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A

ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)

ColorMatrixColorFilter - Invert Image

What else can I do with it? Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia

Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t

Pros % • Easy-ish to understand • Fast (GPU) • Write in Kotlin/ Java • Understand the exact effect being applied Cons & • Can’t do all effects like Sharpen, Unsharp mask etc

ImageFilterView

ImageFilterView Part of ConstraintLayout 2.0 https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

ImageFilterView

ImageFilterView

Pros % • Easy to understand • Fast (GPU) • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you

Glide

Glide Image Transformations Additional to Glide https://github.com/wasabeef/glide-transformations GlideApp.with(this) .load(url) .apply(bitmapTransform(new BlurTransformation(25, 3))) .into(imageView)

Pros % • Easy to understand • Fast • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party

GPUImage

GPUImage - https://github.com/CyberAgent/android-gpuimage gpuImage = GPUImage(this) gpuImage.setGLSurfaceView(surfaceView) gpuImage.setImage(imageUri) gpuImage.setFilter(new GPUImageSepiaFilter())

Pros % • Easy to understand • Fast • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code

Renderscript

What is Renderscript? • Framework on Android for computationally expensive tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code

Renderscript Compute vs Graphics • Graphics - Deprecated • Compute - still widely used for image processing

Scripts

#pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); }

#pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);

static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }

class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)

class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)

fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }

fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }

script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap } fun cleanup(){ renderScript.destroy() script.destroy() } }

No content

Convolution Matrix

0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128

class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }

class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)

0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()

fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }

No content

Built-in Renderscript functions • ScriptIntrinsicBlur - Blurring • ScriptIntrinsicLUT - Lookup table • Many more…

Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw • Basic Renderscript - https:// github.com/googlesamples/android- BasicRenderScript • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2

Pros % • Fast • Highly customisable • Parallel execution on GPU • Interactions between layers isn’t complex Cons & • Need to understand a new language • Manage your own memory • Native Libs

What should I use?

How do I choose? ColorFilter / Renderscript / OpenGL? + ImageFilterView or Glide Transformations

Final Tips

Tips OutOfMemory ↕ Scale to display Use what is easiest for you

Thank you! Rebecca Franks @riggaroo [email protected]