Slide 1

Slide 1 text

On Android Rebecca Franks / @riggaroo Google Developer Expert Practical Image Processing

Slide 2

Slide 2 text

Rebecca Franks @riggaroo riggaroo.co.za Google Developer Expert

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Challenges • Process multiple images • Complex image processing • Storing filters • Custom Canvas Sizes • An iOS app that had the filters already

Slide 5

Slide 5 text

Mistakes we made Rendering project size vs display size ⏳ Loading bitmaps synchronously Many different Image processing tools/libraries

Slide 6

Slide 6 text

What is a Bitmap? 2D Array of Pixels Pixel = [R, G, B, A] (red, green, blue, alpha) Width Height

Slide 7

Slide 7 text

Loading a Bitmap

Slide 8

Slide 8 text

How do I load up a Bitmap? • Manually with BitmapFactory • Glide • Picasso • Fresco • …

Slide 9

Slide 9 text

val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options) options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap

Slide 10

Slide 10 text

Glide GlideApp.with(context) .load(“http://goo.gl/gEgYUd") .into(imageView) https://github.com/bumptech/glide

Slide 11

Slide 11 text

OutOfMemory

Slide 12

Slide 12 text

OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)

Slide 13

Slide 13 text

Bitmap too large Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too large(283435200bytes) bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas .java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)

Slide 14

Slide 14 text

You need to scale your images Height = 1920px Width = 700px 1920px * 700px = 1 344 000 * 4 bytes per pixel = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = 0.0256mb

Slide 15

Slide 15 text

How do I scale down my image? val options = BitmapFactory.Options() options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) BitmapFactory.decodeResource(res, resId, options)

Slide 16

Slide 16 text

Glide GlideApp.with(context) .load(file) .into(imageView)

Slide 17

Slide 17 text

Glide GlideApp.with(context) .asBitmap() .load(file) .submit(size.width, size.height)

Slide 18

Slide 18 text

Applying Effects

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Applying Image Effects • getPixel() & setPixel() • Glide Image Transformations / GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++) • … ⛔ ⛔

Slide 21

Slide 21 text

getPixel() & setPixel()

Slide 22

Slide 22 text

setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }

Slide 23

Slide 23 text

Pros % • Easy to understand • Manipulate per Pixel manually Cons & • Suuuuuper Slow • CPU Execution • No parallel execution

Slide 24

Slide 24 text

ColorFilter

Slide 25

Slide 25 text

What is a Color Matrix Filter? Modifies the color of each pixel drawn with a certain Paint.

Slide 26

Slide 26 text

ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t

Slide 27

Slide 27 text

ColorMatrix 4 * 5 matrix applied to each pixel R’ = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]

Slide 28

Slide 28 text

ColorMatrix Identity Matrix - same image in as out 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A

Slide 29

Slide 29 text

ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)

Slide 30

Slide 30 text

ColorMatrixColorFilter - Invert Image

Slide 31

Slide 31 text

What else can I do with it? Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia

Slide 32

Slide 32 text

Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t

Slide 33

Slide 33 text

Pros % • Easy-ish to understand • Fast (GPU) • Write in Kotlin/ Java • Understand the exact effect being applied Cons & • Can’t do all effects like Sharpen, Unsharp mask etc

Slide 34

Slide 34 text

ImageFilterView

Slide 35

Slide 35 text

ImageFilterView Part of ConstraintLayout 2.0 https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

Slide 36

Slide 36 text

ImageFilterView

Slide 37

Slide 37 text

ImageFilterView

Slide 38

Slide 38 text

Pros % • Easy to understand • Fast (GPU) • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you

Slide 39

Slide 39 text

Glide

Slide 40

Slide 40 text

Glide Image Transformations Additional to Glide https://github.com/wasabeef/glide-transformations GlideApp.with(this) .load(url) .apply(bitmapTransform(new BlurTransformation(25, 3))) .into(imageView)

Slide 41

Slide 41 text

Pros % • Easy to understand • Fast • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party

Slide 42

Slide 42 text

GPUImage

Slide 43

Slide 43 text

GPUImage - https://github.com/CyberAgent/android-gpuimage gpuImage = GPUImage(this) gpuImage.setGLSurfaceView(surfaceView) gpuImage.setImage(imageUri) gpuImage.setFilter(new GPUImageSepiaFilter())

Slide 44

Slide 44 text

Pros % • Easy to understand • Fast • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code

Slide 45

Slide 45 text

Renderscript

Slide 46

Slide 46 text

What is Renderscript? • Framework on Android for computationally expensive tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code

Slide 47

Slide 47 text

Renderscript Compute vs Graphics • Graphics - Deprecated • Compute - still widely used for image processing

Slide 48

Slide 48 text

Scripts

Slide 49

Slide 49 text

#pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); }

Slide 50

Slide 50 text

#pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);

Slide 51

Slide 51 text

static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }

Slide 52

Slide 52 text

class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)

Slide 53

Slide 53 text

class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)

Slide 54

Slide 54 text

fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }

Slide 55

Slide 55 text

fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }

Slide 56

Slide 56 text

script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap } fun cleanup(){ renderScript.destroy() script.destroy() } }

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

Convolution Matrix

Slide 59

Slide 59 text

0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128

Slide 60

Slide 60 text

class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }

Slide 61

Slide 61 text

class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)

Slide 62

Slide 62 text

0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()

Slide 63

Slide 63 text

fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

Built-in Renderscript functions • ScriptIntrinsicBlur - Blurring • ScriptIntrinsicLUT - Lookup table • Many more…

Slide 66

Slide 66 text

Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw • Basic Renderscript - https:// github.com/googlesamples/android- BasicRenderScript • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2

Slide 67

Slide 67 text

Pros % • Fast • Highly customisable • Parallel execution on GPU • Interactions between layers isn’t complex Cons & • Need to understand a new language • Manage your own memory • Native Libs

Slide 68

Slide 68 text

What should I use?

Slide 69

Slide 69 text

How do I choose? ColorFilter / Renderscript / OpenGL? + ImageFilterView or Glide Transformations

Slide 70

Slide 70 text

Final Tips

Slide 71

Slide 71 text

Tips OutOfMemory ↕ Scale to display Use what is easiest for you

Slide 72

Slide 72 text

Thank you! Rebecca Franks @riggaroo [email protected]