Practical Image Processing on Android - DevFest Ukraine 2018

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Transcript

1. On Android Rebecca Franks / @riggaroo Google Developer Expert Practical Image Processing
2. Rebecca Franks @riggaroo riggaroo.co.za Google Developer Expert
4. Challenges • Process multiple images • Complex image processing • Storing filters • Custom Canvas Sizes • An iOS app that had the filters already
5. Mistakes we made Rendering project size vs display size ⏳ Loading bitmaps synchronously Many different Image processing tools/libraries
6. What is a Bitmap? 2D Array of Pixels Pixel = [R, G, B, A] (red, green, blue, alpha) Width Height
7. Loading a Bitmap
8. How do I load up a Bitmap? • Manually with BitmapFactory • Glide • Picasso • Fresco • …
9. val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options) options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap
10. Glide GlideApp.with(context) .load(“http://goo.gl/gEgYUd") .into(imageView) https://github.com/bumptech/glide
11. OutOfMemory
12. OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
13. Bitmap too large Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too large(283435200bytes) bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas .java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
14. You need to scale your images Height = 1920px Width = 700px 1920px * 700px = 1 344 000 * 4 bytes per pixel = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = 0.0256mb
15. How do I scale down my image? val options = BitmapFactory.Options() options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) BitmapFactory.decodeResource(res, resId, options)
16. Glide GlideApp.with(context) .load(file) .into(imageView)
17. Glide GlideApp.with(context) .asBitmap() .load(file) .submit(size.width, size.height)
18. Applying Effects
20. Applying Image Effects • getPixel() & setPixel() • Glide Image Transformations / GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++) • … ⛔ ⛔
21. getPixel() & setPixel()
22. setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }
23. Pros % • Easy to understand • Manipulate per Pixel manually Cons & • Suuuuuper Slow • CPU Execution • No parallel execution
24. ColorFilter
25. What is a Color Matrix Filter? Modifies the color of each pixel drawn with a certain Paint.
26. ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t
27. ColorMatrix 4 * 5 matrix applied to each pixel R’ = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]
28. ColorMatrix Identity Matrix - same image in as out 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A
29. ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)
30. ColorMatrixColorFilter - Invert Image
31. What else can I do with it? Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia
32. Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t
33. Pros % • Easy-ish to understand • Fast (GPU) • Write in Kotlin/ Java • Understand the exact effect being applied Cons & • Can’t do all effects like Sharpen, Unsharp mask etc
34. ImageFilterView
35. ImageFilterView Part of ConstraintLayout 2.0 https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView
36. ImageFilterView <androidx.constraintlayout.utils.widget.ImageFilterView android:layout_width="0dp" android:layout_height="0dp" android:id="@+id/imageView" app:warmth="1.2" app:contrast="1.0" app:saturation="2.0" app:layout_constraintTop_toTopOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintDimensionRatio="16:9" tools:srcCompat="@tools:sample/avatars" />
37. ImageFilterView
38. Pros % • Easy to understand • Fast (GPU) • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you
39. Glide
40. Glide Image Transformations Additional to Glide https://github.com/wasabeef/glide-transformations GlideApp.with(this) .load(url) .apply(bitmapTransform(new BlurTransformation(25, 3))) .into(imageView)
41. Pros % • Easy to understand • Fast • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party
42. GPUImage
43. GPUImage - https://github.com/CyberAgent/android-gpuimage gpuImage = GPUImage(this) gpuImage.setGLSurfaceView(surfaceView) gpuImage.setImage(imageUri) gpuImage.setFilter(new GPUImageSepiaFilter())
44. Pros % • Easy to understand • Fast • Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code
45. Renderscript
46. What is Renderscript? • Framework on Android for computationally expensive tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code
47. Renderscript Compute vs Graphics • Graphics - Deprecated • Compute - still widely used for image processing
48. Scripts
49. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); }
50. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);
51. static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }
52. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)
53. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)
54. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
55. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
56. script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap } fun cleanup(){ renderScript.destroy() script.destroy() } }
58. Convolution Matrix
59. 0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128
60. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }
61. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)
62. 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()
63. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }
65. Built-in Renderscript functions • ScriptIntrinsicBlur - Blurring • ScriptIntrinsicLUT - Lookup table • Many more…
66. Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw • Basic Renderscript - https:// github.com/googlesamples/android- BasicRenderScript • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2
67. Pros % • Fast • Highly customisable • Parallel execution on GPU • Interactions between layers isn’t complex Cons & • Need to understand a new language • Manage your own memory • Native Libs
68. What should I use?
69. How do I choose? ColorFilter / Renderscript / OpenGL? + ImageFilterView or Glide Transformations
70. Final Tips
71. Tips OutOfMemory ↕ Scale to display Use what is easiest for you
72. Thank you! Rebecca Franks @riggaroo hello@riggaroo.co.za