[Rebecca Franks] Practical Image Processing in Android

Presentation from GDG DevFest Ukraine 2018 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

__

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Transcript

1. Practical Image Processing on Android @riggaroo
2. Rebecca Franks @riggaroo riggaroo.co.za Android @ Over
4. Challenges • Process multiple images • Allow for “stacking” of image effects • Complex image processing • Storing filter values • Custom Canvas Sizes
5. Lessons learnt Rendering project size ⏳ Loading bitmaps synchronously Many different Image processing tools/libraries
6. What is in a Bitmap? 2D Array of Pixels Pixel = [R, G, B, A] (red, green, blue, alpha) Width Height
7. Loading images
8. How do I load up an Image? • Manually with BitmapFactory • Glide • Picasso • Fresco • …
9. val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options) options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap
10. Glide GlideApp.with(context) .load(“http://goo.gl/gEgYUd") .into(imageView) https://github.com/bumptech/glide
11. Glide GlideApp.with(context) .asBitmap() .load(file) .submit(size.width, size.height)
12. OutOfMemory
13. OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
14. Canvas Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too large(283435200bytes) bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas.java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
15. Scaling Images Height = 1920px Width = 700px 1920px * 700px = 1 344 000 * 4 bytes per pixel (ARGB_8888) = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = s0.0256mb
16. Applying Effects
18. Applying Image Effects • getPixel() & setPixel() • Glide Image Transformations GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++)
19. getPixel() & setPixel()
20. setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }
21. Pros $ • Easy to understand • Manipulate per Pixel manually Cons % • Suuuuuper Slow • CPU Execution • No parallel execution
22. ColorFilter
23. What is a Color Matrix Filter? Modifies the color of each pixel drawn with a certain Paint.
24. ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t
25. ColorMatrix 4 * 5 matrix applied to each pixel R’ = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]
26. ColorMatrix Identity Matrix - same image in as out 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A
27. ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)
28. ColorMatrixColorFilter - Invert Image
29. What else can I do with it? Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia
30. Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t
31. Pros $ • Easy-ish to understand • Fast (GPU) • Write in Kotlin/ Java • Understand the exact effect being applied Cons % • Can’t do all effects like Sharpen etc
32. ImageFilterVie w
33. ImageFilterView Part of ConstraintLayout 2.0 https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView
34. ImageFilterView <androidx.constraintlayout.utils.widget.ImageFilterView android:layout_width="0dp" android:layout_height="0dp" android:id="@+id/imageView" app:warmth="1.2" app:contrast="1.0" app:saturation="2.0" app:layout_constraintTop_toTopOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintDimensionRatio="16:9" tools:srcCompat="@tools:sample/avatars" />
35. ImageFilterView
36. Pros $ • Easy to understand • Fast (GPU) • Nice API • Java/Kotlin Cons % • Can’t do everything • Values are abstracted away from you
37. Glide
38. Glide Image Transformations Additional to Glide https://github.com/wasabeef/glide-transformations GlideApp.with(this) .load(url) .apply(bitmapTransform(new BlurTransformation(25, 3))) .into(imageView)
39. Pros $ • Easy to understand • Fast • Nice API • Java/Kotlin Cons % • Can’t do everything • Values are abstracted away from you • Dependant on Third Party
40. GPUImage
41. GPUImage - https://github.com/CyberAgent/android-gpuimage gpuImage = GPUImage(this) gpuImage.setGLSurfaceView(surfaceView) gpuImage.setImage(imageUri) gpuImage.setFilter(new GPUImageSepiaFilter())
42. Pros $ • Easy to understand • Fast • Nice API • Java/Kotlin Cons % • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code
43. Renderscript
44. What is Renderscript? • Framework on Android for computationally expensive tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code https://developer.android.com/guide/topics/renderscript/compute
45. Scripts
46. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); }
47. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);
48. static float bright = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }
49. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)
50. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context) private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)
51. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
52. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
53. script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap } fun cleanup(){ renderScript.destroy() script.destroy() } }
55. Convolution Matrix
56. 0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128
57. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }
58. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript = RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)
59. 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()
60. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }
62. Built-in Renderscript functions • ScriptIntrinsicBlur - Blurring • ScriptIntrinsicLUT - Lookup table • Many more…
63. Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2
64. Pros $ • Fast • Highly customisable • Parallel execution on GPU • Interactions between layers isn’t complex Cons % • Need to understand a new language • Manage your own memory • Native Libs
65. What should I use?
66. How do I choose? ColorFilter / Renderscript / OpenGL? * ImageFilterView or Glide Transformations
67. Final Tips
68. Tips OutOfMemory Scale to display Use what is easiest for you
69. Thank you! Rebecca Franks @riggaroo hello@riggaroo.co.za