[Rebecca Franks] Practical Image Processing in Android

[Rebecca Franks] Practical Image Processing in Android

Presentation from GDG DevFest Ukraine 2018 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

__

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

3a6de6bc902de7f75c0e753b3202ed52?s=128

Google Developers Group Lviv

October 13, 2018
Tweet

Transcript

  1. Practical Image Processing on Android @riggaroo

  2. Rebecca Franks @riggaroo riggaroo.co.za Android @ Over

  3. None
  4. Challenges • Process multiple images • Allow for “stacking” of

    image effects • Complex image processing • Storing filter values • Custom Canvas Sizes
  5. Lessons learnt Rendering project size ⏳ Loading bitmaps synchronously Many

    different Image processing tools/libraries
  6. What is in a Bitmap? 2D Array of Pixels Pixel

    = [R, G, B, A] (red, green, blue, alpha) Width Height
  7. Loading images

  8. How do I load up an Image? • Manually with

    BitmapFactory • Glide • Picasso • Fresco • …
  9. val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options)

    options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap
  10. Glide GlideApp.with(context) .load(“http://goo.gl/gEgYUd") .into(imageView) https://github.com/bumptech/glide

  11. Glide GlideApp.with(context) .asBitmap() .load(file) .submit(size.width, size.height)

  12. OutOfMemory

  13. OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte

    allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
  14. Canvas Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too large(283435200bytes)

    bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas.java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
  15. Scaling Images Height = 1920px Width = 700px 1920px *

    700px = 1 344 000 * 4 bytes per pixel (ARGB_8888) = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = s0.0256mb
  16. Applying Effects

  17. None
  18. Applying Image Effects • getPixel() & setPixel() • Glide Image

    Transformations GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++)
  19. getPixel() & setPixel()

  20. setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap =

    bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }
  21. Pros $ • Easy to understand • Manipulate per Pixel

    manually Cons % • Suuuuuper Slow • CPU Execution • No parallel execution
  22. ColorFilter

  23. What is a Color Matrix Filter? Modifies the color of

    each pixel drawn with a certain Paint.
  24. ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a,

    b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t
  25. ColorMatrix 4 * 5 matrix applied to each pixel R’

    = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]
  26. ColorMatrix Identity Matrix - same image in as out 1,

    0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A
  27. ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint

    = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)
  28. ColorMatrixColorFilter - Invert Image

  29. What else can I do with it? Warmth, Exposure, Saturation,

    Contrast, Grayscale, Sepia
  30. Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images

    and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t
  31. Pros $ • Easy-ish to understand • Fast (GPU) •

    Write in Kotlin/ Java • Understand the exact effect being applied Cons % • Can’t do all effects like Sharpen etc
  32. ImageFilterVie w

  33. ImageFilterView Part of ConstraintLayout 2.0 https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

  34. ImageFilterView <androidx.constraintlayout.utils.widget.ImageFilterView android:layout_width="0dp" android:layout_height="0dp" android:id="@+id/imageView" app:warmth="1.2" app:contrast="1.0" app:saturation="2.0" app:layout_constraintTop_toTopOf="parent" app:layout_constraintEnd_toEndOf="parent"

    app:layout_constraintStart_toStartOf="parent" app:layout_constraintDimensionRatio="16:9" tools:srcCompat="@tools:sample/avatars" />
  35. ImageFilterView

  36. Pros $ • Easy to understand • Fast (GPU) •

    Nice API • Java/Kotlin Cons % • Can’t do everything • Values are abstracted away from you
  37. Glide

  38. Glide Image Transformations Additional to Glide https://github.com/wasabeef/glide-transformations GlideApp.with(this) .load(url) .apply(bitmapTransform(new

    BlurTransformation(25, 3))) .into(imageView)
  39. Pros $ • Easy to understand • Fast • Nice

    API • Java/Kotlin Cons % • Can’t do everything • Values are abstracted away from you • Dependant on Third Party
  40. GPUImage

  41. GPUImage - https://github.com/CyberAgent/android-gpuimage gpuImage = GPUImage(this) gpuImage.setGLSurfaceView(surfaceView) gpuImage.setImage(imageUri) gpuImage.setFilter(new GPUImageSepiaFilter())

  42. Pros $ • Easy to understand • Fast • Nice

    API • Java/Kotlin Cons % • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code
  43. Renderscript

  44. What is Renderscript? • Framework on Android for computationally expensive

    tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code https://developer.android.com/guide/topics/renderscript/compute
  45. Scripts

  46. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright

    = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); }
  47. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright

    = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);
  48. static float bright = 0.f; void setBright(float v) { bright

    = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }
  49. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context)

    private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)
  50. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context)

    private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)
  51. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap =

    Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
  52. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap =

    Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
  53. script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap } fun

    cleanup(){ renderScript.destroy() script.destroy() } }
  54. None
  55. Convolution Matrix

  56. 0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen

    an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128
  57. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript =

    RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }
  58. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript =

    RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)
  59. 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap,

    coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()
  60. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap

    = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }
  61. None
  62. Built-in Renderscript functions • ScriptIntrinsicBlur - Blurring • ScriptIntrinsicLUT -

    Lookup table • Many more…
  63. Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw

    • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2
  64. Pros $ • Fast • Highly customisable • Parallel execution

    on GPU • Interactions between layers isn’t complex Cons % • Need to understand a new language • Manage your own memory • Native Libs
  65. What should I use?

  66. How do I choose? ColorFilter / Renderscript / OpenGL? *

    ImageFilterView or Glide Transformations
  67. Final Tips

  68. Tips OutOfMemory Scale to display Use what is easiest for

    you
  69. Thank you! Rebecca Franks @riggaroo hello@riggaroo.co.za