Practical Image Processing on Android - DevFest Ukraine 2018

Practical Image Processing on Android - DevFest Ukraine 2018

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

2a37bf1e025cc1523124774c760df91a?s=128

Rebecca Franks

October 13, 2018
Tweet

Transcript

  1. On Android Rebecca Franks / @riggaroo Google Developer Expert Practical

    Image Processing
  2. Rebecca Franks @riggaroo riggaroo.co.za Google Developer Expert

  3. None
  4. Challenges • Process multiple images • Complex image processing •

    Storing filters • Custom Canvas Sizes • An iOS app that had the filters already
  5. Mistakes we made Rendering project size vs display size ⏳

    Loading bitmaps synchronously Many different Image processing tools/libraries
  6. What is a Bitmap? 2D Array of Pixels Pixel =

    [R, G, B, A] (red, green, blue, alpha) Width Height
  7. Loading a Bitmap

  8. How do I load up a Bitmap? • Manually with

    BitmapFactory • Glide • Picasso • Fresco • …
  9. val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options)

    options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap
  10. Glide GlideApp.with(context) .load(“http://goo.gl/gEgYUd") .into(imageView) https://github.com/bumptech/glide

  11. OutOfMemory

  12. OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte

    allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
  13. Bitmap too large Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw

    too large(283435200bytes) bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas .java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
  14. You need to scale your images Height = 1920px Width

    = 700px 1920px * 700px = 1 344 000 * 4 bytes per pixel = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = 0.0256mb
  15. How do I scale down my image? val options =

    BitmapFactory.Options() options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) BitmapFactory.decodeResource(res, resId, options)
  16. Glide GlideApp.with(context) .load(file) .into(imageView)

  17. Glide GlideApp.with(context) .asBitmap() .load(file) .submit(size.width, size.height)

  18. Applying Effects

  19. None
  20. Applying Image Effects • getPixel() & setPixel() • Glide Image

    Transformations / GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++) • … ⛔ ⛔
  21. getPixel() & setPixel()

  22. setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap =

    bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }
  23. Pros % • Easy to understand • Manipulate per Pixel

    manually Cons & • Suuuuuper Slow • CPU Execution • No parallel execution
  24. ColorFilter

  25. What is a Color Matrix Filter? Modifies the color of

    each pixel drawn with a certain Paint.
  26. ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a,

    b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t
  27. ColorMatrix 4 * 5 matrix applied to each pixel R’

    = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]
  28. ColorMatrix Identity Matrix - same image in as out 1,

    0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A
  29. ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint

    = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)
  30. ColorMatrixColorFilter - Invert Image

  31. What else can I do with it? Warmth, Exposure, Saturation,

    Contrast, Grayscale, Sepia
  32. Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images

    and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t
  33. Pros % • Easy-ish to understand • Fast (GPU) •

    Write in Kotlin/ Java • Understand the exact effect being applied Cons & • Can’t do all effects like Sharpen, Unsharp mask etc
  34. ImageFilterView

  35. ImageFilterView Part of ConstraintLayout 2.0 https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

  36. ImageFilterView <androidx.constraintlayout.utils.widget.ImageFilterView android:layout_width="0dp" android:layout_height="0dp" android:id="@+id/imageView" app:warmth="1.2" app:contrast="1.0" app:saturation="2.0" app:layout_constraintTop_toTopOf="parent" app:layout_constraintEnd_toEndOf="parent"

    app:layout_constraintStart_toStartOf="parent" app:layout_constraintDimensionRatio="16:9" tools:srcCompat="@tools:sample/avatars" />
  37. ImageFilterView

  38. Pros % • Easy to understand • Fast (GPU) •

    Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you
  39. Glide

  40. Glide Image Transformations Additional to Glide https://github.com/wasabeef/glide-transformations GlideApp.with(this) .load(url) .apply(bitmapTransform(new

    BlurTransformation(25, 3))) .into(imageView)
  41. Pros % • Easy to understand • Fast • Nice

    API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party
  42. GPUImage

  43. GPUImage - https://github.com/CyberAgent/android-gpuimage gpuImage = GPUImage(this) gpuImage.setGLSurfaceView(surfaceView) gpuImage.setImage(imageUri) gpuImage.setFilter(new GPUImageSepiaFilter())

  44. Pros % • Easy to understand • Fast • Nice

    API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code
  45. Renderscript

  46. What is Renderscript? • Framework on Android for computationally expensive

    tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code
  47. Renderscript Compute vs Graphics • Graphics - Deprecated • Compute

    - still widely used for image processing
  48. Scripts

  49. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright

    = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); }
  50. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright

    = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);
  51. static float bright = 0.f; void setBright(float v) { bright

    = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }
  52. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context)

    private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)
  53. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context)

    private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)
  54. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap =

    Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
  55. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap =

    Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
  56. script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap } fun

    cleanup(){ renderScript.destroy() script.destroy() } }
  57. None
  58. Convolution Matrix

  59. 0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen

    an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128
  60. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript =

    RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }
  61. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript =

    RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)
  62. 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap,

    coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()
  63. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap

    = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }
  64. None
  65. Built-in Renderscript functions • ScriptIntrinsicBlur - Blurring • ScriptIntrinsicLUT -

    Lookup table • Many more…
  66. Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw

    • Basic Renderscript - https:// github.com/googlesamples/android- BasicRenderScript • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2
  67. Pros % • Fast • Highly customisable • Parallel execution

    on GPU • Interactions between layers isn’t complex Cons & • Need to understand a new language • Manage your own memory • Native Libs
  68. What should I use?

  69. How do I choose? ColorFilter / Renderscript / OpenGL? +

    ImageFilterView or Glide Transformations
  70. Final Tips

  71. Tips OutOfMemory ↕ Scale to display Use what is easiest

    for you
  72. Thank you! Rebecca Franks @riggaroo hello@riggaroo.co.za