[Rebecca Franks] Practical Image Processing in Android

Presentation from GDG DevFest Ukraine 2018 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

__

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Transcript

1. Practical Image Processing
   on Android
   @riggaroo
   

   View Slide

2. Rebecca Franks
   @riggaroo
   riggaroo.co.za
   Android @ Over
   

   View Slide

3. View Slide

4. Challenges
   • Process multiple images
   • Allow for “stacking” of
   image effects
   • Complex image processing
   • Storing filter values
   • Custom Canvas Sizes
   

   View Slide

5. Lessons learnt
   
   Rendering project
   size
   ⏳
   Loading bitmaps
   synchronously
   
   Many different
   Image processing
   tools/libraries
   

   View Slide

6. What is in a Bitmap?
   2D Array of Pixels
   Pixel = [R, G, B, A]
   (red, green, blue, alpha)
   Width
   Height
   

   View Slide

7. Loading
   images
   

   View Slide

8. How do I load up an Image?
   • Manually with BitmapFactory
   • Glide
   • Picasso
   • Fresco
   • …
   

   View Slide

9. val options = BitmapFactory.Options()
   options.inJustDecodeBounds = true
   BitmapFactory.decodeResource(res, resId, options)
   options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight)
   options.inJustDecodeBounds = false
   return BitmapFactory.decodeResource(res, resId, options)
   BitmapFactory
   https://developer.android.com/topic/performance/graphics/load-bitmap
   

   View Slide

10. Glide
    GlideApp.with(context)
    .load(“http://goo.gl/gEgYUd")
    .into(imageView)
    https://github.com/bumptech/glide
    

    View Slide

11. Glide
    GlideApp.with(context)
    .asBitmap()
    .load(file)
    .submit(size.width, size.height)
    

    View Slide

12. OutOfMemory
    

    View Slide

13. OutOfMemory
    Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228
    byte allocation with 4194304 free bytes and 4MB until OOM
    at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java)
    at android.graphics.Bitmap.nativeCreate(Bitmap.java)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:977)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:948)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
    

    View Slide

14. Canvas
    Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too
    large(283435200bytes) bitmap.
    at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas.java:260)
    at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
    

    View Slide

15. Scaling Images
    Height = 1920px
    Width = 700px
    1920px * 700px
    = 1 344 000 * 4 bytes per pixel (ARGB_8888)
    = 5 376 000 bytes
    = 5.375 mb
    Height = 128px
    Width = 50px
    = s0.0256mb
    

    View Slide

16. Applying
    Effects
    

    View Slide

17. View Slide

18. Applying Image Effects
    • getPixel() & setPixel()
    • Glide Image Transformations GPUImage
    • ColorMatrix
    • Renderscript
    • OpenGL
    • Native (C++)
    

    View Slide

19. getPixel() &
    setPixel()
    

    View Slide

20. setPixel()
    override fun filter(bitmap: Bitmap): Bitmap {
    val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true)
    for (i in 0 until bitmap.width){
    for (j in 0 until bitmap.height){
    newBitmap.setPixel(i, j, Color.BLUE)
    }
    }
    return newBitmap
    }
    

    View Slide

21. Pros $
    • Easy to understand
    • Manipulate per
    Pixel manually
    Cons %
    • Suuuuuper Slow
    • CPU Execution
    • No parallel
    execution
    

    View Slide

22. ColorFilter
    

    View Slide

23. What is a Color Matrix Filter?
    Modifies the color of each pixel
    drawn with a certain Paint.
    

    View Slide

24. ColorMatrixColorFilter
    4 * 5 matrix applied to each pixel
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    

    View Slide

25. ColorMatrix
    4 * 5 matrix applied to each pixel
    R’ = a*R + b*G + c*B + d*A + e;
    G’ = f*R + g*G + h*B + i*A + j;
    B’ = k*R + l*G + m*B + n*A + o;
    A’ = p*R + q*G + r*B + s*A + t;
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    [R’, G’, B’, A’]
    

    View Slide

26. ColorMatrix
    Identity Matrix - same image in as out
    1, 0, 0, 0, 0,
    0, 1, 0, 0, 0,
    0, 0, 1, 0, 0,
    0, 0, 0, 1, 0
    R,
    G,
    B,
    A
    * =
    R,
    G,
    B,
    A
    

    View Slide

27. ColorMatrixColorFilter - Invert Image
    val canvas = Canvas(bitmap)
    val paint = Paint()
    paint.colorFilter = ColorMatrixColorFilter(
    ColorMatrix(floatArrayOf(
    -1f, 0f, 0f, 0f, 255f,
    0f, -1f, 0f, 0f, 255f,
    0f, 0f, -1f, 0f, 255f,
    0f, 0f, 0f, 1f, 0f
    )))
    canvas.drawBitmap(original, 0, 0, paint)
    

    View Slide

28. ColorMatrixColorFilter - Invert Image
    

    View Slide

29. What else can I do with it?
    Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia
    

    View Slide

30. Other ColorFilters
    - LightingColorFilter
    - PorterDuffColorFilter
    More info: Manipulating images and Drawables
    with Android’s ColorFilter - Nick Rout
    bit.ly/2Cwvx8t
    

    View Slide

31. Pros $
    • Easy-ish to
    understand
    • Fast (GPU)
    • Write in Kotlin/
    Java
    • Understand the
    exact effect
    being applied
    Cons %
    • Can’t do all
    effects like
    Sharpen etc
    

    View Slide

32. ImageFilterVie
    w
    

    View Slide

33. ImageFilterView
    Part of ConstraintLayout 2.0
    https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView
    

    View Slide

34. ImageFilterView
    android:layout_width="0dp"
    android:layout_height="0dp"
    android:id="@+id/imageView"
    app:warmth="1.2"
    app:contrast="1.0"
    app:saturation="2.0"
    app:layout_constraintTop_toTopOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintDimensionRatio="16:9"
    tools:srcCompat="@tools:sample/avatars"
    />
    

    View Slide

35. ImageFilterView
    

    View Slide

36. Pros $
    • Easy to understand
    • Fast (GPU)
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    

    View Slide

37. Glide
    

    View Slide

38. Glide Image Transformations
    Additional to Glide
    https://github.com/wasabeef/glide-transformations
    GlideApp.with(this)
    .load(url)
    .apply(bitmapTransform(new BlurTransformation(25, 3)))
    .into(imageView)
    

    View Slide

39. Pros $
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    

    View Slide

40. GPUImage
    

    View Slide

41. GPUImage
    - https://github.com/CyberAgent/android-gpuimage
    gpuImage = GPUImage(this)
    gpuImage.setGLSurfaceView(surfaceView)
    gpuImage.setImage(imageUri)
    gpuImage.setFilter(new GPUImageSepiaFilter())
    

    View Slide

42. Pros $
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    • Dependent on OpenGL
    code
    

    View Slide

43. Renderscript
    

    View Slide

44. What is Renderscript?
    • Framework on Android for computationally
    expensive tasks
    • Parallelises tasks across CPU and GPU
    • Language: C99-derived language for writing
    high-performance compute code
    https://developer.android.com/guide/topics/renderscript/compute
    

    View Slide

45. Scripts
    

    View Slide

46. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    

    View Slide

47. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    

    View Slide

48. static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    out->g = clamp((int)(bright * in->g), 0, 255);
    out->b = clamp((int)(bright * in->b), 0, 255);
    }
    

    View Slide

49. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    

    View Slide

50. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    

    View Slide

51. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    

    View Slide

52. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    

    View Slide

53. script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    fun cleanup(){
    renderScript.destroy()
    script.destroy()
    }
    }
    

    View Slide

54. View Slide

55. Convolution
    Matrix
    

    View Slide

56. 0, -1, 0
    -1, 5, -1,
    0, -1, 0
    Sharpen an Image
    Sharpen Kernel
    2, 30, 1
    2, 40, 20,
    4, 20, 5
    Pixel in an Image with
    surrounding pixel values
    Resulting Pixel value
    = (0 * 2) + (-1 * 30) + (0 * 1)
    +(-1 * 2) + (5 * 40) + (-1* 20)
    +(0 * 4) + (-1 * 20) + (0 * 5)
    = 128
    

    View Slide

57. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    

    View Slide

58. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    

    View Slide

59. 0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    

    View Slide

60. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    }
    }
    

    View Slide

61. View Slide

62. Built-in Renderscript functions
    • ScriptIntrinsicBlur - Blurring
    • ScriptIntrinsicLUT - Lookup table
    • Many more…
    

    View Slide

63. Further Examples
    • Efficient Image Processing - Nicolas
    Roard https://youtu.be/j2szUDjScOw
    • Gallery2 - Renderscript Examples
    https://android.googlesource.com/
    platform/packages/apps/Gallery2
    

    View Slide

64. Pros $
    • Fast
    • Highly
    customisable
    • Parallel execution
    on GPU
    • Interactions
    between layers
    isn’t complex
    Cons %
    • Need to understand
    a new language
    • Manage your own
    memory
    • Native Libs
    

    View Slide

65. What should I use?
    

    View Slide

66. How do I choose?
    
    ColorFilter / Renderscript /
    OpenGL?
    *
    ImageFilterView or
    Glide Transformations
    

    View Slide

67. Final Tips
    

    View Slide

68. Tips
    
    OutOfMemory
    
    Scale to display
    
    Use what is
    easiest for you
    

    View Slide

69. Thank you!
    Rebecca Franks
    @riggaroo
    hello@riggaroo.co.za
    

    View Slide