Practical Image Processing on Android - DevFest Ukraine 2018

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Transcript

1. On Android
   Rebecca Franks / @riggaroo
   Google Developer Expert
   Practical Image Processing
   

   View Slide

2. Rebecca Franks
   @riggaroo
   riggaroo.co.za
   Google Developer Expert
   

   View Slide

3. View Slide

4. Challenges
   • Process multiple images
   • Complex image processing
   • Storing filters
   • Custom Canvas Sizes
   • An iOS app that had the
   filters already
   

   View Slide

5. Mistakes we made
   
   Rendering project
   size vs display size
   ⏳
   Loading bitmaps
   synchronously
   
   Many different
   Image processing
   tools/libraries
   

   View Slide

6. What is a Bitmap?
   2D Array of Pixels
   Pixel = [R, G, B, A]
   (red, green, blue, alpha)
   Width
   Height
   

   View Slide

7. Loading a
   Bitmap
   

   View Slide

8. How do I load up a Bitmap?
   • Manually with BitmapFactory
   • Glide
   • Picasso
   • Fresco
   • …
   

   View Slide

9. val options = BitmapFactory.Options()
   options.inJustDecodeBounds = true
   BitmapFactory.decodeResource(res, resId, options)
   options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight)
   options.inJustDecodeBounds = false
   return BitmapFactory.decodeResource(res, resId, options)
   BitmapFactory
   https://developer.android.com/topic/performance/graphics/load-bitmap
   

   View Slide

10. Glide
    GlideApp.with(context)
    .load(“http://goo.gl/gEgYUd")
    .into(imageView)
    https://github.com/bumptech/glide
    

    View Slide

11. OutOfMemory
    

    View Slide

12. OutOfMemory
    Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a
    16777228 byte allocation with 4194304 free bytes and 4MB until OOM
    at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java)
    at android.graphics.Bitmap.nativeCreate(Bitmap.java)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:977)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:948)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
    

    View Slide

13. Bitmap too large
    Fatal Exception: java.lang.RuntimeException: Canvas: trying to
    draw too large(283435200bytes) bitmap.
    at
    android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas
    .java:260)
    at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
    

    View Slide

14. You need to scale your images
    Height = 1920px
    Width = 700px
    1920px * 700px
    = 1 344 000 * 4 bytes per pixel
    = 5 376 000 bytes
    = 5.375 mb
    Height = 128px
    Width = 50px
    = 0.0256mb
    

    View Slide

15. How do I scale down my image?
    val options = BitmapFactory.Options()
    options.inSampleSize = calculateInSampleSize(options,
    reqWidth, reqHeight)
    BitmapFactory.decodeResource(res, resId, options)
    

    View Slide

16. Glide
    GlideApp.with(context)
    .load(file)
    .into(imageView)
    

    View Slide

17. Glide
    GlideApp.with(context)
    .asBitmap()
    .load(file)
    .submit(size.width, size.height)
    

    View Slide

18. Applying
    Effects
    

    View Slide

19. View Slide

20. Applying Image Effects
    • getPixel() & setPixel()
    • Glide Image Transformations / GPUImage
    • ColorMatrix
    • Renderscript
    • OpenGL
    • Native (C++)
    • …
    ⛔
    ⛔
    

    View Slide

21. getPixel() &
    setPixel()
    

    View Slide

22. setPixel()
    override fun filter(bitmap: Bitmap): Bitmap {
    val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true)
    for (i in 0 until bitmap.width){
    for (j in 0 until bitmap.height){
    newBitmap.setPixel(i, j, Color.BLUE)
    }
    }
    return newBitmap
    }
    

    View Slide

23. Pros %
    • Easy to understand
    • Manipulate per
    Pixel manually
    Cons &
    • Suuuuuper Slow
    • CPU Execution
    • No parallel
    execution
    

    View Slide

24. ColorFilter
    

    View Slide

25. What is a Color Matrix Filter?
    Modifies the color of each pixel
    drawn with a certain Paint.
    

    View Slide

26. ColorMatrixColorFilter
    4 * 5 matrix applied to each pixel
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    

    View Slide

27. ColorMatrix
    4 * 5 matrix applied to each pixel
    R’ = a*R + b*G + c*B + d*A + e;
    G’ = f*R + g*G + h*B + i*A + j;
    B’ = k*R + l*G + m*B + n*A + o;
    A’ = p*R + q*G + r*B + s*A + t;
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    [R’, G’, B’, A’]
    

    View Slide

28. ColorMatrix
    Identity Matrix - same image in as out
    1, 0, 0, 0, 0,
    0, 1, 0, 0, 0,
    0, 0, 1, 0, 0,
    0, 0, 0, 1, 0
    R,
    G,
    B,
    A
    * =
    R,
    G,
    B,
    A
    

    View Slide

29. ColorMatrixColorFilter - Invert Image
    val canvas = Canvas(bitmap)
    val paint = Paint()
    paint.colorFilter = ColorMatrixColorFilter(
    ColorMatrix(floatArrayOf(
    -1f, 0f, 0f, 0f, 255f,
    0f, -1f, 0f, 0f, 255f,
    0f, 0f, -1f, 0f, 255f,
    0f, 0f, 0f, 1f, 0f
    )))
    canvas.drawBitmap(original, 0, 0, paint)
    

    View Slide

30. ColorMatrixColorFilter - Invert Image
    

    View Slide

31. What else can I do with it?
    Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia
    

    View Slide

32. Other ColorFilters
    - LightingColorFilter
    - PorterDuffColorFilter
    More info: Manipulating images and Drawables
    with Android’s ColorFilter - Nick Rout
    bit.ly/2Cwvx8t
    

    View Slide

33. Pros %
    • Easy-ish to
    understand
    • Fast (GPU)
    • Write in Kotlin/
    Java
    • Understand the
    exact effect
    being applied
    Cons &
    • Can’t do all
    effects like
    Sharpen, Unsharp
    mask etc
    

    View Slide

34. ImageFilterView
    

    View Slide

35. ImageFilterView
    Part of ConstraintLayout 2.0
    https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView
    

    View Slide

36. ImageFilterView
    android:layout_width="0dp"
    android:layout_height="0dp"
    android:id="@+id/imageView"
    app:warmth="1.2"
    app:contrast="1.0"
    app:saturation="2.0"
    app:layout_constraintTop_toTopOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintDimensionRatio="16:9"
    tools:srcCompat="@tools:sample/avatars"
    />
    

    View Slide

37. ImageFilterView
    

    View Slide

38. Pros %
    • Easy to understand
    • Fast (GPU)
    • Nice API
    • Java/Kotlin
    Cons &
    • Can’t do everything
    • Values are
    abstracted away
    from you
    

    View Slide

39. Glide
    

    View Slide

40. Glide Image Transformations
    Additional to Glide
    https://github.com/wasabeef/glide-transformations
    GlideApp.with(this)
    .load(url)
    .apply(bitmapTransform(new BlurTransformation(25, 3)))
    .into(imageView)
    

    View Slide

41. Pros %
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons &
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    

    View Slide

42. GPUImage
    

    View Slide

43. GPUImage
    - https://github.com/CyberAgent/android-gpuimage
    gpuImage = GPUImage(this)
    gpuImage.setGLSurfaceView(surfaceView)
    gpuImage.setImage(imageUri)
    gpuImage.setFilter(new GPUImageSepiaFilter())
    

    View Slide

44. Pros %
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons &
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    • Dependent on OpenGL
    code
    

    View Slide

45. Renderscript
    

    View Slide

46. What is Renderscript?
    • Framework on Android for computationally
    expensive tasks
    • Parallelises tasks across CPU and GPU
    • Language: C99-derived language for writing
    high-performance compute code
    

    View Slide

47. Renderscript Compute vs Graphics
    • Graphics - Deprecated
    • Compute - still widely used for image
    processing
    

    View Slide

48. Scripts
    

    View Slide

49. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    

    View Slide

50. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    

    View Slide

51. static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    out->g = clamp((int)(bright * in->g), 0, 255);
    out->b = clamp((int)(bright * in->b), 0, 255);
    }
    

    View Slide

52. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    

    View Slide

53. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    

    View Slide

54. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    

    View Slide

55. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    

    View Slide

56. script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    fun cleanup(){
    renderScript.destroy()
    script.destroy()
    }
    }
    

    View Slide

57. View Slide

58. Convolution Matrix
    

    View Slide

59. 0, -1, 0
    -1, 5, -1,
    0, -1, 0
    Sharpen an Image
    Sharpen Kernel
    2, 30, 1
    2, 40, 20,
    4, 20, 5
    Pixel in an Image with
    surrounding pixel values
    Resulting Pixel value
    = (0 * 2) + (-1 * 30) + (0 * 1)
    +(-1 * 2) + (5 * 40) + (-1* 20)
    +(0 * 4) + (-1 * 20) + (0 * 5)
    = 128
    

    View Slide

60. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    

    View Slide

61. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    

    View Slide

62. 0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    

    View Slide

63. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    }
    }
    

    View Slide

64. View Slide

65. Built-in Renderscript functions
    • ScriptIntrinsicBlur - Blurring
    • ScriptIntrinsicLUT - Lookup table
    • Many more…
    

    View Slide

66. Further Examples
    • Efficient Image Processing - Nicolas
    Roard https://youtu.be/j2szUDjScOw
    • Basic Renderscript - https://
    github.com/googlesamples/android-
    BasicRenderScript
    • Gallery2 - Renderscript Examples
    https://android.googlesource.com/
    platform/packages/apps/Gallery2
    

    View Slide

67. Pros %
    • Fast
    • Highly
    customisable
    • Parallel execution
    on GPU
    • Interactions
    between layers
    isn’t complex
    Cons &
    • Need to understand
    a new language
    • Manage your own
    memory
    • Native Libs
    

    View Slide

68. What should I use?
    

    View Slide

69. How do I choose?
    
    ColorFilter / Renderscript /
    OpenGL?
    +
    ImageFilterView or
    Glide Transformations
    

    View Slide

70. Final Tips
    

    View Slide

71. Tips
    
    OutOfMemory
    ↕
    Scale to display
    
    Use what is
    easiest for you
    

    View Slide

72. Thank you!
    Rebecca Franks
    @riggaroo
    hello@riggaroo.co.za
    

    View Slide