Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Image Processing on Android - DevFest Ukraine 2018

Practical Image Processing on Android - DevFest Ukraine 2018

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Rebecca Franks

October 13, 2018
Tweet

More Decks by Rebecca Franks

Other Decks in Programming

Transcript

  1. On Android
    Rebecca Franks / @riggaroo
    Google Developer Expert
    Practical Image Processing

    View full-size slide

  2. Rebecca Franks
    @riggaroo
    riggaroo.co.za
    Google Developer Expert

    View full-size slide

  3. Challenges
    • Process multiple images
    • Complex image processing
    • Storing filters
    • Custom Canvas Sizes
    • An iOS app that had the
    filters already

    View full-size slide

  4. Mistakes we made

    Rendering project
    size vs display size

    Loading bitmaps
    synchronously

    Many different
    Image processing
    tools/libraries

    View full-size slide

  5. What is a Bitmap?
    2D Array of Pixels
    Pixel = [R, G, B, A]
    (red, green, blue, alpha)
    Width
    Height

    View full-size slide

  6. Loading a
    Bitmap

    View full-size slide

  7. How do I load up a Bitmap?
    • Manually with BitmapFactory
    • Glide
    • Picasso
    • Fresco
    • …

    View full-size slide

  8. val options = BitmapFactory.Options()
    options.inJustDecodeBounds = true
    BitmapFactory.decodeResource(res, resId, options)
    options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight)
    options.inJustDecodeBounds = false
    return BitmapFactory.decodeResource(res, resId, options)
    BitmapFactory
    https://developer.android.com/topic/performance/graphics/load-bitmap

    View full-size slide

  9. Glide
    GlideApp.with(context)
    .load(“http://goo.gl/gEgYUd")
    .into(imageView)
    https://github.com/bumptech/glide

    View full-size slide

  10. OutOfMemory
    Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a
    16777228 byte allocation with 4194304 free bytes and 4MB until OOM
    at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java)
    at android.graphics.Bitmap.nativeCreate(Bitmap.java)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:977)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:948)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:915)

    View full-size slide

  11. Bitmap too large
    Fatal Exception: java.lang.RuntimeException: Canvas: trying to
    draw too large(283435200bytes) bitmap.
    at
    android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas
    .java:260)
    at android.graphics.Canvas.drawBitmap(Canvas.java:1373)

    View full-size slide

  12. You need to scale your images
    Height = 1920px
    Width = 700px
    1920px * 700px
    = 1 344 000 * 4 bytes per pixel
    = 5 376 000 bytes
    = 5.375 mb
    Height = 128px
    Width = 50px
    = 0.0256mb

    View full-size slide

  13. How do I scale down my image?
    val options = BitmapFactory.Options()
    options.inSampleSize = calculateInSampleSize(options,
    reqWidth, reqHeight)
    BitmapFactory.decodeResource(res, resId, options)

    View full-size slide

  14. Glide
    GlideApp.with(context)
    .load(file)
    .into(imageView)

    View full-size slide

  15. Glide
    GlideApp.with(context)
    .asBitmap()
    .load(file)
    .submit(size.width, size.height)

    View full-size slide

  16. Applying
    Effects

    View full-size slide

  17. Applying Image Effects
    • getPixel() & setPixel()
    • Glide Image Transformations / GPUImage
    • ColorMatrix
    • Renderscript
    • OpenGL
    • Native (C++)
    • …


    View full-size slide

  18. getPixel() &
    setPixel()

    View full-size slide

  19. setPixel()
    override fun filter(bitmap: Bitmap): Bitmap {
    val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true)
    for (i in 0 until bitmap.width){
    for (j in 0 until bitmap.height){
    newBitmap.setPixel(i, j, Color.BLUE)
    }
    }
    return newBitmap
    }

    View full-size slide

  20. Pros %
    • Easy to understand
    • Manipulate per
    Pixel manually
    Cons &
    • Suuuuuper Slow
    • CPU Execution
    • No parallel
    execution

    View full-size slide

  21. What is a Color Matrix Filter?
    Modifies the color of each pixel
    drawn with a certain Paint.

    View full-size slide

  22. ColorMatrixColorFilter
    4 * 5 matrix applied to each pixel
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t

    View full-size slide

  23. ColorMatrix
    4 * 5 matrix applied to each pixel
    R’ = a*R + b*G + c*B + d*A + e;
    G’ = f*R + g*G + h*B + i*A + j;
    B’ = k*R + l*G + m*B + n*A + o;
    A’ = p*R + q*G + r*B + s*A + t;
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    [R’, G’, B’, A’]

    View full-size slide

  24. ColorMatrix
    Identity Matrix - same image in as out
    1, 0, 0, 0, 0,
    0, 1, 0, 0, 0,
    0, 0, 1, 0, 0,
    0, 0, 0, 1, 0
    R,
    G,
    B,
    A
    * =
    R,
    G,
    B,
    A

    View full-size slide

  25. ColorMatrixColorFilter - Invert Image
    val canvas = Canvas(bitmap)
    val paint = Paint()
    paint.colorFilter = ColorMatrixColorFilter(
    ColorMatrix(floatArrayOf(
    -1f, 0f, 0f, 0f, 255f,
    0f, -1f, 0f, 0f, 255f,
    0f, 0f, -1f, 0f, 255f,
    0f, 0f, 0f, 1f, 0f
    )))
    canvas.drawBitmap(original, 0, 0, paint)

    View full-size slide

  26. ColorMatrixColorFilter - Invert Image

    View full-size slide

  27. What else can I do with it?
    Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia

    View full-size slide

  28. Other ColorFilters
    - LightingColorFilter
    - PorterDuffColorFilter
    More info: Manipulating images and Drawables
    with Android’s ColorFilter - Nick Rout
    bit.ly/2Cwvx8t

    View full-size slide

  29. Pros %
    • Easy-ish to
    understand
    • Fast (GPU)
    • Write in Kotlin/
    Java
    • Understand the
    exact effect
    being applied
    Cons &
    • Can’t do all
    effects like
    Sharpen, Unsharp
    mask etc

    View full-size slide

  30. ImageFilterView

    View full-size slide

  31. ImageFilterView
    Part of ConstraintLayout 2.0
    https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

    View full-size slide

  32. ImageFilterView
    android:layout_width="0dp"
    android:layout_height="0dp"
    android:id="@+id/imageView"
    app:warmth="1.2"
    app:contrast="1.0"
    app:saturation="2.0"
    app:layout_constraintTop_toTopOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintDimensionRatio="16:9"
    tools:srcCompat="@tools:sample/avatars"
    />

    View full-size slide

  33. ImageFilterView

    View full-size slide

  34. Pros %
    • Easy to understand
    • Fast (GPU)
    • Nice API
    • Java/Kotlin
    Cons &
    • Can’t do everything
    • Values are
    abstracted away
    from you

    View full-size slide

  35. Glide Image Transformations
    Additional to Glide
    https://github.com/wasabeef/glide-transformations
    GlideApp.with(this)
    .load(url)
    .apply(bitmapTransform(new BlurTransformation(25, 3)))
    .into(imageView)

    View full-size slide

  36. Pros %
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons &
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party

    View full-size slide

  37. GPUImage
    - https://github.com/CyberAgent/android-gpuimage
    gpuImage = GPUImage(this)
    gpuImage.setGLSurfaceView(surfaceView)
    gpuImage.setImage(imageUri)
    gpuImage.setFilter(new GPUImageSepiaFilter())

    View full-size slide

  38. Pros %
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons &
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    • Dependent on OpenGL
    code

    View full-size slide

  39. Renderscript

    View full-size slide

  40. What is Renderscript?
    • Framework on Android for computationally
    expensive tasks
    • Parallelises tasks across CPU and GPU
    • Language: C99-derived language for writing
    high-performance compute code

    View full-size slide

  41. Renderscript Compute vs Graphics
    • Graphics - Deprecated
    • Compute - still widely used for image
    processing

    View full-size slide

  42. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }

    View full-size slide

  43. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);

    View full-size slide

  44. static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    out->g = clamp((int)(bright * in->g), 0, 255);
    out->b = clamp((int)(bright * in->b), 0, 255);
    }

    View full-size slide

  45. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)

    View full-size slide

  46. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)

    View full-size slide

  47. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }

    View full-size slide

  48. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }

    View full-size slide

  49. script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    fun cleanup(){
    renderScript.destroy()
    script.destroy()
    }
    }

    View full-size slide

  50. Convolution Matrix

    View full-size slide

  51. 0, -1, 0
    -1, 5, -1,
    0, -1, 0
    Sharpen an Image
    Sharpen Kernel
    2, 30, 1
    2, 40, 20,
    4, 20, 5
    Pixel in an Image with
    surrounding pixel values
    Resulting Pixel value
    = (0 * 2) + (-1 * 30) + (0 * 1)
    +(-1 * 2) + (5 * 40) + (-1* 20)
    +(0 * 4) + (-1 * 20) + (0 * 5)
    = 128

    View full-size slide

  52. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }

    View full-size slide

  53. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)

    View full-size slide

  54. 0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()

    View full-size slide

  55. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    }
    }

    View full-size slide

  56. Built-in Renderscript functions
    • ScriptIntrinsicBlur - Blurring
    • ScriptIntrinsicLUT - Lookup table
    • Many more…

    View full-size slide

  57. Further Examples
    • Efficient Image Processing - Nicolas
    Roard https://youtu.be/j2szUDjScOw
    • Basic Renderscript - https://
    github.com/googlesamples/android-
    BasicRenderScript
    • Gallery2 - Renderscript Examples
    https://android.googlesource.com/
    platform/packages/apps/Gallery2

    View full-size slide

  58. Pros %
    • Fast
    • Highly
    customisable
    • Parallel execution
    on GPU
    • Interactions
    between layers
    isn’t complex
    Cons &
    • Need to understand
    a new language
    • Manage your own
    memory
    • Native Libs

    View full-size slide

  59. What should I use?

    View full-size slide

  60. How do I choose?

    ColorFilter / Renderscript /
    OpenGL?
    +
    ImageFilterView or
    Glide Transformations

    View full-size slide

  61. Tips

    OutOfMemory

    Scale to display

    Use what is
    easiest for you

    View full-size slide

  62. Thank you!
    Rebecca Franks
    @riggaroo
    [email protected]

    View full-size slide