Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Rebecca Franks] Practical Image Processing in Android

[Rebecca Franks] Practical Image Processing in Android

Presentation from GDG DevFest Ukraine 2018 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

__

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Google Developers Group Lviv

October 13, 2018
Tweet

More Decks by Google Developers Group Lviv

Other Decks in Technology

Transcript

  1. Practical Image Processing
    on Android
    @riggaroo

    View full-size slide

  2. Rebecca Franks
    @riggaroo
    riggaroo.co.za
    Android @ Over

    View full-size slide

  3. Challenges
    • Process multiple images
    • Allow for “stacking” of
    image effects
    • Complex image processing
    • Storing filter values
    • Custom Canvas Sizes

    View full-size slide

  4. Lessons learnt

    Rendering project
    size

    Loading bitmaps
    synchronously

    Many different
    Image processing
    tools/libraries

    View full-size slide

  5. What is in a Bitmap?
    2D Array of Pixels
    Pixel = [R, G, B, A]
    (red, green, blue, alpha)
    Width
    Height

    View full-size slide

  6. Loading
    images

    View full-size slide

  7. How do I load up an Image?
    • Manually with BitmapFactory
    • Glide
    • Picasso
    • Fresco
    • …

    View full-size slide

  8. val options = BitmapFactory.Options()
    options.inJustDecodeBounds = true
    BitmapFactory.decodeResource(res, resId, options)
    options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight)
    options.inJustDecodeBounds = false
    return BitmapFactory.decodeResource(res, resId, options)
    BitmapFactory
    https://developer.android.com/topic/performance/graphics/load-bitmap

    View full-size slide

  9. Glide
    GlideApp.with(context)
    .load(“http://goo.gl/gEgYUd")
    .into(imageView)
    https://github.com/bumptech/glide

    View full-size slide

  10. Glide
    GlideApp.with(context)
    .asBitmap()
    .load(file)
    .submit(size.width, size.height)

    View full-size slide

  11. OutOfMemory
    Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228
    byte allocation with 4194304 free bytes and 4MB until OOM
    at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java)
    at android.graphics.Bitmap.nativeCreate(Bitmap.java)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:977)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:948)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:915)

    View full-size slide

  12. Canvas
    Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too
    large(283435200bytes) bitmap.
    at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas.java:260)
    at android.graphics.Canvas.drawBitmap(Canvas.java:1373)

    View full-size slide

  13. Scaling Images
    Height = 1920px
    Width = 700px
    1920px * 700px
    = 1 344 000 * 4 bytes per pixel (ARGB_8888)
    = 5 376 000 bytes
    = 5.375 mb
    Height = 128px
    Width = 50px
    = s0.0256mb

    View full-size slide

  14. Applying
    Effects

    View full-size slide

  15. Applying Image Effects
    • getPixel() & setPixel()
    • Glide Image Transformations GPUImage
    • ColorMatrix
    • Renderscript
    • OpenGL
    • Native (C++)

    View full-size slide

  16. getPixel() &
    setPixel()

    View full-size slide

  17. setPixel()
    override fun filter(bitmap: Bitmap): Bitmap {
    val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true)
    for (i in 0 until bitmap.width){
    for (j in 0 until bitmap.height){
    newBitmap.setPixel(i, j, Color.BLUE)
    }
    }
    return newBitmap
    }

    View full-size slide

  18. Pros $
    • Easy to understand
    • Manipulate per
    Pixel manually
    Cons %
    • Suuuuuper Slow
    • CPU Execution
    • No parallel
    execution

    View full-size slide

  19. What is a Color Matrix Filter?
    Modifies the color of each pixel
    drawn with a certain Paint.

    View full-size slide

  20. ColorMatrixColorFilter
    4 * 5 matrix applied to each pixel
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t

    View full-size slide

  21. ColorMatrix
    4 * 5 matrix applied to each pixel
    R’ = a*R + b*G + c*B + d*A + e;
    G’ = f*R + g*G + h*B + i*A + j;
    B’ = k*R + l*G + m*B + n*A + o;
    A’ = p*R + q*G + r*B + s*A + t;
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    [R’, G’, B’, A’]

    View full-size slide

  22. ColorMatrix
    Identity Matrix - same image in as out
    1, 0, 0, 0, 0,
    0, 1, 0, 0, 0,
    0, 0, 1, 0, 0,
    0, 0, 0, 1, 0
    R,
    G,
    B,
    A
    * =
    R,
    G,
    B,
    A

    View full-size slide

  23. ColorMatrixColorFilter - Invert Image
    val canvas = Canvas(bitmap)
    val paint = Paint()
    paint.colorFilter = ColorMatrixColorFilter(
    ColorMatrix(floatArrayOf(
    -1f, 0f, 0f, 0f, 255f,
    0f, -1f, 0f, 0f, 255f,
    0f, 0f, -1f, 0f, 255f,
    0f, 0f, 0f, 1f, 0f
    )))
    canvas.drawBitmap(original, 0, 0, paint)

    View full-size slide

  24. ColorMatrixColorFilter - Invert Image

    View full-size slide

  25. What else can I do with it?
    Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia

    View full-size slide

  26. Other ColorFilters
    - LightingColorFilter
    - PorterDuffColorFilter
    More info: Manipulating images and Drawables
    with Android’s ColorFilter - Nick Rout
    bit.ly/2Cwvx8t

    View full-size slide

  27. Pros $
    • Easy-ish to
    understand
    • Fast (GPU)
    • Write in Kotlin/
    Java
    • Understand the
    exact effect
    being applied
    Cons %
    • Can’t do all
    effects like
    Sharpen etc

    View full-size slide

  28. ImageFilterVie
    w

    View full-size slide

  29. ImageFilterView
    Part of ConstraintLayout 2.0
    https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

    View full-size slide

  30. ImageFilterView
    android:layout_width="0dp"
    android:layout_height="0dp"
    android:id="@+id/imageView"
    app:warmth="1.2"
    app:contrast="1.0"
    app:saturation="2.0"
    app:layout_constraintTop_toTopOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintDimensionRatio="16:9"
    tools:srcCompat="@tools:sample/avatars"
    />

    View full-size slide

  31. ImageFilterView

    View full-size slide

  32. Pros $
    • Easy to understand
    • Fast (GPU)
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you

    View full-size slide

  33. Glide Image Transformations
    Additional to Glide
    https://github.com/wasabeef/glide-transformations
    GlideApp.with(this)
    .load(url)
    .apply(bitmapTransform(new BlurTransformation(25, 3)))
    .into(imageView)

    View full-size slide

  34. Pros $
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party

    View full-size slide

  35. GPUImage
    - https://github.com/CyberAgent/android-gpuimage
    gpuImage = GPUImage(this)
    gpuImage.setGLSurfaceView(surfaceView)
    gpuImage.setImage(imageUri)
    gpuImage.setFilter(new GPUImageSepiaFilter())

    View full-size slide

  36. Pros $
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    • Dependent on OpenGL
    code

    View full-size slide

  37. Renderscript

    View full-size slide

  38. What is Renderscript?
    • Framework on Android for computationally
    expensive tasks
    • Parallelises tasks across CPU and GPU
    • Language: C99-derived language for writing
    high-performance compute code
    https://developer.android.com/guide/topics/renderscript/compute

    View full-size slide

  39. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }

    View full-size slide

  40. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);

    View full-size slide

  41. static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    out->g = clamp((int)(bright * in->g), 0, 255);
    out->b = clamp((int)(bright * in->b), 0, 255);
    }

    View full-size slide

  42. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)

    View full-size slide

  43. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)

    View full-size slide

  44. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }

    View full-size slide

  45. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }

    View full-size slide

  46. script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    fun cleanup(){
    renderScript.destroy()
    script.destroy()
    }
    }

    View full-size slide

  47. Convolution
    Matrix

    View full-size slide

  48. 0, -1, 0
    -1, 5, -1,
    0, -1, 0
    Sharpen an Image
    Sharpen Kernel
    2, 30, 1
    2, 40, 20,
    4, 20, 5
    Pixel in an Image with
    surrounding pixel values
    Resulting Pixel value
    = (0 * 2) + (-1 * 30) + (0 * 1)
    +(-1 * 2) + (5 * 40) + (-1* 20)
    +(0 * 4) + (-1 * 20) + (0 * 5)
    = 128

    View full-size slide

  49. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }

    View full-size slide

  50. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)

    View full-size slide

  51. 0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()

    View full-size slide

  52. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    }
    }

    View full-size slide

  53. Built-in Renderscript functions
    • ScriptIntrinsicBlur - Blurring
    • ScriptIntrinsicLUT - Lookup table
    • Many more…

    View full-size slide

  54. Further Examples
    • Efficient Image Processing - Nicolas
    Roard https://youtu.be/j2szUDjScOw
    • Gallery2 - Renderscript Examples
    https://android.googlesource.com/
    platform/packages/apps/Gallery2

    View full-size slide

  55. Pros $
    • Fast
    • Highly
    customisable
    • Parallel execution
    on GPU
    • Interactions
    between layers
    isn’t complex
    Cons %
    • Need to understand
    a new language
    • Manage your own
    memory
    • Native Libs

    View full-size slide

  56. What should I use?

    View full-size slide

  57. How do I choose?

    ColorFilter / Renderscript /
    OpenGL?
    *
    ImageFilterView or
    Glide Transformations

    View full-size slide

  58. Tips

    OutOfMemory

    Scale to display

    Use what is
    easiest for you

    View full-size slide

  59. Thank you!
    Rebecca Franks
    @riggaroo
    [email protected]

    View full-size slide