Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Rebecca Franks] Practical Image Processing in Android

[Rebecca Franks] Practical Image Processing in Android

Presentation from GDG DevFest Ukraine 2018 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

__

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Google Developers Group Lviv

October 13, 2018
Tweet

More Decks by Google Developers Group Lviv

Other Decks in Technology

Transcript

  1. Practical Image Processing
    on Android
    @riggaroo

    View Slide

  2. Rebecca Franks
    @riggaroo
    riggaroo.co.za
    Android @ Over

    View Slide

  3. View Slide

  4. Challenges
    • Process multiple images
    • Allow for “stacking” of
    image effects
    • Complex image processing
    • Storing filter values
    • Custom Canvas Sizes

    View Slide

  5. Lessons learnt

    Rendering project
    size

    Loading bitmaps
    synchronously

    Many different
    Image processing
    tools/libraries

    View Slide

  6. What is in a Bitmap?
    2D Array of Pixels
    Pixel = [R, G, B, A]
    (red, green, blue, alpha)
    Width
    Height

    View Slide

  7. Loading
    images

    View Slide

  8. How do I load up an Image?
    • Manually with BitmapFactory
    • Glide
    • Picasso
    • Fresco
    • …

    View Slide

  9. val options = BitmapFactory.Options()
    options.inJustDecodeBounds = true
    BitmapFactory.decodeResource(res, resId, options)
    options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight)
    options.inJustDecodeBounds = false
    return BitmapFactory.decodeResource(res, resId, options)
    BitmapFactory
    https://developer.android.com/topic/performance/graphics/load-bitmap

    View Slide

  10. Glide
    GlideApp.with(context)
    .load(“http://goo.gl/gEgYUd")
    .into(imageView)
    https://github.com/bumptech/glide

    View Slide

  11. Glide
    GlideApp.with(context)
    .asBitmap()
    .load(file)
    .submit(size.width, size.height)

    View Slide

  12. OutOfMemory

    View Slide

  13. OutOfMemory
    Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228
    byte allocation with 4194304 free bytes and 4MB until OOM
    at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java)
    at android.graphics.Bitmap.nativeCreate(Bitmap.java)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:977)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:948)
    at android.graphics.Bitmap.createBitmap(Bitmap.java:915)

    View Slide

  14. Canvas
    Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw too
    large(283435200bytes) bitmap.
    at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas.java:260)
    at android.graphics.Canvas.drawBitmap(Canvas.java:1373)

    View Slide

  15. Scaling Images
    Height = 1920px
    Width = 700px
    1920px * 700px
    = 1 344 000 * 4 bytes per pixel (ARGB_8888)
    = 5 376 000 bytes
    = 5.375 mb
    Height = 128px
    Width = 50px
    = s0.0256mb

    View Slide

  16. Applying
    Effects

    View Slide

  17. View Slide

  18. Applying Image Effects
    • getPixel() & setPixel()
    • Glide Image Transformations GPUImage
    • ColorMatrix
    • Renderscript
    • OpenGL
    • Native (C++)

    View Slide

  19. getPixel() &
    setPixel()

    View Slide

  20. setPixel()
    override fun filter(bitmap: Bitmap): Bitmap {
    val newBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true)
    for (i in 0 until bitmap.width){
    for (j in 0 until bitmap.height){
    newBitmap.setPixel(i, j, Color.BLUE)
    }
    }
    return newBitmap
    }

    View Slide

  21. Pros $
    • Easy to understand
    • Manipulate per
    Pixel manually
    Cons %
    • Suuuuuper Slow
    • CPU Execution
    • No parallel
    execution

    View Slide

  22. ColorFilter

    View Slide

  23. What is a Color Matrix Filter?
    Modifies the color of each pixel
    drawn with a certain Paint.

    View Slide

  24. ColorMatrixColorFilter
    4 * 5 matrix applied to each pixel
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t

    View Slide

  25. ColorMatrix
    4 * 5 matrix applied to each pixel
    R’ = a*R + b*G + c*B + d*A + e;
    G’ = f*R + g*G + h*B + i*A + j;
    B’ = k*R + l*G + m*B + n*A + o;
    A’ = p*R + q*G + r*B + s*A + t;
    a, b, c, d, e,
    f, g, h, i, j,
    k, l, m, n, o,
    p, q, r, s, t
    [R’, G’, B’, A’]

    View Slide

  26. ColorMatrix
    Identity Matrix - same image in as out
    1, 0, 0, 0, 0,
    0, 1, 0, 0, 0,
    0, 0, 1, 0, 0,
    0, 0, 0, 1, 0
    R,
    G,
    B,
    A
    * =
    R,
    G,
    B,
    A

    View Slide

  27. ColorMatrixColorFilter - Invert Image
    val canvas = Canvas(bitmap)
    val paint = Paint()
    paint.colorFilter = ColorMatrixColorFilter(
    ColorMatrix(floatArrayOf(
    -1f, 0f, 0f, 0f, 255f,
    0f, -1f, 0f, 0f, 255f,
    0f, 0f, -1f, 0f, 255f,
    0f, 0f, 0f, 1f, 0f
    )))
    canvas.drawBitmap(original, 0, 0, paint)

    View Slide

  28. ColorMatrixColorFilter - Invert Image

    View Slide

  29. What else can I do with it?
    Warmth, Exposure, Saturation, Contrast, Grayscale, Sepia

    View Slide

  30. Other ColorFilters
    - LightingColorFilter
    - PorterDuffColorFilter
    More info: Manipulating images and Drawables
    with Android’s ColorFilter - Nick Rout
    bit.ly/2Cwvx8t

    View Slide

  31. Pros $
    • Easy-ish to
    understand
    • Fast (GPU)
    • Write in Kotlin/
    Java
    • Understand the
    exact effect
    being applied
    Cons %
    • Can’t do all
    effects like
    Sharpen etc

    View Slide

  32. ImageFilterVie
    w

    View Slide

  33. ImageFilterView
    Part of ConstraintLayout 2.0
    https://developer.android.com/reference/android/support/constraint/utils/ImageFilterView

    View Slide

  34. ImageFilterView
    android:layout_width="0dp"
    android:layout_height="0dp"
    android:id="@+id/imageView"
    app:warmth="1.2"
    app:contrast="1.0"
    app:saturation="2.0"
    app:layout_constraintTop_toTopOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintDimensionRatio="16:9"
    tools:srcCompat="@tools:sample/avatars"
    />

    View Slide

  35. ImageFilterView

    View Slide

  36. Pros $
    • Easy to understand
    • Fast (GPU)
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you

    View Slide

  37. Glide

    View Slide

  38. Glide Image Transformations
    Additional to Glide
    https://github.com/wasabeef/glide-transformations
    GlideApp.with(this)
    .load(url)
    .apply(bitmapTransform(new BlurTransformation(25, 3)))
    .into(imageView)

    View Slide

  39. Pros $
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party

    View Slide

  40. GPUImage

    View Slide

  41. GPUImage
    - https://github.com/CyberAgent/android-gpuimage
    gpuImage = GPUImage(this)
    gpuImage.setGLSurfaceView(surfaceView)
    gpuImage.setImage(imageUri)
    gpuImage.setFilter(new GPUImageSepiaFilter())

    View Slide

  42. Pros $
    • Easy to understand
    • Fast
    • Nice API
    • Java/Kotlin
    Cons %
    • Can’t do everything
    • Values are
    abstracted away
    from you
    • Dependant on Third
    Party
    • Dependent on OpenGL
    code

    View Slide

  43. Renderscript

    View Slide

  44. What is Renderscript?
    • Framework on Android for computationally
    expensive tasks
    • Parallelises tasks across CPU and GPU
    • Language: C99-derived language for writing
    high-performance compute code
    https://developer.android.com/guide/topics/renderscript/compute

    View Slide

  45. Scripts

    View Slide

  46. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }

    View Slide

  47. #pragma version(1)
    #pragma rs java_package_name(app.rigs.renderscriptfun)
    #pragma rs_fp_full
    static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);

    View Slide

  48. static float bright = 0.f;
    void setBright(float v) {
    bright = 255.f / (255.f - v);
    }
    void exposure(const uchar4 *in, uchar4 *out)
    {
    out->r = clamp((int)(bright * in->r), 0, 255);
    out->g = clamp((int)(bright * in->g), 0, 255);
    out->b = clamp((int)(bright * in->b), 0, 255);
    }

    View Slide

  49. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)

    View Slide

  50. class ExposureFilter(val context: Context) {
    private val renderScript = RenderScript.create(context)
    private val script = ScriptC_exposure(renderScript)
    fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)

    View Slide

  51. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }

    View Slide

  52. fun filter(exposure: Float, bitmap: Bitmap): Bitmap {
    val outputBitmap = Bitmap.createBitmap(bitmap)
    val tmpIn = Allocation.createFromBitmap(renderScript,
    bitmap)
    val tmpOut = Allocation.createFromBitmap(renderScript,
    outputBitmap)
    script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }

    View Slide

  53. script.invoke_setBright(exposure)
    script.forEach_exposure(tmpIn, tmpOut)
    tmpOut.copyTo(outputBitmap)
    tmpIn.destroy()
    tmpOut.destroy()
    return outputBitmap
    }
    fun cleanup(){
    renderScript.destroy()
    script.destroy()
    }
    }

    View Slide

  54. View Slide

  55. Convolution
    Matrix

    View Slide

  56. 0, -1, 0
    -1, 5, -1,
    0, -1, 0
    Sharpen an Image
    Sharpen Kernel
    2, 30, 1
    2, 40, 20,
    4, 20, 5
    Pixel in an Image with
    surrounding pixel values
    Resulting Pixel value
    = (0 * 2) + (-1 * 30) + (0 * 1)
    +(-1 * 2) + (5 * 40) + (-1* 20)
    +(0 * 4) + (-1 * 20) + (0 * 5)
    = 128

    View Slide

  57. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }

    View Slide

  58. class SharpenImageFilter(context: Context, val multiplier: Float) {
    val renderScript = RenderScript.create(context)
    val convolution = ScriptIntrinsicConvolve3x3.create(renderScript,
    Element.U8_4(renderScript))
    override fun applyFilter(bitmap: Bitmap): Bitmap {
    val matrixSharpen = floatArrayOf(
    0f, -multiplier, 0f,
    -multiplier, 1 + 4 * multiplier, -multiplier,
    0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)

    View Slide

  59. 0f, -multiplier, 0f)
    return applyFilterConvolve(bitmap, matrixSharpen)
    }
    fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()

    View Slide

  60. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap {
    val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width,
    bitmapIn.height, bitmapIn.config)
    val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn)
    val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut)
    convolution.setInput(allocationIn)
    convolution.setCoefficients(coefficients)
    convolution.forEach(allocationOut)
    allocationOut.copyTo(bitmapOut)
    allocationIn.destroy()
    allocationOut.destroy()
    return bitmapOut
    }
    fun cleanup(){
    renderScript.destroy()
    }
    }

    View Slide

  61. View Slide

  62. Built-in Renderscript functions
    • ScriptIntrinsicBlur - Blurring
    • ScriptIntrinsicLUT - Lookup table
    • Many more…

    View Slide

  63. Further Examples
    • Efficient Image Processing - Nicolas
    Roard https://youtu.be/j2szUDjScOw
    • Gallery2 - Renderscript Examples
    https://android.googlesource.com/
    platform/packages/apps/Gallery2

    View Slide

  64. Pros $
    • Fast
    • Highly
    customisable
    • Parallel execution
    on GPU
    • Interactions
    between layers
    isn’t complex
    Cons %
    • Need to understand
    a new language
    • Manage your own
    memory
    • Native Libs

    View Slide

  65. What should I use?

    View Slide

  66. How do I choose?

    ColorFilter / Renderscript /
    OpenGL?
    *
    ImageFilterView or
    Glide Transformations

    View Slide

  67. Final Tips

    View Slide

  68. Tips

    OutOfMemory

    Scale to display

    Use what is
    easiest for you

    View Slide

  69. Thank you!
    Rebecca Franks
    @riggaroo
    [email protected]

    View Slide