A LUT(lot) of image Filters on Android

A LUT(lot) of image Filters on Android

In this talk, we will learn how to create image filters with LUTs (Lookup Tables) using Renderscript on Android. We will briefly look into various approaches one can take for creating filters on Android. We will then learn about the basics of LUTs and how we can use them with Renderscript for creating filters. Finally, we will focus on the challenges that we faced while implementing filters on Android with the Renderscript framework. We will try to use graphical representations wherever possible to explain all these concepts.

A3958eeb9a7f402b134c0c017d6614ee?s=128

ragdroid

July 02, 2019
Tweet

Transcript

  1. Droidcon Berlin 2019 @ragdroid Garima Jain A LUT(lot) of image

    filters on Android
  2. Droidcon Berlin 2019 @ragdroid A LUT(lot) of image filters on

    Android Garima Jain
  3. Droidcon Berlin 2019 @ragdroid Who?

  4. Who? • Image Processing on Android? • No Image Processing?

  5. Droidcon Berlin 2019 @ragdroid What?

  6. BeautyPlus Google Photos Instagram

  7. Adobe Lightroom

  8. Droidcon Berlin 2019 @ragdroid Different Approaches to Image Filters

  9. Different Approaches to Image Filters 1. Presets 2. LUT

  10. Presets { "saturation": 0, "sharpness": 73, "luminance": 50, "vignetteAmount": 10,

    "vignetteRadius": 1.0, "shadows": 0.0, "contrast": 23, ... }
  11. Adobe Lightroom

  12. Adobe Lightroom

  13. Presets • Achieve great results • Lots of customizable options

    • Do we even need it? • Ship Faster
  14. LUT (Lookup Table) • Replaces runtime computation with simpler array

    indexing operation • Different Types: ✦ 1D ✦ 3 x 1D ✦ 3D
  15. 1D LUT (Lookup Table) 0 0 1 85 2 169

    3 255 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 1D Identity LUT (index, color)
  16. 3 x 0 1 2 3 0 1 2 3

    0 1 2 3 1D LUT (Lookup Table) 1D Identity LUT 3 x 0 0 1 85 2 169 3 255
  17. 3D LUT (Lookup Table) (3,3,3) (0,0,0) R G B 3D

    Identity LUT (4 x 4 x 4)
  18. 3D LUT (Lookup Table) 0 0 0 0 0 1

    0 0 2 0 0 3 0 1 0 0 1 1 0 1 2 0 1 3 0 2 0 0 2 1 0 2 2 0 2 3 0 3 0 0 3 1 0 3 2 0 3 3 B G R 1 0 0 1 0 1 1 0 2 1 0 3 1 1 0 1 1 1 1 1 2 1 1 3 1 2 0 1 2 1 1 2 2 1 2 3 1 3 0 1 3 1 1 3 2 1 3 3 2 0 0 2 0 1 2 0 2 2 0 3 2 1 0 2 1 1 2 1 2 2 1 3 2 2 0 2 2 1 2 2 2 2 2 3 2 3 0 2 3 1 2 3 2 2 3 3 3 0 0 3 0 1 3 0 2 3 0 3 3 1 0 3 1 1 3 1 2 3 1 3 3 2 0 3 2 1 3 2 2 3 2 3 3 3 0 3 3 1 3 3 2 3 3 3 3D Identity LUT (4 x 4 x 4) B G R B G R B G R
  19. 3D LUT (Lookup Table) • 4 x 4 x 4

    is not enough • Practically 64 x 64 x 64 LUT images are used • 64 LUT images can map 262144 (64 ^3) colors
  20. 3D Square LUT (64 x 64 x 64)

  21. Designers Pipeline • Create Filters • Start with Identity LUT

    image • Apply effects to it • Export and create new LUTs
  22. Designers Pipeline

  23. Droidcon Berlin 2019 @ragdroid How?

  24. easyLUT Library

  25. easyLUT Library

  26. • getPixel() and setPixel() • Not Fast • Efficient Image

    Processing on Android - Nicolas Roard • Practical Image Processing in Android - Rebecca Franks easyLUT Library
  27. Droidcon Berlin 2019 @ragdroid Different Approaches to Image Processing

  28. Image Processing • In Java - getPixel() and setPixel() •

    Native Code (JNI calls to C/C++) • OpenGL • RenderScript
  29. Droidcon Berlin 2019 @ragdroid RenderScript

  30. RenderScript 1. Baked-in Intrinsics - ScriptIntrinsic3DLUT 2. Custom Script

  31. Droidcon Berlin 2019 @ragdroid 1. ScriptIntrinsic3DLUT

  32. ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a

    3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”
  33. ScriptIntrinsic3DLUT setLUT(Allocation lut) Sets the Allocation to be used as

    the lookup table
  34. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  35. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  36. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  37. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  38. createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString()

    createSized() createTyped() LUT3DParams - silvaren/easyrs
  39. Cube { int[] cube rSize int gSize int bSize int

    } LUT3DParams - silvaren/easyrs
  40. Cube Creation Cube { int[] cube rSize int gSize int

    bSize int }
  41. Cube Creation for (r in 0 until dimension) { for

    (g in 0 until dimension) { val p = r + g * w for (b in 0 until dimension) { val newPixel = pixels[p + b * h] lut[i++] = newPixel } } }
  42. ScriptIntrinsic3DLUT val lutScript = ScriptIntrinsic3DLUT.create(renderScript, Element.U8_4(renderScript)) lutScript.setLUT(allocation) lutScript.forEach(allocationIn, allocationOut)

  43. Cube Creation

  44. Droidcon Berlin 2019 @ragdroid Transparent Test

  45. Transparent Test Failed

  46. ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a

    3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”
  47. Droidcon Berlin 2019 @ragdroid 2. Custom Script

  48. Custom Script For each pixel value (R,G,B) find the corresponding

    value in LUT image and use that value instead
  49. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  50. Droidcon Berlin 2019 @ragdroid Lookup pixel in LUT

  51. Square LUT 512 x 512

  52. None
  53. LUTFilter • Blue : Square within the LUT • Green

    and Red : (x, y) within a Square
  54. LUTFilter • Example : Input Pixel (127, 127, 127) •

    64 LUTs can not process all 256 colors so we divide by 4. • divide (127, 127, 127) by 4 = (31, 31, 31) ★ Blue : Square within the LUT : 31st square ★ Green and Red : (x, y) within a Square : (31, 31) within the square
  55. 31st square

  56. R (31) G (31) (31, 31) within the square

  57. x Y

  58. Droidcon Berlin 2019 @ragdroid LUTFilter x = b % 8

    * 64 + r y = floor(b/8.0f) * 64 + g
  59. Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb =

    rsGetElementAt_uchar4(lut, x, y); Annd it works!
  60. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  61. Droidcon Berlin 2019 @ragdroid Transparent Test

  62. Transparent Test Again Failed

  63. Droidcon Berlin 2019 @ragdroid Pre-multiplied Alpha

  64. Droidcon Berlin 2019 @ragdroid The Curious case of Android premultiplied

    Alpha
  65. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r *

    (256 - src.a) + src.r * src.a dest.g = dest.g * (256 - src.a) + src.g * src.a dest.b = dest.b * (256 - src.a) + src.b * src.a
  66. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r *

    (256 - src.a) + src.premultipliedR dest.g = dest.g * (256 - src.a) + src.premultipliedG dest.b = dest.b * (256 - src.a) + src.premultipliedB
  67. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha (Issue) • When we

    lookup input pixel in LUT • Wrong value lookup
  68. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha: Solution • BitmapFactory.Options.inPremultiplied •

    Handle it yourself
  69. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  70. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  71. Droidcon Berlin 2019 @ragdroid Un-premultiply Alpha float div = in.a

    / 256.0; modifiedR = clamp(in.r / div, 0.0, 255.0); modifiedG = clamp(in.g / div, 0.0, 255.0); modifiedB = clamp(in.b / div, 0.0, 255.0);
  72. Droidcon Berlin 2019 @ragdroid LUTFilter x = modifiedB % 8

    * 64 + modifiedR y = floor(modifiedB/8.0f) * 64 + modifiedG
  73. Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb =

    rsGetElementAt_uchar4(lut, x, y);
  74. Transparent Test Passed!

  75. Transparent Test Passed!

  76. LUTFilter

  77. Droidcon Berlin 2019 @ragdroid Banding & Interpolation

  78. Banding

  79. Banding

  80. Banding Input Image LUT

  81. Banding Input Image LUT R : 0 - 255 G:

    0 - 255 B: 0 - 255 R : 0 - 63 G: 0 - 63 B: 0 - 63 (255 ^ 3 = 16581375 colors) (64 ^ 3 = 262144 colors)
  82. Droidcon Berlin 2019 @ragdroid Banding x = (b / 4.0)

    % 8 * 64 + r/4.0 y = floor((b/4.0)/8.0f) * 64 + g/4.0
  83. 64 LUT Limitations • 64 LUT images can map 262144

    (64 ^3) colors • Jpg / png images have 16777216 (256 ^3) colors! • Lossy Technique
  84. Droidcon Berlin 2019 @ragdroid Interpolation “ Interpolation is a method

    of constructing new data points within the range of A discrete set of known data points. ” — Wikipedia
  85. Droidcon Berlin 2019 @ragdroid Linear Interpolation (x,y) x0 x1 y0

    y1
  86. Droidcon Berlin 2019 @ragdroid Trilinear Interpolation Source: Computational Color Technology,

    Henry R. Kang
  87. Droidcon Berlin 2019 @ragdroid Trilinear Interpolation • Source: Computational Color

    Technology, Henry R. Kang • Wikipedia : https://en.wikipedia.org/wiki/Trilinear_interpolation
  88. Without Interpolation : Banding

  89. With Interpolation: Reduced Banding

  90. It Works!

  91. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  92. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate
  93. Droidcon Berlin 2019 @ragdroid Other Issues

  94. Droidcon Berlin 2019 @ragdroid Renderscript Stack Trace

  95. 2 days later

  96. Some utilities • Ndk-stack : https://developer.android.com/ndk/guides/ndk-stack • AddressSanitizerOnAndroid : https://github.com/google/sanitizers/wiki/

    AddressSanitizerOnAndroid
  97. 2 more days later ArrayIndexOutOfBounds

  98. Droidcon Berlin 2019 @ragdroid LUT Formats

  99. Expectation Reality

  100. Hald LUT 512 x 512

  101. None
  102. Droidcon Berlin 2019 @ragdroid LUTFilter (HALD) x = g %

    8 * 64 + r y = floor(g/8.0f) + b * 8
  103. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate
  104. Droidcon Berlin 2019 @ragdroid LUTFilter (HALD)

  105. Droidcon Berlin 2019 @ragdroid Finally

  106. References • Computational Color Technology, Henry R. Kang (Interpolation) •

    easyLUT Library • silvaren/easyrs (LUT3DParams) • Renderscript docs • The Curious case of Android premultiplied Alpha (pre-multiplied Alpha
  107. What Next? • Building Filters with Go - Wayne Ashley

    Berry • .cube file format • Improve speed even further • Better Interpolation
  108. Droidcon Berlin 2019 @ragdroid

  109. Droidcon Berlin 2019 @ragdroid A LUT(lot) of image filters

  110. Droidcon Berlin 2019 @ragdroid Questions?

  111. Droidcon Berlin 2019 @ragdroid