A LUT(lot) of image Filters on Android

In this talk, we will learn how to create image filters with LUTs (Lookup Tables) using Renderscript on Android. We will briefly look into various approaches one can take for creating filters on Android. We will then learn about the basics of LUTs and how we can use them with Renderscript for creating filters. Finally, we will focus on the challenges that we faced while implementing filters on Android with the Renderscript framework. We will try to use graphical representations wherever possible to explain all these concepts.

Transcript

1. Droidcon Berlin 2019 @ragdroid Garima Jain A LUT(lot) of image filters on Android
2. Droidcon Berlin 2019 @ragdroid A LUT(lot) of image filters on Android Garima Jain
3. Droidcon Berlin 2019 @ragdroid Who?
4. Who? • Image Processing on Android? • No Image Processing?
5. Droidcon Berlin 2019 @ragdroid What?
6. BeautyPlus Google Photos Instagram
7. Adobe Lightroom
8. Droidcon Berlin 2019 @ragdroid Different Approaches to Image Filters
9. Different Approaches to Image Filters 1. Presets 2. LUT
10. Presets { "saturation": 0, "sharpness": 73, "luminance": 50, "vignetteAmount": 10, "vignetteRadius": 1.0, "shadows": 0.0, "contrast": 23, ... }
11. Adobe Lightroom
12. Adobe Lightroom
13. Presets • Achieve great results • Lots of customizable options • Do we even need it? • Ship Faster
14. LUT (Lookup Table) • Replaces runtime computation with simpler array indexing operation • Different Types: ✦ 1D ✦ 3 x 1D ✦ 3D
15. 1D LUT (Lookup Table) 0 0 1 85 2 169 3 255 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 1D Identity LUT (index, color)
16. 3 x 0 1 2 3 0 1 2 3 0 1 2 3 1D LUT (Lookup Table) 1D Identity LUT 3 x 0 0 1 85 2 169 3 255
17. 3D LUT (Lookup Table) (3,3,3) (0,0,0) R G B 3D Identity LUT (4 x 4 x 4)
18. 3D LUT (Lookup Table) 0 0 0 0 0 1 0 0 2 0 0 3 0 1 0 0 1 1 0 1 2 0 1 3 0 2 0 0 2 1 0 2 2 0 2 3 0 3 0 0 3 1 0 3 2 0 3 3 B G R 1 0 0 1 0 1 1 0 2 1 0 3 1 1 0 1 1 1 1 1 2 1 1 3 1 2 0 1 2 1 1 2 2 1 2 3 1 3 0 1 3 1 1 3 2 1 3 3 2 0 0 2 0 1 2 0 2 2 0 3 2 1 0 2 1 1 2 1 2 2 1 3 2 2 0 2 2 1 2 2 2 2 2 3 2 3 0 2 3 1 2 3 2 2 3 3 3 0 0 3 0 1 3 0 2 3 0 3 3 1 0 3 1 1 3 1 2 3 1 3 3 2 0 3 2 1 3 2 2 3 2 3 3 3 0 3 3 1 3 3 2 3 3 3 3D Identity LUT (4 x 4 x 4) B G R B G R B G R
19. 3D LUT (Lookup Table) • 4 x 4 x 4 is not enough • Practically 64 x 64 x 64 LUT images are used • 64 LUT images can map 262144 (64 ^3) colors
20. 3D Square LUT (64 x 64 x 64)
21. Designers Pipeline • Create Filters • Start with Identity LUT image • Apply effects to it • Export and create new LUTs
22. Designers Pipeline
23. Droidcon Berlin 2019 @ragdroid How?
24. easyLUT Library
25. easyLUT Library
26. • getPixel() and setPixel() • Not Fast • Efficient Image Processing on Android - Nicolas Roard • Practical Image Processing in Android - Rebecca Franks easyLUT Library
27. Droidcon Berlin 2019 @ragdroid Different Approaches to Image Processing
28. Image Processing • In Java - getPixel() and setPixel() • Native Code (JNI calls to C/C++) • OpenGL • RenderScript
29. Droidcon Berlin 2019 @ragdroid RenderScript
30. RenderScript 1. Baked-in Intrinsics - ScriptIntrinsic3DLUT 2. Custom Script
31. Droidcon Berlin 2019 @ragdroid 1. ScriptIntrinsic3DLUT
32. ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a 3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”
33. ScriptIntrinsic3DLUT setLUT(Allocation lut) Sets the Allocation to be used as the lookup table
34. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
35. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
36. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
37. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
38. createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped() LUT3DParams - silvaren/easyrs
39. Cube { int[] cube rSize int gSize int bSize int } LUT3DParams - silvaren/easyrs
40. Cube Creation Cube { int[] cube rSize int gSize int bSize int }
41. Cube Creation for (r in 0 until dimension) { for (g in 0 until dimension) { val p = r + g * w for (b in 0 until dimension) { val newPixel = pixels[p + b * h] lut[i++] = newPixel } } }
42. ScriptIntrinsic3DLUT val lutScript = ScriptIntrinsic3DLUT.create(renderScript, Element.U8_4(renderScript)) lutScript.setLUT(allocation) lutScript.forEach(allocationIn, allocationOut)
43. Cube Creation
44. Droidcon Berlin 2019 @ragdroid Transparent Test
45. Transparent Test Failed
46. ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a 3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”
47. Droidcon Berlin 2019 @ragdroid 2. Custom Script
48. Custom Script For each pixel value (R,G,B) find the corresponding value in LUT image and use that value instead
49. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
50. Droidcon Berlin 2019 @ragdroid Lookup pixel in LUT
51. Square LUT 512 x 512
53. LUTFilter • Blue : Square within the LUT • Green and Red : (x, y) within a Square
54. LUTFilter • Example : Input Pixel (127, 127, 127) • 64 LUTs can not process all 256 colors so we divide by 4. • divide (127, 127, 127) by 4 = (31, 31, 31) ★ Blue : Square within the LUT : 31st square ★ Green and Red : (x, y) within a Square : (31, 31) within the square
55. 31st square
56. R (31) G (31) (31, 31) within the square
57. x Y
58. Droidcon Berlin 2019 @ragdroid LUTFilter x = b % 8 * 64 + r y = floor(b/8.0f) * 64 + g
59. Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb = rsGetElementAt_uchar4(lut, x, y); Annd it works!
60. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
61. Droidcon Berlin 2019 @ragdroid Transparent Test
62. Transparent Test Again Failed
63. Droidcon Berlin 2019 @ragdroid Pre-multiplied Alpha
64. Droidcon Berlin 2019 @ragdroid The Curious case of Android premultiplied Alpha
65. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r * (256 - src.a) + src.r * src.a dest.g = dest.g * (256 - src.a) + src.g * src.a dest.b = dest.b * (256 - src.a) + src.b * src.a
66. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r * (256 - src.a) + src.premultipliedR dest.g = dest.g * (256 - src.a) + src.premultipliedG dest.b = dest.b * (256 - src.a) + src.premultipliedB
67. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha (Issue) • When we lookup input pixel in LUT • Wrong value lookup
68. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha: Solution • BitmapFactory.Options.inPremultiplied • Handle it yourself
69. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
70. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
71. Droidcon Berlin 2019 @ragdroid Un-premultiply Alpha float div = in.a / 256.0; modifiedR = clamp(in.r / div, 0.0, 255.0); modifiedG = clamp(in.g / div, 0.0, 255.0); modifiedB = clamp(in.b / div, 0.0, 255.0);
72. Droidcon Berlin 2019 @ragdroid LUTFilter x = modifiedB % 8 * 64 + modifiedR y = floor(modifiedB/8.0f) * 64 + modifiedG
73. Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb = rsGetElementAt_uchar4(lut, x, y);
74. Transparent Test Passed!
75. Transparent Test Passed!
76. LUTFilter
77. Droidcon Berlin 2019 @ragdroid Banding & Interpolation
78. Banding
79. Banding
80. Banding Input Image LUT
81. Banding Input Image LUT R : 0 - 255 G: 0 - 255 B: 0 - 255 R : 0 - 63 G: 0 - 63 B: 0 - 63 (255 ^ 3 = 16581375 colors) (64 ^ 3 = 262144 colors)
82. Droidcon Berlin 2019 @ragdroid Banding x = (b / 4.0) % 8 * 64 + r/4.0 y = floor((b/4.0)/8.0f) * 64 + g/4.0
83. 64 LUT Limitations • 64 LUT images can map 262144 (64 ^3) colors • Jpg / png images have 16777216 (256 ^3) colors! • Lossy Technique
84. Droidcon Berlin 2019 @ragdroid Interpolation “ Interpolation is a method of constructing new data points within the range of A discrete set of known data points. ” — Wikipedia
85. Droidcon Berlin 2019 @ragdroid Linear Interpolation (x,y) x0 x1 y0 y1
86. Droidcon Berlin 2019 @ragdroid Trilinear Interpolation Source: Computational Color Technology, Henry R. Kang
87. Droidcon Berlin 2019 @ragdroid Trilinear Interpolation • Source: Computational Color Technology, Henry R. Kang • Wikipedia : https://en.wikipedia.org/wiki/Trilinear_interpolation
88. Without Interpolation : Banding
89. With Interpolation: Reduced Banding
90. It Works!
91. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
92. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate
93. Droidcon Berlin 2019 @ragdroid Other Issues
94. Droidcon Berlin 2019 @ragdroid Renderscript Stack Trace
95. 2 days later
96. Some utilities • Ndk-stack : https://developer.android.com/ndk/guides/ndk-stack • AddressSanitizerOnAndroid : https://github.com/google/sanitizers/wiki/ AddressSanitizerOnAndroid
97. 2 more days later ArrayIndexOutOfBounds
98. Droidcon Berlin 2019 @ragdroid LUT Formats
99. Expectation Reality
100. Hald LUT 512 x 512
102. Droidcon Berlin 2019 @ragdroid LUTFilter (HALD) x = g % 8 * 64 + r y = floor(g/8.0f) + b * 8
103. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate
104. Droidcon Berlin 2019 @ragdroid LUTFilter (HALD)
105. Droidcon Berlin 2019 @ragdroid Finally
106. References • Computational Color Technology, Henry R. Kang (Interpolation) • easyLUT Library • silvaren/easyrs (LUT3DParams) • Renderscript docs • The Curious case of Android premultiplied Alpha (pre-multiplied Alpha
107. What Next? • Building Filters with Go - Wayne Ashley Berry • .cube file format • Improve speed even further • Better Interpolation
108. Droidcon Berlin 2019 @ragdroid
109. Droidcon Berlin 2019 @ragdroid A LUT(lot) of image filters
110. Droidcon Berlin 2019 @ragdroid Questions?
111. Droidcon Berlin 2019 @ragdroid