A LUT(lot) of image Filters on Android

Slide 1

Slide 1 text

Droidcon Berlin 2019 @ragdroid Garima Jain A LUT(lot) of image ﬁlters on Android

Slide 2

Slide 2 text

Droidcon Berlin 2019 @ragdroid A LUT(lot) of image ﬁlters on Android Garima Jain

Slide 3

Slide 3 text

Droidcon Berlin 2019 @ragdroid Who?

Slide 4

Slide 4 text

Who? • Image Processing on Android? • No Image Processing?

Slide 5

Slide 5 text

Droidcon Berlin 2019 @ragdroid What?

Slide 6

Slide 6 text

BeautyPlus Google Photos Instagram

Slide 7

Slide 7 text

Adobe Lightroom

Slide 8

Slide 8 text

Droidcon Berlin 2019 @ragdroid Diﬀerent Approaches to Image Filters

Slide 9

Slide 9 text

Diﬀerent Approaches to Image Filters 1. Presets 2. LUT

Slide 10

Slide 10 text

Presets { "saturation": 0, "sharpness": 73, "luminance": 50, "vignetteAmount": 10, "vignetteRadius": 1.0, "shadows": 0.0, "contrast": 23, ... }

Slide 11

Slide 11 text

Adobe Lightroom

Slide 12

Slide 12 text

Adobe Lightroom

Slide 13

Slide 13 text

Presets • Achieve great results • Lots of customizable options • Do we even need it? • Ship Faster

Slide 14

Slide 14 text

LUT (Lookup Table) • Replaces runtime computation with simpler array indexing operation • Diﬀerent Types: ✦ 1D ✦ 3 x 1D ✦ 3D

Slide 15

Slide 15 text

1D LUT (Lookup Table) 0 0 1 85 2 169 3 255 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 1D Identity LUT (index, color)

Slide 16

Slide 16 text

3 x 0 1 2 3 0 1 2 3 0 1 2 3 1D LUT (Lookup Table) 1D Identity LUT 3 x 0 0 1 85 2 169 3 255

Slide 17

Slide 17 text

3D LUT (Lookup Table) (3,3,3) (0,0,0) R G B 3D Identity LUT (4 x 4 x 4)

Slide 18

Slide 18 text

3D LUT (Lookup Table) 0 0 0 0 0 1 0 0 2 0 0 3 0 1 0 0 1 1 0 1 2 0 1 3 0 2 0 0 2 1 0 2 2 0 2 3 0 3 0 0 3 1 0 3 2 0 3 3 B G R 1 0 0 1 0 1 1 0 2 1 0 3 1 1 0 1 1 1 1 1 2 1 1 3 1 2 0 1 2 1 1 2 2 1 2 3 1 3 0 1 3 1 1 3 2 1 3 3 2 0 0 2 0 1 2 0 2 2 0 3 2 1 0 2 1 1 2 1 2 2 1 3 2 2 0 2 2 1 2 2 2 2 2 3 2 3 0 2 3 1 2 3 2 2 3 3 3 0 0 3 0 1 3 0 2 3 0 3 3 1 0 3 1 1 3 1 2 3 1 3 3 2 0 3 2 1 3 2 2 3 2 3 3 3 0 3 3 1 3 3 2 3 3 3 3D Identity LUT (4 x 4 x 4) B G R B G R B G R

Slide 19

Slide 19 text

3D LUT (Lookup Table) • 4 x 4 x 4 is not enough • Practically 64 x 64 x 64 LUT images are used • 64 LUT images can map 262144 (64 ^3) colors

Slide 20

Slide 20 text

3D Square LUT (64 x 64 x 64)

Slide 21

Slide 21 text

Designers Pipeline • Create Filters • Start with Identity LUT image • Apply eﬀects to it • Export and create new LUTs

Slide 22

Slide 22 text

Designers Pipeline

Slide 23

Slide 23 text

Droidcon Berlin 2019 @ragdroid How?

Slide 24

Slide 24 text

easyLUT Library

Slide 25

Slide 25 text

easyLUT Library

Slide 26

Slide 26 text

• getPixel() and setPixel() • Not Fast • Eﬃcient Image Processing on Android - Nicolas Roard • Practical Image Processing in Android - Rebecca Franks easyLUT Library

Slide 27

Slide 27 text

Droidcon Berlin 2019 @ragdroid Diﬀerent Approaches to Image Processing

Slide 28

Slide 28 text

Image Processing • In Java - getPixel() and setPixel() • Native Code (JNI calls to C/C++) • OpenGL • RenderScript

Slide 29

Slide 29 text

Droidcon Berlin 2019 @ragdroid RenderScript

Slide 30

Slide 30 text

RenderScript 1. Baked-in Intrinsics - ScriptIntrinsic3DLUT 2. Custom Script

Slide 31

Slide 31 text

Droidcon Berlin 2019 @ragdroid 1. ScriptIntrinsic3DLUT

Slide 32

Slide 32 text

ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a 3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”

Slide 33

Slide 33 text

ScriptIntrinsic3DLUT setLUT(Allocation lut) Sets the Allocation to be used as the lookup table

Slide 34

Slide 34 text

How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped() LUT3DParams - silvaren/easyrs

Slide 39

Slide 39 text

Cube { int[] cube rSize int gSize int bSize int } LUT3DParams - silvaren/easyrs

Slide 40

Slide 40 text

Cube Creation Cube { int[] cube rSize int gSize int bSize int }

Slide 41

Slide 41 text

Cube Creation for (r in 0 until dimension) { for (g in 0 until dimension) { val p = r + g * w for (b in 0 until dimension) { val newPixel = pixels[p + b * h] lut[i++] = newPixel } } }

Slide 42

Slide 42 text

ScriptIntrinsic3DLUT val lutScript = ScriptIntrinsic3DLUT.create(renderScript, Element.U8_4(renderScript)) lutScript.setLUT(allocation) lutScript.forEach(allocationIn, allocationOut)

Slide 43

Slide 43 text

Cube Creation

Slide 44

Slide 44 text

Droidcon Berlin 2019 @ragdroid Transparent Test

Slide 45

Slide 45 text

Transparent Test Failed

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Droidcon Berlin 2019 @ragdroid 2. Custom Script

Slide 48

Slide 48 text

Custom Script For each pixel value (R,G,B) ﬁnd the corresponding value in LUT image and use that value instead

Slide 49

Slide 49 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap

Slide 50

Slide 50 text

Droidcon Berlin 2019 @ragdroid Lookup pixel in LUT

Slide 51

Slide 51 text

Square LUT 512 x 512

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

LUTFilter • Blue : Square within the LUT • Green and Red : (x, y) within a Square

Slide 54

Slide 54 text

LUTFilter • Example : Input Pixel (127, 127, 127) • 64 LUTs can not process all 256 colors so we divide by 4. • divide (127, 127, 127) by 4 = (31, 31, 31) ★ Blue : Square within the LUT : 31st square ★ Green and Red : (x, y) within a Square : (31, 31) within the square

Slide 55

Slide 55 text

31st square

Slide 56

Slide 56 text

R (31) G (31) (31, 31) within the square

Slide 57

Slide 57 text

x Y

Slide 58

Slide 58 text

Droidcon Berlin 2019 @ragdroid LUTFilter x = b % 8 * 64 + r y = floor(b/8.0f) * 64 + g

Slide 59

Slide 59 text

Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb = rsGetElementAt_uchar4(lut, x, y); Annd it works!

Slide 60

Slide 60 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap

Slide 61

Slide 61 text

Droidcon Berlin 2019 @ragdroid Transparent Test

Slide 62

Slide 62 text

Transparent Test Again Failed

Slide 63

Slide 63 text

Droidcon Berlin 2019 @ragdroid Pre-multiplied Alpha

Slide 64

Slide 64 text

Droidcon Berlin 2019 @ragdroid The Curious case of Android premultiplied Alpha

Slide 65

Slide 65 text

Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r * (256 - src.a) + src.r * src.a dest.g = dest.g * (256 - src.a) + src.g * src.a dest.b = dest.b * (256 - src.a) + src.b * src.a

Slide 66

Slide 66 text

Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r * (256 - src.a) + src.premultipliedR dest.g = dest.g * (256 - src.a) + src.premultipliedG dest.b = dest.b * (256 - src.a) + src.premultipliedB

Slide 67

Slide 67 text

Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha (Issue) • When we lookup input pixel in LUT • Wrong value lookup

Slide 68

Slide 68 text

Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha: Solution • BitmapFactory.Options.inPremultiplied • Handle it yourself

Slide 69

Slide 69 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap

Slide 70

Slide 70 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap

Slide 71

Slide 71 text

Droidcon Berlin 2019 @ragdroid Un-premultiply Alpha float div = in.a / 256.0; modifiedR = clamp(in.r / div, 0.0, 255.0); modifiedG = clamp(in.g / div, 0.0, 255.0); modifiedB = clamp(in.b / div, 0.0, 255.0);

Slide 72

Slide 72 text

Droidcon Berlin 2019 @ragdroid LUTFilter x = modifiedB % 8 * 64 + modifiedR y = floor(modifiedB/8.0f) * 64 + modifiedG

Slide 73

Slide 73 text

Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb = rsGetElementAt_uchar4(lut, x, y);

Slide 74

Slide 74 text

Transparent Test Passed!

Slide 75

Slide 75 text

Transparent Test Passed!

Slide 76

Slide 76 text

LUTFilter

Slide 77

Slide 77 text

Droidcon Berlin 2019 @ragdroid Banding & Interpolation

Slide 78

Slide 78 text

Banding

Slide 79

Slide 79 text

Banding

Slide 80

Slide 80 text

Banding Input Image LUT

Slide 81

Slide 81 text

Banding Input Image LUT R : 0 - 255 G: 0 - 255 B: 0 - 255 R : 0 - 63 G: 0 - 63 B: 0 - 63 (255 ^ 3 = 16581375 colors) (64 ^ 3 = 262144 colors)

Slide 82

Slide 82 text

Droidcon Berlin 2019 @ragdroid Banding x = (b / 4.0) % 8 * 64 + r/4.0 y = floor((b/4.0)/8.0f) * 64 + g/4.0

Slide 83

Slide 83 text

64 LUT Limitations • 64 LUT images can map 262144 (64 ^3) colors • Jpg / png images have 16777216 (256 ^3) colors! • Lossy Technique

Slide 84

Slide 84 text

Droidcon Berlin 2019 @ragdroid Interpolation “ Interpolation is a method of constructing new data points within the range of A discrete set of known data points. ” — Wikipedia

Slide 85

Slide 85 text

Droidcon Berlin 2019 @ragdroid Linear Interpolation (x,y) x0 x1 y0 y1

Slide 86

Slide 86 text

Droidcon Berlin 2019 @ragdroid Trilinear Interpolation Source: Computational Color Technology, Henry R. Kang

Slide 87

Slide 87 text

Droidcon Berlin 2019 @ragdroid Trilinear Interpolation • Source: Computational Color Technology, Henry R. Kang • Wikipedia : https://en.wikipedia.org/wiki/Trilinear_interpolation

Slide 88

Slide 88 text

Without Interpolation : Banding

Slide 89

Slide 89 text

With Interpolation: Reduced Banding

Slide 90

Slide 90 text

It Works!

Slide 91

Slide 91 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap

Slide 92

Slide 92 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate

Slide 93

Slide 93 text

Droidcon Berlin 2019 @ragdroid Other Issues

Slide 94

Slide 94 text

Droidcon Berlin 2019 @ragdroid Renderscript Stack Trace

Slide 95

Slide 95 text

2 days later

Slide 96

Slide 96 text

Some utilities • Ndk-stack : https://developer.android.com/ndk/guides/ndk-stack • AddressSanitizerOnAndroid : https://github.com/google/sanitizers/wiki/ AddressSanitizerOnAndroid

Slide 97

Slide 97 text

2 more days later ArrayIndexOutOfBounds

Slide 98

Slide 98 text

Droidcon Berlin 2019 @ragdroid LUT Formats

Slide 99

Slide 99 text

Expectation Reality

Slide 100

Slide 100 text

Hald LUT 512 x 512

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

Droidcon Berlin 2019 @ragdroid LUTFilter (HALD) x = g % 8 * 64 + r y = floor(g/8.0f) + b * 8

Slide 103

Slide 103 text

Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate

Slide 104

Slide 104 text

Droidcon Berlin 2019 @ragdroid LUTFilter (HALD)

Slide 105

Slide 105 text

Droidcon Berlin 2019 @ragdroid Finally

Slide 106

Slide 106 text

References • Computational Color Technology, Henry R. Kang (Interpolation) • easyLUT Library • silvaren/easyrs (LUT3DParams) • Renderscript docs • The Curious case of Android premultiplied Alpha (pre-multiplied Alpha

Slide 107

Slide 107 text

What Next? • Building Filters with Go - Wayne Ashley Berry • .cube ﬁle format • Improve speed even further • Better Interpolation