Mobile Image Processing at London Web Perf Meetup, 2016

A9a379f2e92c7ded4564190c5b286b78?s=47 Tim Kadlec
February 11, 2016

Mobile Image Processing at London Web Perf Meetup, 2016

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at London Web Perf Meetup in London, UK on Feb 11, 2015.

A9a379f2e92c7ded4564190c5b286b78?s=128

Tim Kadlec

February 11, 2016
Tweet

Transcript

  1. MOBILE IMAGE PROCESSING https://flic.kr/p/qRJKUz Tim Kadlec | @tkadlec

  2. https://flic.kr/p/aShfSP

  3. None
  4. https://flic.kr/p/oPn6ZV

  5. None
  6. 10% of photos ever 2012 Source: http://time.com/3445111/#1

  7. Source: http://time.com/3445111/#1 1839 2012

  8. Source: http://time.com/3445111/#1 1839 2012 2011 172 years, 90%

  9. Source: http://time.com/3445111/#1 1839 2012 2011 1 year, 10%

  10. Source: http://time.com/3445111/#1 1839 2012 2011 1 year, 10%

  11. 0.1 Source: http://www.kpcb.com/internet-trends

  12. 1.8 billion photos PER DAY Source: http://www.kpcb.com/internet-trends

  13. None
  14. None
  15. None
  16. 360px x 510px

  17. 1300px x 1024px

  18. Request Decode Copy to GPU Display

  19. Request Copy to GPU Display Decode

  20. Request Copy to GPU Display Decode

  21. Source: May 1st run of http://httparchive.org 46% JPG 29% PNG

    23% GIF 2% Other
  22. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  23. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  24. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  25. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  26. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  27. YUV to RGB Chroma Upsampling iDCT / Dequantization Huffman Decoding

  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. Resized: 7.86ms

  36. Resized: 7.86ms Double: 26.34ms

  37. Resized: 7.86ms Double: 26.34ms Large: 266.49ms

  38. Resized: 7.86ms Double: +235.1% Large: +3291.7%

  39. None
  40. 2GB RAM Quad-core 1.5 GHz Krait CPU Adreno 320 GPU

  41. Resized: 30.38ms

  42. Resized: 30.38ms Double: 102.77ms

  43. Resized: 30.38ms Double: 102.77ms Large (6x): 1,534.99ms

  44. Resized: 30.38ms Double: +238.3% Large (6x): +4952.6%

  45. None
  46. None
  47. RESIZE IMAGES

  48. None
  49. “HUGER”

  50. ROOTER

  51. ROUTER

  52. COLOUR

  53. COLOUR

  54. Request Display Decode Copy to GPU

  55. Request Display Decode Copy to GPU

  56. Request Copy to GPU Display Store in memory Decode

  57. None
  58. None
  59. R: 45 G: 48 B: 46 A: 1 R: 177

    G: 167 B: 157 A: 1 R: 190 G: 187 B: 181 A: 1
  60. W x H x 4

  61. 1300px x 1024px

  62. 1300 x 1024 x 4

  63. 5,324,800 1300 x 1024 x 4

  64. 5.1 MB 1300 x 1024 x 4

  65. None
  66. None
  67. None
  68. 360 x 510 x 4

  69. 734,400

  70. - 734,400 5,324,800 4,590,400

  71. 4.38 MB

  72. “…25% of new Android phones have only 512MB of RAM.”

    Jen Fitzpatrick VP of product management for Google Maps
  73. 270px x 936px 987 kb

  74. RESIZE IMAGES

  75. COLOR

  76. 600px x 600px 200px x 200px

  77. 550px x 550px 150px x 150px

  78. (600 x 600 - 550 x 550) x 4 =

    230,000 wasted bytes
  79. (200 x 200 - 150 x 150) x 4 =

    70,000 wasted bytes 230,000 wasted bytes
  80. 70,000 wasted bytes 230,000 wasted bytes

  81. 700px 1200px 200px

  82. 1200px 200px 700px

  83. MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD

  84. None
  85. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB

  86. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB

  87. None
  88. Luma (light) Chroma (color)

  89. Y CB CR

  90. Chroma Subsampling

  91. J : a : b

  92. None
  93. 4 : a : b

  94. None
  95. None
  96. None
  97. 4 : 4 : 4

  98. 4

  99. 4 4

  100. 4 4 4 8 total

  101. None
  102. None
  103. None
  104. 4 : 2 : 2

  105. 4

  106. 4 2

  107. 2 2 4 4 total (50% savings)

  108. None
  109. None
  110. None
  111. None
  112. 4 : 1 : 1

  113. 4

  114. 4 1

  115. 4 1 1 2 total (75% savings)

  116. None
  117. None
  118. None
  119. None
  120. 4 : 2 : 0

  121. 4

  122. 4 2

  123. 4 2 0 2 total (75% savings)

  124. None
  125. None
  126. None
  127. None
  128. None
  129. Request Decode Copy to GPU Display

  130. Request Decode Copy to GPU Display Decode

  131. (W x H x 3) - (W x H x

    SUBSAMPLE_LEVEL x 2)
  132. RGBA 5,324,800 4:4:4 3,993,600 4:2:2 2,662,400 4:2:0 1,996,800 1300px x

    1024px
  133. 62.5% memory savings 4:2:0 Source: http://time.com/3445111/#1

  134. https://flic.kr/p/aaThYm

  135. https://flic.kr/p/aaThYm

  136. Source: May 1st run of http://calendar.perfplanet.com/2015/why-arent-your-images-using-chroma-subsampling/ 39.3% 4:2:0 0.9% Other

    59.7% 4:4:4
  137. None
  138. None
  139. None
  140. None
  141. None
  142. None
  143. > 26MB

  144. Browser OS

  145. Browser OS Psst….need some memory?

  146. Browser OS Yeah, I’ll take some of that.

  147. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB

  148. None
  149. <meta name=“viewport" content=“width=device-width, minimum-scale=1.0” />

  150. None
  151. Resized: 30.38ms

  152. Resized: 5.62ms

  153. Resized: 5.62ms Double: 102.77ms

  154. Resized: 5.62ms Double: 27.74ms

  155. Resized: 5.62ms Double: 27.74ms Large (6x): 1534.99ms

  156. Resized: 5.62ms Double: 27.74ms Large (6x): 462.11ms

  157. Resized: 5.62ms Double: +393.6% Large (6x): +8122.6%

  158. None
  159. <meta name=“viewport” content=“initial-scale = 1.0, maximum-scale = 1.0” />

  160. <meta name=“viewport” content=“initial-scale = 1.0, maximum-scale = 1.0, width=device-width” />

  161. MICRO-OPTIMIZATIONS?

  162. None
  163. None
  164. None
  165. None
  166. None
  167. None
  168. None
  169. “…as bandwidth grows, and as processing power grows, and as

    browsers get better we just keep filling everything up. ” Jeff Veen
  170. NETWORK MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD

  171. RESIZE IMAGES 1

  172. RESIZE IMAGES 1 2 SMALL BREAKPOINTS FOR LARGE IMAGES

  173. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3

    SMALL BREAKPOINTS FOR LARGE IMAGES
  174. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3

    USE 4:2:0 SUBSAMPLING 4 SMALL BREAKPOINTS FOR LARGE IMAGES
  175. THANK YOU Tim Kadlec | @tkadlec