Mobile Image Processing at Velocity SC, 2015

Mobile Image Processing at Velocity SC, 2015

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at Velocity SC 2015 in Santa Clara, CA on May 29, 2015.

A9a379f2e92c7ded4564190c5b286b78?s=128

Tim Kadlec

May 29, 2015
Tweet

Transcript

  1. MOBILE IMAGE PROCESSING https://flic.kr/p/qRJKUz Tim Kadlec | @tkadlec

  2. https://flic.kr/p/aShfSP

  3. None
  4. https://flic.kr/p/oPn6ZV

  5. None
  6. 10% of photos ever 2012 Source: http://time.com/3445111/#1

  7. Source: http://time.com/3445111/#1 1839 2012 2011 172 years, 90%

  8. Source: http://time.com/3445111/#1 1839 2012 2011 1 year, 10%

  9. billion photos PER DAY Source: http://www.kpcb.com/internet-trends 1.8

  10. Source: May 1st run of http://httparchive.org 789kb Everything else 1310kb

    Images
  11. Source: May 1st run of http://httparchive.org 36.5% Everything else 62.4%

    Images
  12. None
  13. None
  14. 360px x 202px

  15. 1280px x 520px

  16. Request Decode Copy to GPU Display

  17. Request Copy to GPU Display Decode

  18. 1280px x 520px

  19. Source: May 1st run of http://httparchive.org 46% JPG 29% PNG

    23% GIF 2% Other
  20. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  21. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  22. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  23. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  24. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding

  25. YUV to RGB Chroma Upsampling iDCT / Dequantization Huffman Decoding

  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. Resized: 7.86ms

  36. Resized: 7.86ms Double: 26.34ms

  37. Resized: 7.86ms Double: 26.34ms Large: 266.49ms

  38. Resized: 7.86ms Double: +235.1% Large: +3291.7%

  39. None
  40. 2GB RAM Quad-core 1.5 GHz Krait CPU Adreno 320 GPU

  41. Resized: 30.38ms

  42. Resized: 30.38ms Double: 102.77ms

  43. Resized: 30.38ms Double: 102.77ms Large (6x): 1,534.99ms

  44. Resized: 30.38ms Double: +238.3% Large (6x): +4952.6%

  45. None
  46. RESIZE IMAGES

  47. RESIZE IMAGES

  48. Request Display Decode Copy to GPU

  49. Request Copy to GPU Display Store in memory Decode

  50. None
  51. R: 65 G: 47 B: 35 A: 1 R: 26

    G: 26 B: 29 A: 1 R: 30 G: 29 B: 32 A: 1
  52. W x H x 4

  53. 1280px x 520px

  54. 1280 x 520 x 4

  55. 1280 x 520 x 4 2,662,400

  56. 2.54 MB 1280 x 520 x 4

  57. None
  58. None
  59. “…25% of new Android phones have only 512MB of RAM.”

    Jen Fitzpatrick VP of product management for Google Maps
  60. 360 x 202 x 4

  61. 360 x 202 x 4 290,880

  62. - 290,880 2,662,400 2,371,520

  63. 2.26 MB

  64. 270px x 936px 987 kb

  65. RESIZE IMAGES

  66. 600px x 600px 200px x 200px

  67. 550px x 550px 150px x 150px

  68. (600 x 600 - 550 x 550) x 4 =

    230,000 wasted bytes
  69. (200 x 200 - 150 x 150) x 4 =

    70,000 wasted bytes 230,000 wasted bytes
  70. 70,000 wasted bytes 230,000 wasted bytes

  71. 700px 1200px 200px

  72. 1200px 200px 700px

  73. MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD

  74. None
  75. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB

  76. None
  77. Luma (light) Chroma (color)

  78. Y CB CR

  79. Chroma Subsampling

  80. J : a : b

  81. None
  82. 4 : a : b

  83. None
  84. None
  85. None
  86. 4 : 4 : 4

  87. 4

  88. 4 4

  89. 4 4 4 8 total

  90. None
  91. None
  92. None
  93. 4 : 2 : 2

  94. 4

  95. 4 2

  96. 2 2 4 4 total (50% savings)

  97. None
  98. None
  99. None
  100. None
  101. 4 : 1 : 1

  102. 4

  103. 4 1

  104. 4 1 1 2 total (75% savings)

  105. None
  106. None
  107. None
  108. None
  109. 4 : 2 : 0

  110. 4

  111. 4 2

  112. 4 2 0 2 total (75% savings)

  113. None
  114. None
  115. None
  116. None
  117. Request Decode Copy to GPU Display

  118. Request Decode Copy to GPU Display Decode

  119. (W x H x 3) - (W x H x

    SUBSAMPLE_LEVEL x 2)
  120. RGBA 2,662,400 4:4:4 1,996,800 4:2:2 1,331,200 4:2:0 998,400 1280px x

    520px
  121. 62.5% memory savings 4:2:0 Source: http://time.com/3445111/#1

  122. None
  123. None
  124. None
  125. None
  126. > 26MB

  127. Browser OS

  128. Browser OS Psst….need some memory?

  129. Browser OS Yeah, I’ll take some of that.

  130. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB

  131. None
  132. <meta name=“viewport" content=“width=device-width, minimum-scale=1.0” />

  133. None
  134. Resized: 30.38ms

  135. Resized: 5.62ms

  136. Resized: 5.62ms Double: 102.77ms

  137. Resized: 5.62ms Double: 27.74ms

  138. Resized: 5.62ms Double: 27.74ms Large (6x): 1534.99ms

  139. Resized: 5.62ms Double: 27.74ms Large (6x): 462.11ms

  140. Resized: 5.62ms Double: +393.6% Large (6x): +8122.6%

  141. MICRO-OPTIMIZATIONS?

  142. “…as bandwidth grows, and as processing power grows, and as

    browsers get better we just keep filling everything up. ” Jeff Veen
  143. NETWORK MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD

  144. RESIZE IMAGES 1

  145. RESIZE IMAGES 1 2 SMALL BREAKPOINTS FOR LARGE IMAGES

  146. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3

    SMALL BREAKPOINTS FOR LARGE IMAGES
  147. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3

    USE 4:2:0 SUBSAMPLING 4 SMALL BREAKPOINTS FOR LARGE IMAGES
  148. THANK YOU Tim Kadlec | @tkadlec