Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mobile Image Processing at London Web Perf Meetup, 2016

Tim Kadlec
February 11, 2016
Programming
1
150

Mobile Image Processing at London Web Perf Meetup, 2016

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at London Web Perf Meetup in London, UK on Feb 11, 2015.

Tim Kadlec

February 11, 2016
Tweet

More Decks by Tim Kadlec

See All by Tim Kadlec
The State of Node.js Security, at Node.js Interactive 2017
tkadlec
1
210
JavaScript Exposed at Midwest JS
tkadlec
3
240
Focusing On What Matters, at Fluent, 2017
tkadlec
0
94
Once More, With Feeling at Code 2016 in Sydney
tkadlec
1
480
Once More, With Feeling
tkadlec
9
1.4k
Better By Proxy at Velocity NY 2015
tkadlec
3
460
Getting Started with Performance Budgets at HighEdWeb Technical Academy, 2015
tkadlec
9
1k
Reaching Everyone, Fast at From the Front, 2015
tkadlec
8
6.3k
Better by Proxy at Smashing Conf NY
tkadlec
2
780

Other Decks in Programming

See All in Programming
ざっくりつかむLangChainのメンタルモデル [説明欄にソースあり]
sagawa23
6
930
Introduction of new in Kotlin for Android
yumakann
0
460
Domain-Driven Transformation
hschwentner
0
200
Look Back Over Deep Links
jmatsu
0
230
Kotlin Multiplatform Library 배포하기
fornewid
0
210
Mackerel のアクセシビリティ改善事例
azukiazusa1
0
470
Github CopilotとChatGPTを使って感じた使い分けの糸口 / JavaDo #22
gishi_yama
0
180
Go/Cgoで映像・音声のリアルタイム処理をやるまでの道のり - Go Conference 2023
octu0
3
1.5k
NotionのURLをWorkersで綺麗にしよう！
totto2727
0
210
初めてのKubernetes (ハンズオン)
nearme_tech
0
150
Studio Botを遠くから眺める
shinmiy
0
170
Semantic Kernel(C#)でAzure OpenAI ServiceのGPT-4を使ってみる
tomokusaba
1
170

Featured

See All Featured
Practical Orchestrator
shlominoach
179
9.2k
Principles of Awesome APIs and How to Build Them.
keavy
119
16k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
Bootstrapping a Software Product
garrettdimon
PRO
299
110k
Code Reviewing Like a Champion
maltzj
508
38k
The Straight Up "How To Draw Better" Workshop
denniskardys
226
130k
Building a Modern Day 
E-commerce SEO Strategy
aleyda
9
5.3k
jQuery: Nuts, Bolts and Bling
dougneiner
57
6.7k
Making the Leap to Tech Lead
cromwellryan
118
7.8k
Building Adaptive Systems
keathley
29
1.5k
Design by the Numbers
sachag
272
18k
We Have a Design System, Now What?
morganepeng
38
6.2k

Transcript

  1. MOBILE IMAGE
    PROCESSING
    https://flic.kr/p/qRJKUz
    Tim Kadlec | @tkadlec

    View Slide

  2. https://flic.kr/p/aShfSP

    View Slide

  3. View Slide

  4. https://flic.kr/p/oPn6ZV

    View Slide

  5. View Slide

  6. 10%
    of photos ever
    2012
    Source: http://time.com/3445111/#1

    View Slide

  7. Source: http://time.com/3445111/#1
    1839 2012

    View Slide

  8. Source: http://time.com/3445111/#1
    1839 2012
    2011
    172 years, 90%

    View Slide

  9. Source: http://time.com/3445111/#1
    1839 2012
    2011
    1 year, 10%

    View Slide

  10. Source: http://time.com/3445111/#1
    1839 2012
    2011
    1 year, 10%

    View Slide

  11. 0.1
    Source: http://www.kpcb.com/internet-trends

    View Slide

  12. 1.8
    billion photos
    PER DAY
    Source: http://www.kpcb.com/internet-trends

    View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. 360px x 510px

    View Slide

  17. 1300px x 1024px

    View Slide

  18. Request Decode Copy to GPU Display

    View Slide

  19. Request Copy to GPU Display
    Decode

    View Slide

  20. Request Copy to GPU Display
    Decode

    View Slide

  21. Source: May 1st run of http://httparchive.org
    46%
    JPG
    29%
    PNG
    23%
    GIF
    2%
    Other

    View Slide

  22. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View Slide

  23. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View Slide

  24. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View Slide

  25. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View Slide

  26. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View Slide

  27. YUV to RGB
    Chroma
    Upsampling
    iDCT /
    Dequantization
    Huffman
    Decoding

    View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. Resized: 7.86ms

    View Slide

  36. Resized: 7.86ms
    Double: 26.34ms

    View Slide

  37. Resized: 7.86ms
    Double: 26.34ms
    Large: 266.49ms

    View Slide

  38. Resized: 7.86ms
    Double: +235.1%
    Large: +3291.7%

    View Slide

  39. View Slide

  40. 2GB RAM
    Quad-core 1.5 GHz Krait CPU
    Adreno 320 GPU

    View Slide

  41. Resized: 30.38ms

    View Slide

  42. Resized: 30.38ms
    Double: 102.77ms

    View Slide

  43. Resized: 30.38ms
    Double: 102.77ms
    Large (6x): 1,534.99ms

    View Slide

  44. Resized: 30.38ms
    Double: +238.3%
    Large (6x): +4952.6%

    View Slide

  45. View Slide

  46. View Slide

  47. RESIZE
    IMAGES

    View Slide

  48. View Slide

  49. “HUGER”

    View Slide

  50. ROOTER

    View Slide

  51. ROUTER

    View Slide

  52. COLOUR

    View Slide

  53. COLOUR

    View Slide

  54. Request Display
    Decode Copy to GPU

    View Slide

  55. Request Display
    Decode Copy to GPU

    View Slide

  56. Request Copy to GPU Display
    Store in memory
    Decode

    View Slide

  57. View Slide

  58. View Slide

  59. R: 45
    G: 48
    B: 46
    A: 1
    R: 177
    G: 167
    B: 157
    A: 1
    R: 190
    G: 187
    B: 181
    A: 1

    View Slide

  60. W x H x 4

    View Slide

  61. 1300px x 1024px

    View Slide

  62. 1300 x 1024 x 4

    View Slide

  63. 5,324,800
    1300 x 1024 x 4

    View Slide

  64. 5.1 MB
    1300 x 1024 x 4

    View Slide

  65. View Slide

  66. View Slide

  67. View Slide

  68. 360 x 510 x 4

    View Slide

  69. 734,400

    View Slide

  70. - 734,400
    5,324,800
    4,590,400

    View Slide

  71. 4.38 MB

    View Slide

  72. “…25% of new Android phones
    have only 512MB of RAM.”
    Jen Fitzpatrick
    VP of product management for Google Maps

    View Slide

  73. 270px x 936px
    987 kb

    View Slide

  74. RESIZE
    IMAGES

    View Slide

  75. COLOR

    View Slide

  76. 600px x 600px 200px x 200px

    View Slide

  77. 550px x 550px 150px x 150px

    View Slide

  78. (600 x 600 - 550 x 550) x 4 =
    230,000 wasted bytes

    View Slide

  79. (200 x 200 - 150 x 150) x 4 =
    70,000 wasted bytes
    230,000 wasted bytes

    View Slide

  80. 70,000 wasted bytes
    230,000 wasted bytes

    View Slide

  81. 700px 1200px
    200px

    View Slide

  82. 1200px
    200px 700px

    View Slide

  83. MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD

    View Slide

  84. View Slide

  85. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View Slide

  86. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View Slide

  87. View Slide

  88. Luma
    (light)
    Chroma
    (color)

    View Slide

  89. Y
    CB
    CR

    View Slide

  90. Chroma
    Subsampling

    View Slide

  91. J : a : b

    View Slide

  92. View Slide

  93. 4 : a : b

    View Slide

  94. View Slide

  95. View Slide

  96. View Slide

  97. 4 : 4 : 4

    View Slide

  98. 4

    View Slide

  99. 4
    4

    View Slide

  100. 4
    4
    4
    8 total

    View Slide

  101. View Slide

  102. View Slide

  103. View Slide

  104. 4 : 2 : 2

    View Slide

  105. 4

    View Slide

  106. 4
    2

    View Slide

  107. 2
    2
    4
    4 total
    (50% savings)

    View Slide

  108. View Slide

  109. View Slide

  110. View Slide

  111. View Slide

  112. 4 : 1 : 1

    View Slide

  113. 4

    View Slide

  114. 4
    1

    View Slide

  115. 4
    1
    1
    2 total
    (75% savings)

    View Slide

  116. View Slide

  117. View Slide

  118. View Slide

  119. View Slide

  120. 4 : 2 : 0

    View Slide

  121. 4

    View Slide

  122. 4
    2

    View Slide

  123. 4
    2
    0
    2 total
    (75% savings)

    View Slide

  124. View Slide

  125. View Slide

  126. View Slide

  127. View Slide

  128. View Slide

  129. Request Decode Copy to GPU Display

    View Slide

  130. Request Decode Copy to GPU Display
    Decode

    View Slide

  131. (W x H x 3)
    -
    (W x H x
    SUBSAMPLE_LEVEL x 2)

    View Slide

  132. RGBA 5,324,800
    4:4:4 3,993,600
    4:2:2 2,662,400
    4:2:0 1,996,800
    1300px x 1024px

    View Slide

  133. 62.5%
    memory savings
    4:2:0
    Source: http://time.com/3445111/#1

    View Slide

  134. https://flic.kr/p/aaThYm

    View Slide

  135. https://flic.kr/p/aaThYm

    View Slide

  136. Source: May 1st run of http://calendar.perfplanet.com/2015/why-arent-your-images-using-chroma-subsampling/
    39.3%
    4:2:0
    0.9%
    Other
    59.7%
    4:4:4

    View Slide

  137. View Slide

  138. View Slide

  139. View Slide

  140. View Slide

  141. View Slide

  142. View Slide

  143. > 26MB

    View Slide

  144. Browser OS

    View Slide

  145. Browser OS
    Psst….need
    some memory?

    View Slide

  146. Browser OS
    Yeah, I’ll take
    some of that.

    View Slide

  147. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View Slide

  148. View Slide

  149. content=“width=device-width,
    minimum-scale=1.0”
    />

    View Slide

  150. View Slide

  151. Resized: 30.38ms

    View Slide

  152. Resized: 5.62ms

    View Slide

  153. Resized: 5.62ms
    Double: 102.77ms

    View Slide

  154. Resized: 5.62ms
    Double: 27.74ms

    View Slide

  155. Resized: 5.62ms
    Double: 27.74ms
    Large (6x): 1534.99ms

    View Slide

  156. Resized: 5.62ms
    Double: 27.74ms
    Large (6x): 462.11ms

    View Slide

  157. Resized: 5.62ms
    Double: +393.6%
    Large (6x): +8122.6%

    View Slide

  158. View Slide

  159. name=“viewport”
    content=“initial-scale = 1.0,
    maximum-scale = 1.0”
    />

    View Slide

  160. name=“viewport”
    content=“initial-scale = 1.0,
    maximum-scale = 1.0,
    width=device-width”
    />

    View Slide

  161. MICRO-OPTIMIZATIONS?

    View Slide

  162. View Slide

  163. View Slide

  164. View Slide

  165. View Slide

  166. View Slide

  167. View Slide

  168. View Slide

  169. “…as bandwidth grows, and as
    processing power grows, and as
    browsers get better we just keep
    filling everything up. ”
    Jeff Veen

    View Slide

  170. NETWORK
    MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD

    View Slide

  171. RESIZE IMAGES
    1

    View Slide

  172. RESIZE IMAGES
    1
    2 SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View Slide

  173. RESIZE IMAGES
    1
    2
    META VIEWPORT IS YOUR FRIEND
    3
    SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View Slide

  174. RESIZE IMAGES
    1
    2
    META VIEWPORT IS YOUR FRIEND
    3
    USE 4:2:0 SUBSAMPLING
    4
    SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View Slide

  175. THANK YOU
    Tim Kadlec | @tkadlec

    View Slide