Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mobile Image Processing at London Web Perf Meetup, 2016

Tim Kadlec
February 11, 2016

Mobile Image Processing at London Web Perf Meetup, 2016

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at London Web Perf Meetup in London, UK on Feb 11, 2015.

Tim Kadlec

February 11, 2016
Tweet

More Decks by Tim Kadlec

Other Decks in Programming

Transcript

  1. MOBILE IMAGE
    PROCESSING
    https://flic.kr/p/qRJKUz
    Tim Kadlec | @tkadlec

    View full-size slide

  2. https://flic.kr/p/aShfSP

    View full-size slide

  3. https://flic.kr/p/oPn6ZV

    View full-size slide

  4. 10%
    of photos ever
    2012
    Source: http://time.com/3445111/#1

    View full-size slide

  5. Source: http://time.com/3445111/#1
    1839 2012

    View full-size slide

  6. Source: http://time.com/3445111/#1
    1839 2012
    2011
    172 years, 90%

    View full-size slide

  7. Source: http://time.com/3445111/#1
    1839 2012
    2011
    1 year, 10%

    View full-size slide

  8. Source: http://time.com/3445111/#1
    1839 2012
    2011
    1 year, 10%

    View full-size slide

  9. 0.1
    Source: http://www.kpcb.com/internet-trends

    View full-size slide

  10. 1.8
    billion photos
    PER DAY
    Source: http://www.kpcb.com/internet-trends

    View full-size slide

  11. 360px x 510px

    View full-size slide

  12. 1300px x 1024px

    View full-size slide

  13. Request Decode Copy to GPU Display

    View full-size slide

  14. Request Copy to GPU Display
    Decode

    View full-size slide

  15. Request Copy to GPU Display
    Decode

    View full-size slide

  16. Source: May 1st run of http://httparchive.org
    46%
    JPG
    29%
    PNG
    23%
    GIF
    2%
    Other

    View full-size slide

  17. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  18. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  19. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  20. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  21. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  22. YUV to RGB
    Chroma
    Upsampling
    iDCT /
    Dequantization
    Huffman
    Decoding

    View full-size slide

  23. Resized: 7.86ms

    View full-size slide

  24. Resized: 7.86ms
    Double: 26.34ms

    View full-size slide

  25. Resized: 7.86ms
    Double: 26.34ms
    Large: 266.49ms

    View full-size slide

  26. Resized: 7.86ms
    Double: +235.1%
    Large: +3291.7%

    View full-size slide

  27. 2GB RAM
    Quad-core 1.5 GHz Krait CPU
    Adreno 320 GPU

    View full-size slide

  28. Resized: 30.38ms

    View full-size slide

  29. Resized: 30.38ms
    Double: 102.77ms

    View full-size slide

  30. Resized: 30.38ms
    Double: 102.77ms
    Large (6x): 1,534.99ms

    View full-size slide

  31. Resized: 30.38ms
    Double: +238.3%
    Large (6x): +4952.6%

    View full-size slide

  32. RESIZE
    IMAGES

    View full-size slide

  33. Request Display
    Decode Copy to GPU

    View full-size slide

  34. Request Display
    Decode Copy to GPU

    View full-size slide

  35. Request Copy to GPU Display
    Store in memory
    Decode

    View full-size slide

  36. R: 45
    G: 48
    B: 46
    A: 1
    R: 177
    G: 167
    B: 157
    A: 1
    R: 190
    G: 187
    B: 181
    A: 1

    View full-size slide

  37. 1300px x 1024px

    View full-size slide

  38. 1300 x 1024 x 4

    View full-size slide

  39. 5,324,800
    1300 x 1024 x 4

    View full-size slide

  40. 5.1 MB
    1300 x 1024 x 4

    View full-size slide

  41. 360 x 510 x 4

    View full-size slide

  42. - 734,400
    5,324,800
    4,590,400

    View full-size slide

  43. “…25% of new Android phones
    have only 512MB of RAM.”
    Jen Fitzpatrick
    VP of product management for Google Maps

    View full-size slide

  44. 270px x 936px
    987 kb

    View full-size slide

  45. RESIZE
    IMAGES

    View full-size slide

  46. 600px x 600px 200px x 200px

    View full-size slide

  47. 550px x 550px 150px x 150px

    View full-size slide

  48. (600 x 600 - 550 x 550) x 4 =
    230,000 wasted bytes

    View full-size slide

  49. (200 x 200 - 150 x 150) x 4 =
    70,000 wasted bytes
    230,000 wasted bytes

    View full-size slide

  50. 70,000 wasted bytes
    230,000 wasted bytes

    View full-size slide

  51. 700px 1200px
    200px

    View full-size slide

  52. 1200px
    200px 700px

    View full-size slide

  53. MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD

    View full-size slide

  54. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View full-size slide

  55. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View full-size slide

  56. Luma
    (light)
    Chroma
    (color)

    View full-size slide

  57. Chroma
    Subsampling

    View full-size slide

  58. 2
    2
    4
    4 total
    (50% savings)

    View full-size slide

  59. 4
    1
    1
    2 total
    (75% savings)

    View full-size slide

  60. 4
    2
    0
    2 total
    (75% savings)

    View full-size slide

  61. Request Decode Copy to GPU Display

    View full-size slide

  62. Request Decode Copy to GPU Display
    Decode

    View full-size slide

  63. (W x H x 3)
    -
    (W x H x
    SUBSAMPLE_LEVEL x 2)

    View full-size slide

  64. RGBA 5,324,800
    4:4:4 3,993,600
    4:2:2 2,662,400
    4:2:0 1,996,800
    1300px x 1024px

    View full-size slide

  65. 62.5%
    memory savings
    4:2:0
    Source: http://time.com/3445111/#1

    View full-size slide

  66. https://flic.kr/p/aaThYm

    View full-size slide

  67. https://flic.kr/p/aaThYm

    View full-size slide

  68. Source: May 1st run of http://calendar.perfplanet.com/2015/why-arent-your-images-using-chroma-subsampling/
    39.3%
    4:2:0
    0.9%
    Other
    59.7%
    4:4:4

    View full-size slide

  69. Browser OS
    Psst….need
    some memory?

    View full-size slide

  70. Browser OS
    Yeah, I’ll take
    some of that.

    View full-size slide

  71. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View full-size slide

  72. content=“width=device-width,
    minimum-scale=1.0”
    />

    View full-size slide

  73. Resized: 30.38ms

    View full-size slide

  74. Resized: 5.62ms

    View full-size slide

  75. Resized: 5.62ms
    Double: 102.77ms

    View full-size slide

  76. Resized: 5.62ms
    Double: 27.74ms

    View full-size slide

  77. Resized: 5.62ms
    Double: 27.74ms
    Large (6x): 1534.99ms

    View full-size slide

  78. Resized: 5.62ms
    Double: 27.74ms
    Large (6x): 462.11ms

    View full-size slide

  79. Resized: 5.62ms
    Double: +393.6%
    Large (6x): +8122.6%

    View full-size slide

  80. name=“viewport”
    content=“initial-scale = 1.0,
    maximum-scale = 1.0”
    />

    View full-size slide

  81. name=“viewport”
    content=“initial-scale = 1.0,
    maximum-scale = 1.0,
    width=device-width”
    />

    View full-size slide

  82. MICRO-OPTIMIZATIONS?

    View full-size slide

  83. “…as bandwidth grows, and as
    processing power grows, and as
    browsers get better we just keep
    filling everything up. ”
    Jeff Veen

    View full-size slide

  84. NETWORK
    MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD

    View full-size slide

  85. RESIZE IMAGES
    1

    View full-size slide

  86. RESIZE IMAGES
    1
    2 SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View full-size slide

  87. RESIZE IMAGES
    1
    2
    META VIEWPORT IS YOUR FRIEND
    3
    SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View full-size slide

  88. RESIZE IMAGES
    1
    2
    META VIEWPORT IS YOUR FRIEND
    3
    USE 4:2:0 SUBSAMPLING
    4
    SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View full-size slide

  89. THANK YOU
    Tim Kadlec | @tkadlec

    View full-size slide