Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mobile Image Processing at Velocity SC, 2015

Mobile Image Processing at Velocity SC, 2015

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at Velocity SC 2015 in Santa Clara, CA on May 29, 2015.

Tim Kadlec

May 29, 2015
Tweet

More Decks by Tim Kadlec

Other Decks in Technology

Transcript

  1. MOBILE IMAGE
    PROCESSING
    https://flic.kr/p/qRJKUz
    Tim Kadlec | @tkadlec

    View full-size slide

  2. https://flic.kr/p/aShfSP

    View full-size slide

  3. https://flic.kr/p/oPn6ZV

    View full-size slide

  4. 10%
    of photos ever
    2012
    Source: http://time.com/3445111/#1

    View full-size slide

  5. Source: http://time.com/3445111/#1
    1839 2012
    2011
    172 years, 90%

    View full-size slide

  6. Source: http://time.com/3445111/#1
    1839 2012
    2011
    1 year, 10%

    View full-size slide

  7. billion photos
    PER DAY
    Source: http://www.kpcb.com/internet-trends
    1.8

    View full-size slide

  8. Source: May 1st run of http://httparchive.org
    789kb
    Everything else
    1310kb
    Images

    View full-size slide

  9. Source: May 1st run of http://httparchive.org
    36.5%
    Everything else
    62.4%
    Images

    View full-size slide

  10. 360px x 202px

    View full-size slide

  11. 1280px x 520px

    View full-size slide

  12. Request Decode Copy to GPU Display

    View full-size slide

  13. Request Copy to GPU Display
    Decode

    View full-size slide

  14. 1280px x 520px

    View full-size slide

  15. Source: May 1st run of http://httparchive.org
    46%
    JPG
    29%
    PNG
    23%
    GIF
    2%
    Other

    View full-size slide

  16. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  17. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  18. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  19. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  20. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding

    View full-size slide

  21. YUV to RGB
    Chroma
    Upsampling
    iDCT /
    Dequantization
    Huffman
    Decoding

    View full-size slide

  22. Resized: 7.86ms

    View full-size slide

  23. Resized: 7.86ms
    Double: 26.34ms

    View full-size slide

  24. Resized: 7.86ms
    Double: 26.34ms
    Large: 266.49ms

    View full-size slide

  25. Resized: 7.86ms
    Double: +235.1%
    Large: +3291.7%

    View full-size slide

  26. 2GB RAM
    Quad-core 1.5 GHz Krait CPU
    Adreno 320 GPU

    View full-size slide

  27. Resized: 30.38ms

    View full-size slide

  28. Resized: 30.38ms
    Double: 102.77ms

    View full-size slide

  29. Resized: 30.38ms
    Double: 102.77ms
    Large (6x): 1,534.99ms

    View full-size slide

  30. Resized: 30.38ms
    Double: +238.3%
    Large (6x): +4952.6%

    View full-size slide

  31. RESIZE IMAGES

    View full-size slide

  32. RESIZE IMAGES

    View full-size slide

  33. Request Display
    Decode Copy to GPU

    View full-size slide

  34. Request Copy to GPU Display
    Store in memory
    Decode

    View full-size slide

  35. R: 65
    G: 47
    B: 35
    A: 1
    R: 26
    G: 26
    B: 29
    A: 1
    R: 30
    G: 29
    B: 32
    A: 1

    View full-size slide

  36. 1280px x 520px

    View full-size slide

  37. 1280 x 520 x 4

    View full-size slide

  38. 1280 x 520 x 4
    2,662,400

    View full-size slide

  39. 2.54 MB
    1280 x 520 x 4

    View full-size slide

  40. “…25% of new Android phones
    have only 512MB of RAM.”
    Jen Fitzpatrick
    VP of product management for Google Maps

    View full-size slide

  41. 360 x 202 x 4

    View full-size slide

  42. 360 x 202 x 4
    290,880

    View full-size slide

  43. - 290,880
    2,662,400
    2,371,520

    View full-size slide

  44. 270px x 936px
    987 kb

    View full-size slide

  45. RESIZE IMAGES

    View full-size slide

  46. 600px x 600px 200px x 200px

    View full-size slide

  47. 550px x 550px 150px x 150px

    View full-size slide

  48. (600 x 600 - 550 x 550) x 4 =
    230,000 wasted bytes

    View full-size slide

  49. (200 x 200 - 150 x 150) x 4 =
    70,000 wasted bytes
    230,000 wasted bytes

    View full-size slide

  50. 70,000 wasted bytes
    230,000 wasted bytes

    View full-size slide

  51. 700px 1200px
    200px

    View full-size slide

  52. 1200px
    200px 700px

    View full-size slide

  53. MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD

    View full-size slide

  54. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View full-size slide

  55. Luma
    (light)
    Chroma
    (color)

    View full-size slide

  56. Chroma
    Subsampling

    View full-size slide

  57. 2
    2
    4
    4 total
    (50% savings)

    View full-size slide

  58. 4
    1
    1
    2 total
    (75% savings)

    View full-size slide

  59. 4
    2
    0
    2 total
    (75% savings)

    View full-size slide

  60. Request Decode Copy to GPU Display

    View full-size slide

  61. Request Decode Copy to GPU Display
    Decode

    View full-size slide

  62. (W x H x 3)
    -
    (W x H x
    SUBSAMPLE_LEVEL x 2)

    View full-size slide

  63. RGBA 2,662,400
    4:4:4 1,996,800
    4:2:2 1,331,200
    4:2:0 998,400
    1280px x 520px

    View full-size slide

  64. 62.5%
    memory savings
    4:2:0
    Source: http://time.com/3445111/#1

    View full-size slide

  65. Browser OS
    Psst….need
    some memory?

    View full-size slide

  66. Browser OS
    Yeah, I’ll take
    some of that.

    View full-size slide

  67. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB

    View full-size slide

  68. content=“width=device-width,
    minimum-scale=1.0”
    />

    View full-size slide

  69. Resized: 30.38ms

    View full-size slide

  70. Resized: 5.62ms

    View full-size slide

  71. Resized: 5.62ms
    Double: 102.77ms

    View full-size slide

  72. Resized: 5.62ms
    Double: 27.74ms

    View full-size slide

  73. Resized: 5.62ms
    Double: 27.74ms
    Large (6x): 1534.99ms

    View full-size slide

  74. Resized: 5.62ms
    Double: 27.74ms
    Large (6x): 462.11ms

    View full-size slide

  75. Resized: 5.62ms
    Double: +393.6%
    Large (6x): +8122.6%

    View full-size slide

  76. MICRO-OPTIMIZATIONS?

    View full-size slide

  77. “…as bandwidth grows, and as
    processing power grows, and as
    browsers get better we just keep
    filling everything up. ”
    Jeff Veen

    View full-size slide

  78. NETWORK
    MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD

    View full-size slide

  79. RESIZE IMAGES
    1

    View full-size slide

  80. RESIZE IMAGES
    1
    2 SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View full-size slide

  81. RESIZE IMAGES
    1
    2
    META VIEWPORT IS YOUR FRIEND
    3
    SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View full-size slide

  82. RESIZE IMAGES
    1
    2
    META VIEWPORT IS YOUR FRIEND
    3
    USE 4:2:0 SUBSAMPLING
    4
    SMALL BREAKPOINTS FOR
    LARGE IMAGES

    View full-size slide

  83. THANK YOU
    Tim Kadlec | @tkadlec

    View full-size slide