Mobile Image Processing at Velocity SC, 2015

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at Velocity SC 2015 in Santa Clara, CA on May 29, 2015.

Transcript

1. MOBILE IMAGE
   PROCESSING
   https://flic.kr/p/qRJKUz
   Tim Kadlec | @tkadlec
   

   View Slide

2. https://flic.kr/p/aShfSP
   

   View Slide

3. View Slide

4. https://flic.kr/p/oPn6ZV
   

   View Slide

5. View Slide

6. 10%
   of photos ever
   2012
   Source: http://time.com/3445111/#1
   

   View Slide

7. Source: http://time.com/3445111/#1
   1839 2012
   2011
   172 years, 90%
   

   View Slide

8. Source: http://time.com/3445111/#1
   1839 2012
   2011
   1 year, 10%
   

   View Slide

9. billion photos
   PER DAY
   Source: http://www.kpcb.com/internet-trends
   1.8
   

   View Slide

10. Source: May 1st run of http://httparchive.org
    789kb
    Everything else
    1310kb
    Images
    

    View Slide

11. Source: May 1st run of http://httparchive.org
    36.5%
    Everything else
    62.4%
    Images
    

    View Slide

12. View Slide

13. View Slide

14. 360px x 202px
    

    View Slide

15. 1280px x 520px
    

    View Slide

16. Request Decode Copy to GPU Display
    

    View Slide

17. Request Copy to GPU Display
    Decode
    

    View Slide

18. 1280px x 520px
    

    View Slide

19. Source: May 1st run of http://httparchive.org
    46%
    JPG
    29%
    PNG
    23%
    GIF
    2%
    Other
    

    View Slide

20. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding
    

    View Slide

21. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding
    

    View Slide

22. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding
    

    View Slide

23. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding
    

    View Slide

24. RGB to YUV
    Chroma
    Subsampling
    DCT /
    Quantization
    Huffman
    Encoding
    

    View Slide

25. YUV to RGB
    Chroma
    Upsampling
    iDCT /
    Dequantization
    Huffman
    Decoding
    

    View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. Resized: 7.86ms
    

    View Slide

36. Resized: 7.86ms
    Double: 26.34ms
    

    View Slide

37. Resized: 7.86ms
    Double: 26.34ms
    Large: 266.49ms
    

    View Slide

38. Resized: 7.86ms
    Double: +235.1%
    Large: +3291.7%
    

    View Slide

39. View Slide

40. 2GB RAM
    Quad-core 1.5 GHz Krait CPU
    Adreno 320 GPU
    

    View Slide

41. Resized: 30.38ms
    

    View Slide

42. Resized: 30.38ms
    Double: 102.77ms
    

    View Slide

43. Resized: 30.38ms
    Double: 102.77ms
    Large (6x): 1,534.99ms
    

    View Slide

44. Resized: 30.38ms
    Double: +238.3%
    Large (6x): +4952.6%
    

    View Slide

45. View Slide

46. RESIZE IMAGES
    

    View Slide

47. RESIZE IMAGES
    

    View Slide

48. Request Display
    Decode Copy to GPU
    

    View Slide

49. Request Copy to GPU Display
    Store in memory
    Decode
    

    View Slide

50. View Slide

51. R: 65
    G: 47
    B: 35
    A: 1
    R: 26
    G: 26
    B: 29
    A: 1
    R: 30
    G: 29
    B: 32
    A: 1
    

    View Slide

52. W x H x 4
    

    View Slide

53. 1280px x 520px
    

    View Slide

54. 1280 x 520 x 4
    

    View Slide

55. 1280 x 520 x 4
    2,662,400
    

    View Slide

56. 2.54 MB
    1280 x 520 x 4
    

    View Slide

57. View Slide

58. View Slide

59. “…25% of new Android phones
    have only 512MB of RAM.”
    Jen Fitzpatrick
    VP of product management for Google Maps
    

    View Slide

60. 360 x 202 x 4
    

    View Slide

61. 360 x 202 x 4
    290,880
    

    View Slide

62. - 290,880
    2,662,400
    2,371,520
    

    View Slide

63. 2.26 MB
    

    View Slide

64. 270px x 936px
    987 kb
    

    View Slide

65. RESIZE IMAGES
    

    View Slide

66. 600px x 600px 200px x 200px
    

    View Slide

67. 550px x 550px 150px x 150px
    

    View Slide

68. (600 x 600 - 550 x 550) x 4 =
    230,000 wasted bytes
    

    View Slide

69. (200 x 200 - 150 x 150) x 4 =
    70,000 wasted bytes
    230,000 wasted bytes
    

    View Slide

70. 70,000 wasted bytes
    230,000 wasted bytes
    

    View Slide

71. 700px 1200px
    200px
    

    View Slide

72. 1200px
    200px 700px
    

    View Slide

73. MEMORY
    BATTERY
    LIFE OF DEVICE
    CPU/GPU OVERHEAD
    

    View Slide

74. View Slide

75. iDCT /
    Dequantization
    Huffman
    Decoding
    Chroma
    Upsampling
    YUV to RGB
    

    View Slide

76. View Slide

77. Luma
    (light)
    Chroma
    (color)
    

    View Slide

78. Y
    CB
    CR
    

    View Slide

79. Chroma
    Subsampling
    

    View Slide

80. J : a : b
    

    View Slide

81. View Slide

82. 4 : a : b
    

    View Slide

83. View Slide

84. View Slide

85. View Slide

86. 4 : 4 : 4
    

    View Slide

87. View Slide

88. View Slide

89. 4
    4
    4
    8 total
    

    View Slide

90. View Slide

91. View Slide

92. View Slide

93. 4 : 2 : 2
    

    View Slide

94. View Slide

95. View Slide

96. 2
    2
    4
    4 total
    (50% savings)
    

    View Slide

97. View Slide

98. View Slide

99. View Slide

100. View Slide

101. 4 : 1 : 1
     

     View Slide

102. View Slide

103. View Slide

104. 4
     1
     1
     2 total
     (75% savings)
     

     View Slide

105. View Slide

106. View Slide

107. View Slide

108. View Slide

109. 4 : 2 : 0
     

     View Slide

110. View Slide

111. View Slide

112. 4
     2
     0
     2 total
     (75% savings)
     

     View Slide

113. View Slide

114. View Slide

115. View Slide

116. View Slide

117. Request Decode Copy to GPU Display
     

     View Slide

118. Request Decode Copy to GPU Display
     Decode
     

     View Slide

119. (W x H x 3)
     -
     (W x H x
     SUBSAMPLE_LEVEL x 2)
     

     View Slide

120. RGBA 2,662,400
     4:4:4 1,996,800
     4:2:2 1,331,200
     4:2:0 998,400
     1280px x 520px
     

     View Slide

121. 62.5%
     memory savings
     4:2:0
     Source: http://time.com/3445111/#1
     

     View Slide

122. View Slide

123. View Slide

124. View Slide

125. View Slide

126. > 26MB
     

     View Slide

127. Browser OS
     

     View Slide

128. Browser OS
     Psst….need
     some memory?
     

     View Slide

129. Browser OS
     Yeah, I’ll take
     some of that.
     

     View Slide

130. iDCT /
     Dequantization
     Huffman
     Decoding
     Chroma
     Upsampling
     YUV to RGB
     

     View Slide

131. View Slide

132. content=“width=device-width,
     minimum-scale=1.0”
     />
     

     View Slide

133. View Slide

134. Resized: 30.38ms
     

     View Slide

135. Resized: 5.62ms
     

     View Slide

136. Resized: 5.62ms
     Double: 102.77ms
     

     View Slide

137. Resized: 5.62ms
     Double: 27.74ms
     

     View Slide

138. Resized: 5.62ms
     Double: 27.74ms
     Large (6x): 1534.99ms
     

     View Slide

139. Resized: 5.62ms
     Double: 27.74ms
     Large (6x): 462.11ms
     

     View Slide

140. Resized: 5.62ms
     Double: +393.6%
     Large (6x): +8122.6%
     

     View Slide

141. MICRO-OPTIMIZATIONS?
     

     View Slide

142. “…as bandwidth grows, and as
     processing power grows, and as
     browsers get better we just keep
     filling everything up. ”
     Jeff Veen
     

     View Slide

143. NETWORK
     MEMORY
     BATTERY
     LIFE OF DEVICE
     CPU/GPU OVERHEAD
     

     View Slide

144. RESIZE IMAGES
     1
     

     View Slide

145. RESIZE IMAGES
     1
     2 SMALL BREAKPOINTS FOR
     LARGE IMAGES
     

     View Slide

146. RESIZE IMAGES
     1
     2
     META VIEWPORT IS YOUR FRIEND
     3
     SMALL BREAKPOINTS FOR
     LARGE IMAGES
     

     View Slide

147. RESIZE IMAGES
     1
     2
     META VIEWPORT IS YOUR FRIEND
     3
     USE 4:2:0 SUBSAMPLING
     4
     SMALL BREAKPOINTS FOR
     LARGE IMAGES
     

     View Slide

148. THANK YOU
     Tim Kadlec | @tkadlec
     

     View Slide