Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Performance Images

Colin Bendell
November 09, 2016

High Performance Images

Images are an important part of the user experience for the Web and apps alike. However, images are no longer simple. There are many strategies for optimizing images: new image formats, responsive image markup, client hints, chroma subsampling, structural similarity, HTTP/2 priorities, progressive encoding, and more. Each of these strategies improves the end-user experience but can add new complexities to creating maintaining and monitoring the enterprise.

It isn’t just the end consumer that needs high-performance images; other competing users need to be considered, including marketing, devs, operations, and security. These users different requirements and expectations:

Users not only prefer images over text but want images that load quickly on their device (without eating up cell data)
Marketing teams want high-quality, fantastic-looking images to promote the brand and products
The web teams want to quickly iterate over the frontend and adjust image requirements for responsive web layouts
Operations and infrastructure want highly cacheable, limited file footprints and a small disaster-recovery window
Security wants to make sure that each image transformed isn’t creating a backdoor vulnerability for “bad guys”
Sometimes promoting one set of performance improvements feels like a Faustian bargain, where some users are benefiting more than others. In this practical, Choose Your Own Adventure-style tutorial, Colin Bendell tries to find the perfect balance for all users, offering a practical guide that explores different strategies and outlines recommendations to address each of the different user profiles.

Colin Bendell

November 09, 2016
Tweet

More Decks by Colin Bendell

Other Decks in Technology

Transcript

  1. VS.

  2. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business"
  3. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business"
  4. H T T PA R C H I V E

    . O R G : I M A G E S > 6 0 % 0KB 400KB 800KB 1,200KB 1,600KB 0 25 50 75 100 125 15-NOV-10 15-MAR-11 15-JUL-11 15-NOV-11 15-MAR-12 15-JUL-12 15-NOV-12 15-MAR-13 15-JUL-13 15-NOV-13 1-APR-14 1-AUG-14 1-DEC-14 1-APR-15 1-AUG-15 1-DEC-15 1-APR-16 1-AUG-16 1,577KB 416KB 48 56
  5. R W D : O N E U R L

    , A D A P T T O S C R E E N S I Z E
  6. AUTOMOTIVE BUSINESS CMG EDUCATION FIN. TECH NOT 4 P GAMING

    HIGH TECH TRAVEL MANUFACTURIN MEDIA HEALTH PUBLIC SECTOR RETAIL SAAS 38KB 27KB 46KB 29KB 22KB 36KB 42KB 51KB 45KB 56KB 42KB 71KB 33KB 47KB 72KB AVERAGE JPG SIZE
  7. 360X 760X 1260X 370 KB 293 KB 168 KB 635

    KB 623 KB 604 KB Responsive Images can save 72% of bytes! Tim Kadlec
  8. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business"
  9. S P E C T R A L S E

    N S I T I V I T Y 
 ( P R O B A B I L I T Y V. WAV E L E N G T H )
  10. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business"
  11. 0 18.75 37.5 56.25 75 4KB SHALLOW DIRECTORY STRUCTURE 4KB

    DEEP DIRECTORY SHALLOW 4MB SHALLOW DIRECTORY STRUCTURE 4MB DEEP DIRECTORY STRUCTURE 72MB/s 80MB/s 2MB/s 4MB/s
  12. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business"
  13. CRITICAL VULNERABILITIES AND EXPOSURE (CVE) 0 4.5 9 13.5 18

    2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 ImageMagick libjpegturbo libjpeg libpng giflib openjpeg
  14. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business"
  15. 0 2 4 6 8 10 12 14 16 18

    20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 M A G I C N U M B E R : 2 S E C O N D S ( E V E N O N C E L L U L A R )
  16. ☑︎make images "fast"
 ☑︎use responsive images
 ☑︎high quality (p3, retina)


    ☑︎Disaster Recovery 
 ☑︎secure
 ☑︎get stuff done by monday
  17. O P T I M I Z E S V

    G • Scour
 www.coderead.com/scour/ • SVGO
 github.com/svg/svgo • SVGOMG (gui)
 jakearchibald.github.io/svgomg
  18. B O N U S : S V G C

    S S S T Y L E W I T H W I D E G A M U T .main { background: rgb(241, 0, 0); } @media (color-gamut: p3) { .main { background: rgb(244, 0, 0); } } /* future */ @media (color-gamut: p3) { .main { background: color(p3 1.0 0 0); } }
  19. RASTER IMAGES:
 MADE OF PIXELS • GIF • PNG •

    WebP • JPEG • JPEG 2000 • JPEG XR
  20. G I F • 1987 (Previously Patented) • 256 Colors

    • “Simple” Transparency • Supported Everywhere • Universal support for Animation • image/gif
  21. P N G – P O R TA B L

    E N E T W O R K G R A P H I C S • 1996 (no patent) • PNG8 and PNG24 • Supports Alpha Transparency • Good browser support • image/png • image/apng
  22. L O S S L E S S P N

    G O P T I M I Z AT I O N - Convert from PNG 24 -> 8 - Trade “Alpha” transparency with simple (100%) transparency - PNG Optimization Tools: TinyPNG, PNGCrush, OptiPNG
  23. J P E G – J O I N T

    P H O T O G R A P H I C E X P E R T S G R O U P - 21 years old (1992) - Supports RGB Colors (8 bit per channel) - First major lossy format. - No transparency support. - Extensive chroma subsampling support. - Supports progressive loading. - Compressed with: - Discrete Cosine Transformation - Quantization - Huffman encoding
  24. J P E G C O M P R E

    S S I O N 1. Divide image into 8x8 pixel blocks
  25. J P E G C O M P R E

    S S I O N 1. Divide image into 8x8 pixel blocks 2. Split Luminance & Chrominance Y Cb Cr
  26. J P E G C O M P R E

    S S I O N - S I M P L I F I E D 1. Divide image into 8x8 pixel blocks 2. Split Luminance & Chrominance 3. Write pixels as ~delta from average • and round (DCT + Quantization) Y Cb Cr
  27. Define Quantization Table 0 precision 0 160 110 100 160

    240 255 255 255 120 120 140 190 255 255 255 255 140 130 160 240 255 255 255 255 140 170 220 255 255 255 255 255 180 220 255 255 255 255 255 255 240 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 Define Quantization Table 1 precision 0 170 180 240 255 255 255 255 255 180 210 255 255 255 255 255 255 240 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 Define Quantization Table 0 precision 0 10 7 6 10 14 24 31 37 7 7 8 11 16 35 36 33 8 8 10 14 24 34 41 34 8 10 13 17 31 52 48 37 11 13 22 34 41 65 62 46 14 21 33 38 49 62 68 55 29 38 47 52 62 73 72 61 43 55 57 59 67 60 62 59 Define Quantization Table 1 precision 0 10 11 14 28 59 59 59 59 11 13 16 40 59 59 59 59 14 16 34 59 59 59 59 59 28 40 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 59 QUALITY 70 QUALITY 5
  28. J P E G C O M P R E

    S S I O N 1. Divide image into 8x8 pixel blocks 2. Split Luminance & Chrominance 3. Write pixels as ~delta from average and round (DCT + Quantization) 4. Write as line, compress (Huffman) = -9,-1,0(x62)
  29. J P E G C O M P R E

    S S I O N 1. Divide image into 8x8 pixel blocks 2. Split Luminance & Chrominance 3. Write pixels as delta from average and round (DCT + Quantization) 4. Write as line, compress (Huffman) Q=10 Q=50 Original
  30. L O S S Y C O M P R

    E S S I O N – N O T F O R E V E RY O N E
  31. J P E G B A S E L I

    N E V S . P R O G R E S S I V E http://www.bookofspeed.com/chapter5.html 1 2 0 0 1 0 0 0 3 -1 0 0 1 2 0 0 1 0 0 0 3 -1 0 0 1 1 3 2 0 -1 0 0 0 0 0 0 1 1 3 2 0 -1 EOB
  32. J P E G 2 0 0 0 • Supports

    full transparency • Supports chroma subsampling • Supports progressive loading • Compressed with: • Discrete Wavelet Transformation • Quantization • Arithmetic encoding
  33. W E B P ( 2 0 1 1 )

    • Supports full transparency. • Supports animation on some recent browsers. • Supports only 4:2:0 chroma subsampling. • Does NOT support progressive loading. • Compresses with: • Discrete Cosine Transformation • Block prediction • Quantization • Arithmetic encoding
  34. J P E G X R • Supports full transparency.

    • Supports chroma subsampling. • Supports progressive loading. • Compressed with: • Hierarchical Lapped Biorthogonal Transform • Block prediction • Quantization • Huffman encoding
  35. F O R M AT F E AT U R

    E S U P P O R T
  36. O P T I M I Z E R A

    S T E R • Use JPeg2000 for iOS/Safari • Use WebP for Android/Chrome • Progressive Jpeg • Image Optim (Jpeg/PNG/Gif): https://imageoptim.com • Tiny PNG: https://tinypng.com/
  37. 0 2 4 6 8 10 12 14 16 18

    20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 NOT YOU YOU
  38. 0 2 4 6 8 10 12 14 16 18

    20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 M A G I C N U M B E R : 2 S E C O N D S ( E V E N O N C E L L U L A R )
  39. Nexus 7 Android 1 Moto e Moto g Nexus 5

    334ms 1002ms 1180ms 222ms 494ms Desktop 103ms 2.0s 1.0s 3.0s N O T A L L M O B I L E D E V I C E S M A D E E Q U A L
  40. 0 1250 2500 3750 5000 CARRIER 1 CARRIER 3 CARRIER

    4 CARRIER 5 CARRIER 6 HTTP TLS
  41. 1 2 8 0 X 5 2 0 X 4

    B Y T E S = 2 . 6 M B !
  42. Nexus 7 Android 1 Moto e Moto g Nexus 5

    334ms 1002ms 1180ms 222ms 494ms Desktop 103ms 2.0s 1.0s 3.0s N O T A L L M O B I L E D E V I C E S M A D E E Q U A L
  43. SSIM(x, y) = (2µx µy +c 1 )(2σxy +c 2

    ) (µx 2 +µy 2 +c 1 )(σx 2 +σy 2 +c 2 ) Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli
  44. QUALITY V. DSSIM 0.0000 0.0400 0.0800 0.1200 0.1600 0.2000 0.2400

    0.2800 0.3200 0.3600 0.4000 100 95 90 85 80 75 70 65 60 55 50 45 40 35 30 25 20 15 10 5 0 JPEG (libjpegturbo) JPEG (mozjpeg) JPEG 2000 (OpenJPEG) JPEG 2000 (Kakadu) JPEG XR WebP
  45. QUALITY (90-40) V. DSSIM 0.0000 0.0109 0.0218 0.0327 0.0436 0.0545

    0.0655 0.0764 0.0873 0.0982 0.1091 0.1200 90 85 80 75 70 65 60 55 50 45 40 JPEG (libjpegturbo) JPEG (mozjpeg) JPEG 2000 (OpenJPEG) JPEG 2000 (Kakadu) JPEG XR WebP
  46. O P T I M I Z E F O

    R S S I M • RGBA Structural Similarity
 kornel.ski/dssim • butteraulgi
 github.com/google/butteraugli
  47. R G B A = 2 . 6 M B

    ! 4 : 4 : 4 = 1 . 9 M B 4 : 2 : 2 = 1 . 3 M B 4 : 2 : 0 = 0 . 9 M B
  48. O P T I M I Z E M E

    TA D ATA • STRIP Most Metadata • Keep Copyright • Keep ICC (convert to RGB first!)
  49. <div style="width: 240px"> <div class="delayed-image-load" data-src="/is_awesome_{width_{pixel_ratio}.jpg"> </div> </div> <script> new

    Imager({ availableWidths: [200, 260, 320, 600], availablePixelRatios: [1, 1.3, 2] }); </script>
  50. <img src="/is_awesome_150.jpg" srcset="/is_awesome_150.jpg 150w, /is_awesome_300.jpg 300w, /is_awesome_600.jpg 600w, /is_awesome_800.jpg 800w,

    /is_awesome_1000.jpg 1000w, /is_awesome_1200.jpg 1200w, /is_awesome_2000.jpg 2000w" sizes="(min-width: 500px) 33.3vw, 100vw" />
  51. <img src="/is_awesome_150.jpg" srcset="/is_awesome_150.jpg 150w, /is_awesome_300.jpg 300w, /is_awesome_600.jpg 600w, /is_awesome_800.jpg 800w,

    /is_awesome_1000.jpg 1000w, /is_awesome_1200.jpg 1200w, /is_awesome_2000.jpg 2000w" sizes="(min-width: 500px) 33.3vw, 100vw" />
  52. <img src="/is_awesome_150.jpg" srcset="/is_awesome_150.jpg 150w, /is_awesome_300.jpg 300w, /is_awesome_600.jpg 600w, /is_awesome_800.jpg 800w,

    /is_awesome_1000.jpg 1000w, /is_awesome_1200.jpg 1200w, /is_awesome_2000.jpg 2000w" sizes="(min-width: 500px) 33.3vw, 100vw" />
  53. <img src="/is_awesome_150.jpg" srcset="/is_awesome_150.jpg 150w, /is_awesome_300.jpg 300w, /is_awesome_600.jpg 600w, /is_awesome_800.jpg 800w,

    /is_awesome_1000.jpg 1000w, /is_awesome_1200.jpg 1200w, /is_awesome_2000.jpg 2000w" sizes="(min-width: 500px) 33.3vw, 100vw" />
  54. 1. Use srcset to list image dimensions 2. Use 'w'

    notation for pixel 'widths' 3. don't use 'x' for DPR 4. Use sizes to help the browser pick the right size
 (defaults to using the device viewport) 5. Use small src for default
  55. 0KB 23KB 45KB 68KB 90KB 150X 300X 600X 800X 123.0KB

    58.3KB 21.2KB 7.6KB 74.7KB 33.9KB 12.0KB 4.1KB 46.3KB 20.3KB 7.3KB 2.5KB 25th Median 75th
  56. 0KB 175KB 350KB 525KB 700KB 1000X 1200X 2000X 4000X 953KB

    388KB 251KB 186KB 432KB 320KB 144KB 113KB 217KB 120KB 83KB 70KB 25th Median 75th
  57. 1. Use SrcSet not <picture> for Responsive Images 2. Use

    a polyfill 
 scottjehl.github.io/picturefill/ 3. Use lazy sizes polyfill
 github.com/aFarkas/lazysizes 4. Hide hidden Images 5. Hide Blow the Fold R E S P O N S I V E I M A G E S
  58. <picture> <source media="(min-width: 320px)" srcset="" <source media="(color-gamut: p3)" srcset="" <source

    type="image/webp" srcset="" <source media="(min-width: 320px) and (color-gamut: p3)" type="image/webp" srcset=""
  59. <picture> <source type="image/webp" srcset="/is_awesome_200.webp 100w, /is_awesome_400.webp 400w, /is_awesome_800.webp 800w, /is_awesome_1000.webp

    1000w, /is_awesome_1400.webp 1400w, /is_awesome_1800.webp 1800w, /is_awesome_2200.webp 2200w"/> <source type="image/vnd.ms-photo" srcset="/is_awesome_200.jxr 100w, /is_awesome_400.jxr 400w, /is_awesome_800.jxr 800w, /is_awesome_1000.jxr 1000w, /is_awesome_1400.jxr 1400w, /is_awesome_1800.jxr 1800w, /is_awesome_2200.jxr 2200w"/> <source type="image/jp2" srcset="/is_awesome_200.jp2 100w, /is_awesome_400.jp2 400w, /is_awesome_800.jp2 800w, /is_awesome_1000.jp2 1000w, /is_awesome_1400.jp2 1400w, /is_awesome_1800.jp2 1800w, /is_awesome_2200.jp2 2200w"/> <img src="/is_awesome_100.jpg" srcset="/is_awesome_200.jpg 100w, /is_awesome_400.jpg 400w, /is_awesome_800.jpg 800w, /is_awesome_1000.jpg 1000w, /is_awesome_1400.jpg 1400w, /is_awesome_1800.jpg 1800w, /is_awesome_2200.jpg 2200w" sizes="(min-width: 500px) 33.3vw, 100vw"/> </picture>
  60. GET /You-Won't-Believe-What-Happened-Next.html 200 OK (PS: send Hints) GET /giant-galoshes.jpg (PS:

    I'm a S7 note; on cellular) (PPS: I'm on fire) 200 OK (PS: I shrunk the image) CLIENT HINTS: BACK TO BASICS
  61. GET /is_awesome.jpg Accept: image/webp, */* DPR: 2 Width: 300 Viewport-Width:

    375 User-Agent: Mozilla/5.0 (iPhone; CPU OS 10_0 ...
  62. HTTP/1.1 200 OK Content-Type: image/jp2 Content-DPR: 2 Vary: Accept, DPR,

    Width response.setHeader('Content-Type', 'image/jp2'); response.setHeader('Content-DPR', '2'); response.setHeader('Vary', 'Accept, DPR, Width');
  63. <html> <head> <script type="application/javascript"> // place JavaScript at top of

    the page to prevent race condition // reset the CH (ClientHint) cookie values with actual display width if (document.cookie.match(/(^|;)\s*CH-Verify=/)) { var width = (screen.availWidth || screen.width); var dpr = (window.devicePixelRatio || 1); document.cookie = 'CH=Viewport-Width=' + width + ',DPR=' + dpr + ';expires=Fri, 31 Dec 9999 23:59:59 GMT;path=/'; document.cookie ='CH-Set=; expires=Thu, 01 Jan 1970 00:00:01 GMT;' } </script>
  64. if (Save-Data = 'on') then DPR = 1 let image_width

    = Width || Viewport-Width * DPR || Cookie.Viewport-Width * Cookie.DPR || DefaultWidth
  65. /// /// Add Client Hints to the HTTP request for

    images to populate the UIImageView. /// This will interrogate the screen and UIView to determine the hint values. /// func clientHints(imageUrl: String, targetImage:UIImageView) { let nsURL = NSURL(string: imageUrl) let config = NSURLSessionConfiguration.defaultSessionConfiguration() let screen = UIScreen.mainScreen() /// use the main screen for size and scale; UIView frame for let viewportWidthPx = screen.bounds.size.width let dpr = screen.scale let width = targetImage.frame.size.width /// convert to CSS Pixels let viewportWidth = Int(Double(viewportWidthPx) / Double(dpr)) config.HTTPAdditionalHeaders = ["Viewport-Width" : viewportWidth] config.HTTPAdditionalHeaders = ["DPR" : dpr] config.HTTPAdditionalHeaders = ["Width" : width] config.HTTPAdditionalHeaders = ["Save-Data" : "on"] /// usual NSURLSession to UIImage work from here /// Accept-CH likely won't make sense since your app /// controls the UIView dimensions let session = NSURLSession(configuration: config) let task = session.dataTaskWithURL(nsURL) { (data, response, error) in if !error { /// make sure that the image is drawn on screen var image:UIImage = UIImage(data: data) dispatch_async(dispatch_get_main_queue(), { targetImage.image = image }) } } task.resume() }
  66. F O R M AT F E AT U R

    E S U P P O R T
  67. 1. Opt-In to Client Hints (for 3rd party content!) <meta

    http-equiv="Accept-CH" content="DPR, Viewport-Width, Width"> 2. Add sizes to <img> <img src="/is_awesome.jpg" sizes="(min-width: 500px) 33.3vw, 100vw" /> 3. Use Accepts header and Device Detection P R I O R I T Y L I S T
  68. User Batman Security Marketing Dev Ops W H O C

    A R E S A B O U T I M A G E S ? "Business" ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
  69. 1. Optimize for Mobile a) Use SVG, WebP & Jpeg

    2000 b) Resize Images & Use Chroma-Subsampling c) DSSIM not Quality Index d) Use TLS 2. Use Client Hints (& Lazy Loading) with 
 Breakpoints: 150, 300, 600, 800 1000, 1200, 2000, 4000 3. Build a Disaster-Recovery-Plan 4. Monitor user performance S U M M A RY
  70. RGB

  71. YCgCo v. YCbCr R = Y + 1.402 (Cr -

    128) G = Y - 0.34414 (Cb - 128) - 0.71414 (Cr - 128) B = Y + 1.772 (Cb - 128) Y 1/4 1/2 1/4 R Cg = -1/4 1/2 -1/4 G Co 1/2 0 -1/2 B [ [ [ [ [ [
  72. Pixels Bytes Browser Cache 128x128 1.39KB 3.6 KB 128x128 3.34KB

    5.6 KB 128x128 6.55 KB 9.6 KB 128x128 5.42 KB 9.6 KB
  73. <video controls autoplay muted loop> <source src="video.mp4" type="video/mp4"> <img src="fallback.jpg"

    alt="Video Screenshot" onclick=""> </video> ffmpeg -i video.gif -c:v libx264 -an -movflags faststart 
 -pix_fmt yuv420p -s 544x292 video.mp4 GIF = 4,3 MB WebP = 3,3 MB MP4 = 143 kB