Share
Embed
Headers for Hackers
The web is growing up and getting faster and more secure. Making that the default is hard to achieve when you have to be backwards compatible, and some of the stuff we built 10 years ago is now a serious security liability. The answer: headers. Lots of headers.
Andrew Betts
March 09, 2018
Transcript
97 slides
-
1 HTTP headers: serverless programming for the network Uncovering the
Web’s superpowers Andrew Betts, Fastly -
2 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com
HTTP/1.1 200 OK Date: Tue, 27 Feb 2018 13:28:47 GMT Content-Type: text/html Content-Length: 34882 <html> ... -
3 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com
HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Access-Control-Allow-Origin: * Accept-Ranges: bytes Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=31536000; includeSubdomains; Content-Security-Policy: default-src 'self'; report-uri https://csp.example.com/ Expect-CT: max-age=86400, enforce, report-uri="https://ect.example.com/" Alt-Svc: h2="new.example.com:443"; ma=600 Link: /script/bundle.rev-983c15.js>;rel=preload;as=script;charset=UTF-8; Accept-CH: DPR, Width, Viewport-Width Feature-Policy: vibrate 'none'; geolocation 'none'; unsized-media ‘none’ <html> ... -
8 • Alexa top 1,000,000 websites • Around 500,000 pages
analysed • Over 50 million requests per run • Captures full request and response data, timing metrics etc. • Runs using WebPageTest • Makes raw result data available in BigQuery -
15 (I'm going to mess up saying this on stage
aren't I) Platform for Privacy Preferences Project? • Intended as a declaration of privacy policy • Too hard for users to understand/use • Only ever implemented by Internet Explorer, to gate access to third party cookies in IFRAMEs. – ... but not validated • Commonly set to “this is not a P3P policy” which satisfies the check -
20 “Note: if a response includes a Cache-Control field with
the max-age directive, a recipient MUST ignore the Expires field.” -
22 Domains sending Expires 78% Domains sending Expires and Cache-Control
with max-age: 64% https://bigquery.cloud.google.com/savedquery/598614557294:98e14323d29740678fe1b3012c9186db -
23 Expires: Thu, 01 Dec 1994 16:00:00 GMT Cache-Control: private,
no-store, no-cache, no-transform, must-revalidate, max-age=0, post-check=0, pre-check=0 Pragma: no-cache -
26 Headers being emitted by > 5000 domains in HTTP
Archive: Meaningless to the browser https://bigquery.cloud.google.com/savedquery/598614557294:2463981d0f444b6ba6c1a8c376079b90 x-cache x-aspnet-version x-varnish x-request-id x-cache-hits x-cacheable x-aspnetmvc-version x-runtime x-generator x-drupal-cache host referer x-served-by x-proxy-cache server x-type x-cache-group x-cache-status x-accel-version -
27 Removing the cruft if (!req.http.Reveal-Debug && !req.http.Cookie:RevealDebug) { unset
resp.http.Server; unset resp.http.X-Powered-By; unset resp.http.X-Cache; // ... etc } app.disable('x-powered-by'); ExpressJS (Node): VCL (Varnish / Fastly): -
29 X-Frame-Options: SAMEORIGIN Stop anyone from framing your site: Content-Security-Policy:
frame-ancestors 'self' But... equivalent to... https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors -
34 Via in summary The bad parts • It prevents
forwarding loops – but might cause upstream servers to disable some HTTP features – and you can strip it in CDN config if you want to disable loop detection • It's both a request and a response header – but the response part is kind of useless • It provides for protocol negotiation – but only for H0 vs H1. HTTP2 negotiation happens a different way • It uses the name of the proxy engine – but that can be a bit misleading, eg Fastly sets 'varnish', and Heroku 'vegur', both of which are open-source products -
40 X-Cache: HIT Less of this sort of nonsense Cache:
HIT_FRESH More of this instead: -
44 HTTP/1.1 504 Gateway Timeout Proxy-Status: connection_timeout; proxy=SomeCDN; origin=abc; tries=3
Reasons for obscure proxy errors: DNS Timeout DNS Error Destination Not Found Destination Unavailable Destination IP Prohibited Destination IP Unroutable Connection Refused Connection Terminated Connection Timeout Connection Read Timeout Connection Write Timeout Connection Limit Reached HTTP Response Status HTTP Incomplete Response HTTP Protocol Error HTTP Response Header Block Too Large HTTP Response Header Too Large HTTP Response Body Too Large HTTP Response Transfer-Coding Error HTTP Response Content-Coding Error HTTP Response Timeout TLS Handshake Error TLS Untrusted Peer Certificate TLS Expired Peer Certificate TLS Unexpected Peer Certificate TLS Missing Proxy Certificate TLS Rejected Proxy Certificate TLS Error HTTP Request Error HTTP Request Denied HTTP Upgrade Failed Proxy Internal Response Proxy Internal Error -
47 Content-Security-Policy: default-src 'self'; img-src *; media-src media1.com media2.com; script-src
userscripts.example.com Simple CSP example: By default, only allow access to the same origin as the page. But allow images to come from anywhere. Media and scripts from a whitelist of specific origins. -
48 Content-Security-Policy: default-src 'self'; font-src 'self' data: http://*.gstatic.com https://*.gstatic.com https://*.symantec.com
https://*.criteo.com http://*.resultspage.com https://*.resultspage.com; child-src 'self' https://www.google.com/ads/ https://*.listrak.com https://www.google.pl/ads/ https://www.google.ua/ads/ http://*.google.com.ua/ads/ https://www.youtube.com/ https://www.googleadservices.com/ https://googleads.g.doubleclick.net/pagead/ https://*.symantec.com https://*.criteo.com http://*.rfksrv.com https://*.rfksrv.com http://*.resultspage.com https://*.resultspage.com; object-src 'self' http://*.verisign.com http://*.iesnare.com https://*.verisign.com https://*.iesnare.com https://*.symantec.com https://*.zmags.com; img-src 'self' data: https://ssl.emailcli.com *.sli-spark.com https://*.veinteractive.com http://*.bazaarvoice.com/ http://blog.natchezss.com/ https://blog.natchezss.com/ http://*.avmws.com http://*.websecurity.norton.com http://*.google.com.ua http://*.natchezss.com https://www.pepperjamnetwork.com/ http://*.gstatic.com http://*.ywxi.net https://www.google.com/ads/ http://s3.amazonaws.com/product.reflektion.com/ http://*.reflektion.com/ https://stats.g.doubleclick.net/ https://d26opx5dl8t69i.cloudfront.net/ http://*.google-analytics.com/ https://seal.networksolutions.com/ http://*.listrakbi.com/ http://*.chartbeat.net/ http://*.scanalert.com/ http://*.avantlink.com https://*.bazaarvoice.com/ https://*.listrakbi.com/ https://*.chartbeat.net/ https://*.google-analytics.com/ https://*.reflektion.com/ https://*.bazaarvoice.com/ https://*.google.com.ua https://*.listrakbi.com/ https://*.chartbeat.net/ https://*.scanalert.com/ https://*.avantlink.com https://*.amazonaws.com/ https://*.scanalert.com/ https://*.norton.com https://*.ywxi.net/ https://*.cloudfront.net/ https://*.zmags.com/ https://*.adnxs.com http://*.adnxs.com https://*.symantec.com https://*.r1cdn.com https://ad.doubleclick.net https://go.flx1.com https://*.g.doubleclick.net https://*.optimizely.com https://*.yahoo.com https://*.sitescout.com https://*.1rx.io https://*.tubemogul.com https://*.simpli.fi https://*.ipredictive.com https://*.wtp101.com https://*.pubmatic.com https://*.media.net https://*.demdex.net https://*.smartclip.net https://*.bit.ly https://*.criteo.com http://*.rfksrv.com https://*.rfksrv.com http://*.resultspage.com https://*.resultspage.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' *.sli-r.com *.sli-spark.com *.resultspage.com *.resultsstage.com tagmanager.google.com *.googletagmanager.com https://*.veinteractive.com http://*.avmws.com http://*.optimizely.com http://*.iesnare.com https://*.listrak.com https://www.googleadservices.com/ http://*.verisign.com http://*.googleapis.com http://*.nr-data.net http://*.newrelic.com http://*.ywxi.net http://*.bazaarvoice.com/ http://*.cloudfront.net/ http://*.listrakbi.com/ https://seal.networksolutions.com/ http://*.google-analytics.com/ https://ping.chartbeat.net/ http://*.reflektion.com/ http://*.chartbeat.com/ https://*.listrak.com/ https://product.reflektion.com/ https://display.ugc.bazaarvoice.com/ https://www.google-analytics.com/ https://cdn.listrakbi.com/ https://*.verisign.com https://*.googleapis.com https://*.nr-data.net https://*.newrelic.com https://*.ywxi.net https://*.bazaarvoice.com/ https://*.cloudfront.net/ https://*.listrakbi.com/ https://*.google-analytics.com/ https://*.reflektion.com/ https://*.chartbeat.com/ https://*.optimizely.com https://*.avmws.com/ http://*.zmags.com https://*.zmags.com https://*.dpmsrv.com http://*.dpmsrv.com https://*.adnxs.com/ http://*.adnxs.com/ https://*.doubleclick.net http://*.doubleclick.net https://*.google.com https://*.gstatic.com/ https://*.symantec.com https://*.statsstory.com https://*.hotjar.com https://*.cloudfront.net https://*.jsdelivr.net https://c.vepxl1.net https://c.flx1.com https://go.flx1.com https://*.youtube.com https://s3.amazonaws.com https://*.ytimg.com http://*.criteo.com https://*.criteo.com http://*.criteo.net https://*.criteo.net http://*.resultspage.com https://*.resultspage.com https://*.sli-r.com; style-src 'self' 'unsafe-inline' tagmanager.google.com http://*.amazonaws.com/ http://*.reflektion.com/ http://*.googleapis.com http://*.bazaarvoice.com/ https://*.bazaarvoice.com/ https://*.amazonaws.com/ https://*.listrakbi.com http://*.listrakbi.com https://*.symantec.com https://*.cloudfront.net https://*.criteo.com http://*.rfksrv.com https://*.rfksrv.com http://*.resultspage.com https://*.resultspage.com; frame-src 'self' http://*.bazaarvoice.com http://*.listrak.com http://*.youtube.com/ http://*.zmags.com/ woobox.com https://*.bazaarvoice.com https://*.listrak.com https://*.youtube.com/ https://*.zmags.com/ https://*.listrakbi.com https://*.googleadservices.com http://*.googleadservices.com https://*.doubleclick.net http://*.doubleclick.net https://*.google.com https://*.google.ad https://*.google.ae https://*.google.com.af https://*.google.com.ag https://*.google.com.ai https://*.google.al https://*.google.am https://*.google.co.ao https://*.google.com.ar https://*.google.as https://*.google.at https://*.google.com.au https://*.google.az https://*.google.ba https://*.google.com.bd https://*.google.be https://*.google.bf https://*.google.bg https://*.google.com.bh https://*.google.bi https://*.google.bj https://*.google.com.bn https://*.google.com.bo https://*.google.com.br https://*.google.bs https://*.google.bt https://*.google.co.bw https://*.google.by https://*.google.com.bz https://*.google.ca https://*.google.cd https://*.google.cf https://*.google.cg https://*.google.ch https://*.google.ci https://*.google.co.ck https://*.google.cl https://*.google.cm https://*.google.cn https://*.google.com.co https://*.google.co.cr https://*.google.com.cu https://*.google.cv https://*.google.com.cy https://*.google.cz https://*.google.de https://*.google.dj https://*.google.dk https://*.google.dm https://*.google.com.do https://*.google.dz https://*.google.com.ec https://*.google.ee https://*.google.com.eg https://*.google.es https://*.google.com.et https://*.google.fi https://*.google.com.fj https://*.google.fm https://*.google.fr https://*.google.ga https://*.google.ge https://*.google.gg https://*.google.com.gh https://*.google.com.gi https://*.google.gl https://*.google.gm https://*.google.gp https://*.google.gr https://*.google.com.gt https://*.google.gy https://*.google.com.hk https://*.google.hn https://*.google.hr https://*.google.ht https://*.google.hu https://*.google.co.id https://*.google.ie https://*.google.co.il https://*.google.im https://*.google.co.in https://*.google.iq https://*.google.is https://*.google.it https://*.google.je https://*.google.com.jm https://*.google.jo https://*.google.co.jp https://*.google.co.ke https://*.google.com.kh https://*.google.ki https://*.google.kg https://*.google.co.kr https://*.google.com.kw https://*.google.kz https://*.google.la https://*.google.com.lb https://*.google.li https://*.google.lk https://*.google.co.ls https://*.google.lt https://*.google.lu https://*.google.lv https://*.google.com.ly https://*.google.co.ma https://*.google.md https://*.google.me https://*.google.mg https://*.google.mk https://*.google.ml https://*.google.com.mm https://*.google.mn https://*.google.ms https://*.google.com.mt https://*.google.mu https://*.google.mv https://*.google.mw https://*.google.com.mx https://*.google.com.my https://*.google.co.mz https://*.google.com.na https://*.google.com.nf https://*.google.com.ng https://*.google.com.ni https://*.google.ne https://*.google.nl https://*.google.no https://*.google.com.np https://*.google.nr https://*.google.nu https://*.google.co.nz https://*.google.com.om https://*.google.com.pa https://*.google.com.pe https://*.google.com.pg https://*.google.com.ph https://*.google.com.pk https://*.google.pl https://*.google.pn https://*.google.com.pr https://*.google.ps https://*.google.pt https://*.google.com.py https://*.google.com.qa https://*.google.ro https://*.google.ru https://*.google.rw https://*.google.com.sa https://*.google.com.sb https://*.google.sc https://*.google.se https://*.google.vu https://*.google.ws https://*.google.rs https://*.google.co.za https://*.google.co.zm https://*.google.co.zw https://*.google.cat https://*.symantec.com https://*.mcafeesecure.com https://*.veinteractive.com https://*.hotjar.com https://*.youtube.com https://*.hotjar.com https://*.criteo.com *.dotomi.com http://*.rfksrv.com https://*.rfksrv.com; connect-src 'self' https://*.statsstory.com https://*.hotjar.com wss://*.hotjar.com https://*.veinteractive.com https://*.servicebus.windows.net https://*.optimizely.com https://*.youtube.com https://*.criteo.com; 9643 bytes! unsafe-inline unsafe-eval *.cloudfront.net tagmanager.google.com *.amazonaws.com -
50 Strict-Transport-Security Always connect over TLS, even for first request
https://tools.ietf.org/html/rfc6797 -
52 Referrer-Policy What to send in referrals from this page
https://w3c.github.io/webappsec-referrer-policy/ -
54 Referrer policy options Policy Referrer (same origin) Referrer (to
foreign origin) no-referrer omitted omitted no-referrer-when-downgrade https://example.com/page.html?query https://example.com/page.html?query origin https://example.com https://example.com origin-when-cross-origin https://example.com/page.html?query https://example.com same-origin https://example.com/page.html?query omitted Or just take my word for it and use 'origin-when-cross-origin' -
56 Access-Control-Allow-Origin: requesting-origin.example.com Access-Control-Allow-Credentials: true Access-Control-Max-Age: 86400 Access-Control-Expose-Headers: X-Debug Access-Control-Allow-Headers:
DNT, User-Agent Allow authenticated cross-origin requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors -
59 Accept-CH Requests client hints data be sent in future
http://httpwg.org/http-extensions/client-hints.html -
60 Accept-CH: DPR, Width, Viewport-Width, Save-Data Accept-CH-Lifetime: 86400 Request that
client send specified CHs: Sec-CH-DPR: 2.0 Sec-CH-Width: 320 Sec-CH-Viewport-Width: 320 Sec-CH-Save-Data: 1 Subsequent requests: -
62 Proposed new client hints! Sec-CH-UA: "Google Chrome 74" Sec-CH-UA-Platform:
"macOS 12" Sec-CH-UA-Arch: "ARM64" Sec-CH-UA-Engine: "Blink" Sec-CH-Lang: "en-US", "en", "de" -
63 Chrome has already started sending these sec-ch-ua is sent
by default: not subject to the Accept-CH opt-in -
64 Link: rel=preload Declare a resource that’s important early on
https://w3c.github.io/preload/#x2.link-type-preload -
65 Link: <some-font-face.woff2>; rel="preload"; as="font"; crossorigin Link: <main-styles.css>; rel="preload"; as="style"
Get your fonts and styles down sharpish: -
68 The status code problem DNS Lookup TLS TTFB Status
code + LINK headers received Database Auth Templating API queries -
70 HTTP/1.1 103 Early Hints Link: <some-font-face.woff2>; rel="preload"; as="font"; crossorigin
Link: <main-styles.css>; rel="preload"; as="style" HTTP/1.1 200 OK Date: Fri, 26 May 2017 10:02:11 GMT Content-Length: 1234 Content-Type: text/html; charset=utf-8 Link: <some-font-face.woff2>; rel="preload"; as="font"; crossorigin Link: <main-styles.css>; rel="preload"; as="style" <!doctype html> Get your fonts and styles down even sharper: -
71 Preloaded resources start preloading earlier DNS Lookup TLS Early
hints + Link headers received Real status received -
73 Server-Timing: miss, db;dur=53, app;dur=47.2 Server-Timing: customView, dc;desc=atl Server-Timing: cache;desc="Cache
Read";dur=23.2 Tell the browser what happened on the server: -
74 Server-timing metrics appear in developer tools But.. no sequencing,
and the only unit for numeric values is milliseconds -
75 JavaScript API In Chrome, Server-Timing values are parsed and
exposed in JavaScript -
77 Server-Timing: customerID;desc=1234567, edition;desc="EMEA" Server-Timing: paywallArticlesRemaining;desc=8 Tell the browser…. anything?:
performance.getEntriesByType('navigation')[0] .serverTiming .reduce((soFar, entry) => ({ ...soFar, [entry.name]: entry.description }), {}) ; {customerID: "1234567", edition: "EMEA", paywallArticlesRemaining: "8"} So we could, like, embed post-cache metadata into headers? -
78 But server-timing might be unknown in headers DNS Lookup
TLS TTFB Headers sent Only now we know: Total bytes sent Last byte timestamp Client data rate This is a bit like the the Early-Hints situation! -
82 HTTP/1.1 200 OK Date: Tue, 27 Feb 2018 13:28:47
GMT Content-Type: text/html Content-Length: 12 Server-Timing: miss, db;dur=53, app;dur=47.2 Trailers: Server-Timing Hello world! Server-Timing: sentduration;dur=234 Headers, but like, at the end? -
83 Triggering trailer support in Fastly Set the x-trailer-server-timing header,
Fastly converts to a trailer -
85 Feature-Policy: autoplay 'none'; camera 'self'; unsized-media some3rdparty.com Turn off
bad practices No-one can autoplay video on this page Only I can use the camera. Images only take size from their contents when loaded from some3rdparty.com -
86 What policies are available? document.featurePolicy.allowedFeatures().sort().join('\n') accelerometer, ambient-light-sensor, autoplay, camera,
ch-device-memory, ch-downlink, ch-dpr, ch-ect, ch-lang, ch-rtt, ch-ua, ch-ua-arch, ch-ua-model, ch-ua-platform, ch-viewport-width, ch-width, document-access, document-domain, document-write, downloads-without-user-activation, encrypted-media, execution-while-not-rendered, execution-while-out-of-viewport, focus-without-user-activation, font-display-late-swap, forms, fullscreen, geolocation, gyroscope, hid, idle-detection, layout-animations, lazyload, loading-frame-default-eager, magnetometer, microphone, midi, modals, orientation-lock, oversized-images, payment, picture-in-picture, pointer-lock, popups, presentation, scripts, serial, sync-script, sync-xhr, top-navigation, unoptimized-lossless-images, unoptimized-lossless-images-strict, unoptimized-lossy-images, unsized-media, usb, vertical-scroll, vr, wake-lock, xr -
87 Feature-Policy: accelerometer 'none'; ambient-light-sensor 'none'; animations 'none'; autoplay 'none';
camera 'none'; document-write 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; image-compression 'none'; legacy-image-formats 'none'; magnetometer 'none'; max-downscaling-image 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; speaker 'none'; sync-script 'none'; sync-xhr 'none'; unsized-media 'none'; usb 'none'; vertical-scroll 'none'; vr 'none'; Today's tightest feature-policy (in Chrome): -
90 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com
HTTP/1.1 200 OK Date: Tue, 27 Feb 2018 13:28:47 GMT Content-Type: text/html Content-Length: 34882 <html> ... -
91 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com
HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=31536000; includeSubdomains; Content-Security-Policy: default-src 'self'; report-uri https://csp.example.com/ Expect-CT: max-age=86400, enforce, report-uri="https://ect.example.com/" Alt-Svc: h2="new.example.com:443"; ma=600 Link: /script/bundle.rev-983c15.js>;rel=preload;as=script;charset=UTF-8; Accept-CH: DPR, Width, Viewport-Width Feature-Policy: vibrate 'none'; geolocation 'none'; unsized-media ‘none’ <html> ... -
92 { "headers": [ { "name": "Content-Security-Policy", "value": "script-src 'self'
https://cdn.example.com", "type": "fallback" }, { "name": "Referrer-Policy", "value": "origin-when-cross-origin", "type": "fallback" }, { "name": "Content-Security-Policy", "value": "object-src 'none'; frame-ancestors 'none'", "type": "baseline" }, { "name": "Strict-Transport-Security", "value": "max-age=10886400; includeSubDomains; preload", "type": "baseline" }, { "name": "X-Content-Type-Options", "value": "nosniff", "type": "baseline" } ], "cors-preflight": { "origins": "*" } } /.well-known/origin-policy/policy-1 -
93 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com
HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=31536000; includeSubdomains; Content-Security-Policy: default-src 'self'; report-uri https://csp.example.com/ Expect-CT: max-age=86400, enforce, report-uri="https://ect.example.com/" Alt-Svc: h2="new.example.com:443"; ma=600 Link: /script/bundle.rev-983c15.js>;rel=preload;as=script;charset=UTF-8; Accept-CH: DPR, Width, Viewport-Width Feature-Policy: vibrate 'none'; geolocation 'none'; unsized-media ‘none’ <html> ... -
94 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com
HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT Sec-Origin-Policy: "policy-1" Vary: sec-origin-policy <html> ... -
95 The whole talk in one slide Use these: •
Clear-Site-Data • Content-Security-Policy • Strict-Transport-Security • Referrer-Policy • Access-Control-* (CORS) • Accept-CH • Link (rel=preload) • Server-Timing • P3P • Expires • X-Cache • X-Frame-Options • Via Don't use these: Follow these: • Feature-Policy • Origin-Policy • CDN-Loop (if you're a CDN) • Cache • Proxy-Status