HTTPS is Hard

Here, Yell’s Head of Web Engineering, Steve Workman, looks back over Yell.com‘s nine-month transition to HTTPS, to raise awareness of the issues with the move in the industry and to make the adoption process easier for other engineering teams.

First presented at Front End London on 31st March 2016, updated for Breaking Borders on June 15th 2016 and updated again for Full Stack Toronto, October 18th 2016

Transcript

1. HTTPS is Hard Steve Workman
2. “We’re a business directory, why do we need to be secure?” Me, to Dan Applequist, January 2015 @steveworkman HTTPS is Hard #fstoco
3. “Think about what queries your users put through that every day, legal counsel, family planning clinics, as well as the regular plumbers and hairdressers. They search for it locally, and that is all personally identifiable. If I were a hacker intercepting this traffic I could work out some pretty interesting stuff about you” Dan Applequist, correcting me, January 2015 @steveworkman HTTPS is Hard #fstoco
4. “Google is pushing hard on this, they made it a ranking factor to encourage the big guys to change. If you’re selling this to your boss, that’s what you’ll major on” Dan Applequist, selling it, January 2015 @steveworkman HTTPS is Hard #fstoco
5. @steveworkman HTTPS is Hard #fstoco
6. @steveworkman HTTPS is Hard #fstoco Engineering Security Operations Product Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
7. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
8. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
9. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
10. @steveworkman HTTPS is Hard #fstoco http ://www.yell.com S Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
11. See what breaks q Some internal URLs, including the canonical URLs q All adverts q Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
12. Fixing things ü Some internal URLs, including the canonical URLs q All adverts q Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
13. Securing Adverts: • AOL/Yahoo’s Advertising network • Can easily serve their scripts over HTTPS • Adverts will then be served over HTTPS • Or at least they should be • You can be your own worst enemy here @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
14. IAB are changing their ways • 80% of the industry supports HTTPS • In October 2015, they admitted they messed up • http://www.iab.com/news/lean/ • Light • Encrypted • Ad Choice Supported • Non-Invasive @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
15. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product
16. Fixing things ü Some internal URLs, including the canonical URLs ü All adverts q Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
17. Third Party 2: Adobe Analytics • Checked our implementation – no joy • Contact Adobe • Enabled first-party domains • Supply certificates • Very cautiously updated to the latest version @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
18. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe
19. Fixing things ü Some internal URLs, including the canonical URLs ü All adverts ü Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
20. “What’s in that shadowy place over there?” @steveworkman HTTPS is Hard #fstoco That’s the reviews system, you must never go there
21. Fixing things ü Some internal URLs, including the canonical URLs ü All adverts ü Adobe Analytics ü The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
22. Acquire Certificates @steveworkman HTTPS is Hard #fstoco • Self-signed • Domain Validated • Extended Validation
23. Why EV Certificates? • It’s a mark of trust in the organisation • It’s not much more expensive than a regular certificate • It’s the only type of certificate that turns the padlock green in Edge • Important for the perception of security @steveworkman HTTPS is Hard #fstoco
24. EV certification isn’t hard, it takes time • More levels of scrutiny and manual steps takes the time • Had to update our domain records due to corporate name changes • Took a total of 4 weeks @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
25. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe Legal Companies House
26. Other third parties • Anti-scraping tool • Costs money to do with EV cert for a private IP • Video hosting CDN • Costs money – host didn’t support SNI • Cross-region agreement means this is still in progress @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
27. The Business Case • Capital Expenditure (spending money) isn’t easy for many developers • Lots will have never written a business case before • Depending on your organisation, this may not be trivial and can take time and effort to push it through @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
28. Pre-live performance concerns • Is TLS Fast Yet? • Yes, it is: www.Istlsfastyet.com • Monitor our performance with RUM tools • Terminate the connection at load balancer (closer to user) • Ensure it is up to date @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
29. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe Legal Companies House Anti-scrape CDN CDO CEO
30. The Big Day @steveworkman HTTPS is Hard #fstoco • Sitemaps (~10M links) • Robots.txt • Google Search Console • 301 redirects for HTTP traffic at the network edge (the flip) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
31. The aftermath @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
32. Java silently stopped sending requests • Java only has some standard Root CA certificates by default • Without these, requests over HTTPS will fail silently • Upgrading Java wholesale is full of risk, simpler to install missing CAs • Pro tip: Always have an internal non-HTTPS route @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
33. What does HTTPS do to your Google search ranking? @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
34. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec • HTTPS is 1 factor out of 200+ and is a “tie-break” factor • It correlates +0.04 to ranking - not strong • https://moz.com/search-ranking-factors/correlations
35. Search ranking can be affected • Wired chose to use 302 redirects initially, causing drops in search ranking • Once they switched to 301 redirects, ranking losses stopped @steveworkman HTTPS is Hard #fstoco Source: https://www.wired.com/2016/09/wired-completely-encrypted/ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
36. Google re-indexing took over 6 months @steveworkman HTTPS is Hard #fstoco 21/06/2015 21/07/2015 21/08/2015 21/09/2015 21/10/2015 21/11/2015 21/12/2015 21/01/2016 % of pages indexed on Google %HTTP % HTTPS Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
37. TLS Performance @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Desktop devices Mobile devices
38. HTTPS is Fast, but it is not Free Un-tuned HTTPS will add 100-200ms to your first render time, and more than that at the extremes of connectivity @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
39. What’s wrong here? • Anti-scrape server isn’t as optimised as it could be – Window Scaling, OCSP stapling, TLS False Start all off • Together they add 2 round-trips to each handshake • So, the impact should theoretically be 30-60ms, not 100- 200ms @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Anti-scrape Origin
40. “I’ve stopped receiving traffic from your site” @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
41. HTTP Referrer • 99% of our customer’s websites are served over HTTP, and of that 1%, a quarter of those are Facebook pages. @steveworkman HTTPS is Hard #fstoco From / To HTTP HTTPS HTTP Pass Pass HTTPS Do not pass Pass Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
42. Referrer Policy <meta rel=“referrer” content=“unsafe”> @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
43. Content Security Policy Level 2 @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
44. We chose to educate our customers instead @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
45. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe Legal Companies House Anti-scrape CDN CDO CEO Sales Marketing Customer Services Telesales
46. HTTPS is not a technology problem, it is a people problem, and that problem is incentives @steveworkman HTTPS is Hard #fstoco
47. Good News Everyone! The internet has listened and is changing for the better @steveworkman HTTPS is Hard #fstoco
48. Problem Certificates aren’t free There’s a performance impact CDNs should offer TLS for free Solution HTTP/2 Most do for DV certificates @steveworkman HTTPS is Hard #fstoco
49. The migration cost is too high Without HTTPS you can’t have @steveworkman HTTPS is Hard #fstoco Privileged Features Geolocation Webcam Microphone Notifications Device motion & orientation Progressive Web Apps Service Worker AMP
50. @steveworkman HTTPS is Hard #fstoco
51. What’s next for Yell? • Work with third-party providers to improve TCP performance • HTTP/2 • HTTP Strict Transport Security (HSTS) • Create a Content Security Policy (CSP) • Ensure server Cookies set with httpsOnly flag • Complete the transition and update our CDN @steveworkman HTTPS is Hard #fstoco
52. @steveworkman HTTPS is Hard #fstoco https://observatory.mozilla.org
53. Thank you Twitter: @steveworkman Slides: https://speakerdeck.com/steveworkman/https-is-hard Epic Blog post: https://blog.yell.com/2016/03/https-is-hard/ @steveworkman HTTPS is Hard #fstoco