HTTPS is Hard

HTTPS is Hard

Here, Yell’s Head of Web Engineering, Steve Workman, looks back over Yell.com‘s nine-month transition to HTTPS, to raise awareness of the issues with the move in the industry and to make the adoption process easier for other engineering teams.

First presented at Front End London on 31st March 2016, updated for Breaking Borders on June 15th 2016 and updated again for Full Stack Toronto, October 18th 2016

2fdb5b62030270813e22f5e17d16f6b9?s=128

Steve Workman

March 31, 2016
Tweet

Transcript

  1. HTTPS is Hard Steve Workman

  2. “We’re a business directory, why do we need to be

    secure?” Me, to Dan Applequist, January 2015 @steveworkman HTTPS is Hard #fstoco
  3. “Think about what queries your users put through that every

    day, legal counsel, family planning clinics, as well as the regular plumbers and hairdressers. They search for it locally, and that is all personally identifiable. If I were a hacker intercepting this traffic I could work out some pretty interesting stuff about you” Dan Applequist, correcting me, January 2015 @steveworkman HTTPS is Hard #fstoco
  4. “Google is pushing hard on this, they made it a

    ranking factor to encourage the big guys to change. If you’re selling this to your boss, that’s what you’ll major on” Dan Applequist, selling it, January 2015 @steveworkman HTTPS is Hard #fstoco
  5. @steveworkman HTTPS is Hard #fstoco

  6. @steveworkman HTTPS is Hard #fstoco Engineering Security Operations Product Jan

    Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  7. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec
  8. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec
  9. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec
  10. @steveworkman HTTPS is Hard #fstoco http ://www.yell.com S Jan Feb

    Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  11. See what breaks q Some internal URLs, including the canonical

    URLs q All adverts q Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  12. Fixing things ü Some internal URLs, including the canonical URLs

    q All adverts q Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  13. Securing Adverts: • AOL/Yahoo’s Advertising network • Can easily serve

    their scripts over HTTPS • Adverts will then be served over HTTPS • Or at least they should be • You can be your own worst enemy here @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  14. IAB are changing their ways • 80% of the industry

    supports HTTPS • In October 2015, they admitted they messed up • http://www.iab.com/news/lean/ • Light • Encrypted • Ad Choice Supported • Non-Invasive @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  15. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product
  16. Fixing things ü Some internal URLs, including the canonical URLs

    ü All adverts q Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  17. Third Party 2: Adobe Analytics • Checked our implementation –

    no joy • Contact Adobe • Enabled first-party domains • Supply certificates • Very cautiously updated to the latest version @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  18. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe
  19. Fixing things ü Some internal URLs, including the canonical URLs

    ü All adverts ü Adobe Analytics q The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  20. “What’s in that shadowy place over there?” @steveworkman HTTPS is

    Hard #fstoco That’s the reviews system, you must never go there
  21. Fixing things ü Some internal URLs, including the canonical URLs

    ü All adverts ü Adobe Analytics ü The entire reviews section @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  22. Acquire Certificates @steveworkman HTTPS is Hard #fstoco • Self-signed •

    Domain Validated • Extended Validation
  23. Why EV Certificates? • It’s a mark of trust in

    the organisation • It’s not much more expensive than a regular certificate • It’s the only type of certificate that turns the padlock green in Edge • Important for the perception of security @steveworkman HTTPS is Hard #fstoco
  24. EV certification isn’t hard, it takes time • More levels

    of scrutiny and manual steps takes the time • Had to update our domain records due to corporate name changes • Took a total of 4 weeks @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  25. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe Legal Companies House
  26. Other third parties • Anti-scraping tool • Costs money to

    do with EV cert for a private IP • Video hosting CDN • Costs money – host didn’t support SNI • Cross-region agreement means this is still in progress @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  27. The Business Case • Capital Expenditure (spending money) isn’t easy

    for many developers • Lots will have never written a business case before • Depending on your organisation, this may not be trivial and can take time and effort to push it through @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  28. Pre-live performance concerns • Is TLS Fast Yet? • Yes,

    it is: www.Istlsfastyet.com • Monitor our performance with RUM tools • Terminate the connection at load balancer (closer to user) • Ensure it is up to date @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  29. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe Legal Companies House Anti-scrape CDN CDO CEO
  30. The Big Day @steveworkman HTTPS is Hard #fstoco • Sitemaps

    (~10M links) • Robots.txt • Google Search Console • 301 redirects for HTTP traffic at the network edge (the flip) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  31. The aftermath @steveworkman HTTPS is Hard #fstoco Jan Feb Mar

    Apr May Jun Jul Aug Sep Oct Nov Dec
  32. Java silently stopped sending requests • Java only has some

    standard Root CA certificates by default • Without these, requests over HTTPS will fail silently • Upgrading Java wholesale is full of risk, simpler to install missing CAs • Pro tip: Always have an internal non-HTTPS route @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  33. What does HTTPS do to your Google search ranking? @steveworkman

    HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  34. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec • HTTPS is 1 factor out of 200+ and is a “tie-break” factor • It correlates +0.04 to ranking - not strong • https://moz.com/search-ranking-factors/correlations
  35. Search ranking can be affected • Wired chose to use

    302 redirects initially, causing drops in search ranking • Once they switched to 301 redirects, ranking losses stopped @steveworkman HTTPS is Hard #fstoco Source: https://www.wired.com/2016/09/wired-completely-encrypted/ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  36. Google re-indexing took over 6 months @steveworkman HTTPS is Hard

    #fstoco 21/06/2015 21/07/2015 21/08/2015 21/09/2015 21/10/2015 21/11/2015 21/12/2015 21/01/2016 % of pages indexed on Google %HTTP % HTTPS Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  37. TLS Performance @steveworkman HTTPS is Hard #fstoco Jan Feb Mar

    Apr May Jun Jul Aug Sep Oct Nov Dec Desktop devices Mobile devices
  38. HTTPS is Fast, but it is not Free Un-tuned HTTPS

    will add 100-200ms to your first render time, and more than that at the extremes of connectivity @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  39. What’s wrong here? • Anti-scrape server isn’t as optimised as

    it could be – Window Scaling, OCSP stapling, TLS False Start all off • Together they add 2 round-trips to each handshake • So, the impact should theoretically be 30-60ms, not 100- 200ms @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Anti-scrape Origin
  40. “I’ve stopped receiving traffic from your site” @steveworkman HTTPS is

    Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  41. HTTP Referrer • 99% of our customer’s websites are served

    over HTTP, and of that 1%, a quarter of those are Facebook pages. @steveworkman HTTPS is Hard #fstoco From / To HTTP HTTPS HTTP Pass Pass HTTPS Do not pass Pass Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  42. Referrer Policy <meta rel=“referrer” content=“unsafe”> @steveworkman HTTPS is Hard #fstoco

    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  43. Content Security Policy Level 2 @steveworkman HTTPS is Hard #fstoco

    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  44. We chose to educate our customers instead @steveworkman HTTPS is

    Hard #fstoco Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
  45. @steveworkman HTTPS is Hard #fstoco Jan Feb Mar Apr May

    Jun Jul Aug Sep Oct Nov Dec Advertising Adtech Ad Sales team Engineering Security Operations Product Analytics Team Adobe Legal Companies House Anti-scrape CDN CDO CEO Sales Marketing Customer Services Telesales
  46. HTTPS is not a technology problem, it is a people

    problem, and that problem is incentives @steveworkman HTTPS is Hard #fstoco
  47. Good News Everyone! The internet has listened and is changing

    for the better @steveworkman HTTPS is Hard #fstoco
  48. Problem Certificates aren’t free There’s a performance impact CDNs should

    offer TLS for free Solution HTTP/2 Most do for DV certificates @steveworkman HTTPS is Hard #fstoco
  49. The migration cost is too high Without HTTPS you can’t

    have @steveworkman HTTPS is Hard #fstoco Privileged Features Geolocation Webcam Microphone Notifications Device motion & orientation Progressive Web Apps Service Worker AMP
  50. @steveworkman HTTPS is Hard #fstoco

  51. What’s next for Yell? • Work with third-party providers to

    improve TCP performance • HTTP/2 • HTTP Strict Transport Security (HSTS) • Create a Content Security Policy (CSP) • Ensure server Cookies set with httpsOnly flag • Complete the transition and update our CDN @steveworkman HTTPS is Hard #fstoco
  52. @steveworkman HTTPS is Hard #fstoco https://observatory.mozilla.org

  53. Thank you Twitter: @steveworkman Slides: https://speakerdeck.com/steveworkman/https-is-hard Epic Blog post: https://blog.yell.com/2016/03/https-is-hard/

    @steveworkman HTTPS is Hard #fstoco