HTTPS is Hard

Here, Yell’s Head of Web Engineering, Steve Workman, looks back over Yell.com‘s nine-month transition to HTTPS, to raise awareness of the issues with the move in the industry and to make the adoption process easier for other engineering teams.

First presented at Front End London on 31st March 2016, updated for Breaking Borders on June 15th 2016 and updated again for Full Stack Toronto, October 18th 2016

Transcript

1. HTTPS is Hard
   Steve Workman
   

   View Slide

2. “We’re a business directory, why
   do we need to be secure?”
   Me, to Dan Applequist, January 2015
   @steveworkman HTTPS is Hard #fstoco
   

   View Slide

3. “Think about what queries your users put through
   that every day, legal counsel, family planning clinics,
   as well as the regular plumbers and hairdressers.
   They search for it locally, and that is all personally
   identifiable.
   If I were a hacker intercepting this traffic I could work
   out some pretty interesting stuff about you”
   Dan Applequist, correcting me, January 2015
   @steveworkman HTTPS is Hard #fstoco
   

   View Slide

4. “Google is pushing hard on this, they made it a
   ranking factor to encourage the big guys to change.
   If you’re selling this to your boss, that’s what you’ll
   major on”
   Dan Applequist, selling it, January 2015
   @steveworkman HTTPS is Hard #fstoco
   

   View Slide

5. @steveworkman HTTPS is Hard #fstoco
   

   View Slide

6. @steveworkman HTTPS is Hard #fstoco
   Engineering
   Security Operations
   Product
   Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
   

   View Slide

7. @steveworkman HTTPS is Hard #fstoco
   Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
   

   View Slide

8. @steveworkman HTTPS is Hard #fstoco
   Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
   

   View Slide

9. @steveworkman HTTPS is Hard #fstoco
   Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
   

   View Slide

10. @steveworkman HTTPS is Hard #fstoco
    http ://www.yell.com
    S
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

11. See what breaks
    q Some internal URLs, including the canonical URLs
    q All adverts
    q Adobe Analytics
    q The entire reviews section
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

12. Fixing things
    ü Some internal URLs, including the canonical URLs
    q All adverts
    q Adobe Analytics
    q The entire reviews section
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

13. Securing Adverts:
    • AOL/Yahoo’s Advertising network
    • Can easily serve their scripts over HTTPS
    • Adverts will then be served over HTTPS
    • Or at least they should be
    • You can be your own worst enemy here
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

14. IAB are changing their ways
    • 80% of the industry supports HTTPS
    • In October 2015, they admitted they messed up
    • http://www.iab.com/news/lean/
    • Light
    • Encrypted
    • Ad Choice Supported
    • Non-Invasive
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

15. @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Advertising
    Adtech
    Ad Sales
    team
    Engineering
    Security Operations
    Product
    

    View Slide

16. Fixing things
    ü Some internal URLs, including the canonical URLs
    ü All adverts
    q Adobe Analytics
    q The entire reviews section
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

17. Third Party 2: Adobe Analytics
    • Checked our implementation – no joy
    • Contact Adobe
    • Enabled first-party domains
    • Supply certificates
    • Very cautiously updated to the latest version
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

18. @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Advertising
    Adtech
    Ad Sales
    team
    Engineering
    Security Operations
    Product
    Analytics
    Team
    Adobe
    

    View Slide

19. Fixing things
    ü Some internal URLs, including the canonical URLs
    ü All adverts
    ü Adobe Analytics
    q The entire reviews section
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

20. “What’s in that shadowy place over
    there?”
    @steveworkman HTTPS is Hard #fstoco
    That’s the reviews system,
    you must never go there
    

    View Slide

21. Fixing things
    ü Some internal URLs, including the canonical URLs
    ü All adverts
    ü Adobe Analytics
    ü The entire reviews section
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

22. Acquire Certificates
    @steveworkman HTTPS is Hard #fstoco
    • Self-signed
    • Domain Validated
    • Extended Validation
    

    View Slide

23. Why EV Certificates?
    • It’s a mark of trust in the organisation
    • It’s not much more expensive than a regular certificate
    • It’s the only type of certificate that turns the padlock green in
    Edge
    • Important for the perception of security
    @steveworkman HTTPS is Hard #fstoco
    

    View Slide

24. EV certification isn’t hard, it takes time
    • More levels of scrutiny and manual steps takes the time
    • Had to update our domain records due to corporate name
    changes
    • Took a total of 4 weeks
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

25. @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Advertising
    Adtech
    Ad Sales
    team
    Engineering
    Security Operations
    Product
    Analytics
    Team
    Adobe
    Legal
    Companies
    House
    

    View Slide

26. Other third parties
    • Anti-scraping tool
    • Costs money to do with EV cert for a private IP
    • Video hosting CDN
    • Costs money – host didn’t support SNI
    • Cross-region agreement means this is still in progress
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

27. The Business Case
    • Capital Expenditure (spending money) isn’t easy for many
    developers
    • Lots will have never written a business case before
    • Depending on your organisation, this may not be trivial and
    can take time and effort to push it through
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

28. Pre-live performance concerns
    • Is TLS Fast Yet?
    • Yes, it is: www.Istlsfastyet.com
    • Monitor our performance with RUM tools
    • Terminate the connection at load balancer (closer to user)
    • Ensure it is up to date
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

29. @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Advertising
    Adtech
    Ad Sales
    team
    Engineering
    Security Operations
    Product
    Analytics
    Team
    Adobe
    Legal
    Companies
    House
    Anti-scrape
    CDN
    CDO
    CEO
    

    View Slide

30. The Big Day
    @steveworkman HTTPS is Hard #fstoco
    • Sitemaps (~10M links)
    • Robots.txt
    • Google Search Console
    • 301 redirects for HTTP
    traffic at the network
    edge (the flip)
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

31. The aftermath
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

32. Java silently stopped sending requests
    • Java only has some standard Root CA certificates by default
    • Without these, requests over HTTPS will fail silently
    • Upgrading Java wholesale is full of risk, simpler to install
    missing CAs
    • Pro tip: Always have an internal non-HTTPS route
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

33. What does HTTPS do to your
    Google search ranking?
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

34. @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    • HTTPS is 1 factor out of 200+ and is a “tie-break” factor
    • It correlates +0.04 to ranking - not strong
    • https://moz.com/search-ranking-factors/correlations
    

    View Slide

35. Search ranking can be affected
    • Wired chose to use 302 redirects initially, causing drops in
    search ranking
    • Once they switched to 301 redirects, ranking losses stopped
    @steveworkman HTTPS is Hard #fstoco
    Source: https://www.wired.com/2016/09/wired-completely-encrypted/
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

36. Google re-indexing took over 6 months
    @steveworkman HTTPS is Hard #fstoco
    21/06/2015 21/07/2015 21/08/2015 21/09/2015 21/10/2015 21/11/2015 21/12/2015 21/01/2016
    % of pages indexed on Google
    %HTTP
    % HTTPS
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

37. TLS Performance
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Desktop devices Mobile devices
    

    View Slide

38. HTTPS is Fast, but it is
    not Free
    Un-tuned HTTPS will add 100-200ms to your first render time, and more
    than that at the extremes of connectivity
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

39. What’s wrong here?
    • Anti-scrape server isn’t as optimised as it could be – Window
    Scaling, OCSP stapling, TLS False Start all off
    • Together they add 2 round-trips to each handshake
    • So, the impact should theoretically be 30-60ms, not 100-
    200ms
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Anti-scrape Origin
    

    View Slide

40. “I’ve stopped receiving traffic
    from your site”
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

41. HTTP Referrer
    • 99% of our customer’s websites are served over HTTP, and of
    that 1%, a quarter of those are Facebook pages.
    @steveworkman HTTPS is Hard #fstoco
    From / To HTTP HTTPS
    HTTP Pass Pass
    HTTPS Do not pass Pass
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

42. Referrer Policy
    
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

43. Content Security Policy Level 2
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

44. We chose to educate our customers instead
    @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    

    View Slide

45. @steveworkman HTTPS is Hard #fstoco
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    Advertising
    Adtech
    Ad Sales
    team
    Engineering
    Security Operations
    Product
    Analytics
    Team
    Adobe
    Legal
    Companies
    House
    Anti-scrape
    CDN
    CDO
    CEO
    Sales
    Marketing
    Customer
    Services
    Telesales
    

    View Slide

46. HTTPS is not a technology
    problem, it is a people problem,
    and that problem is incentives
    @steveworkman HTTPS is Hard #fstoco
    

    View Slide

47. Good News Everyone!
    The internet has listened and is changing
    for the better
    @steveworkman HTTPS is Hard #fstoco
    

    View Slide

48. Problem
    Certificates aren’t free
    There’s a performance impact
    CDNs should offer TLS for free
    Solution
    HTTP/2
    Most do for DV certificates
    @steveworkman HTTPS is Hard #fstoco
    

    View Slide

49. The migration cost is too high
    Without HTTPS you can’t have
    @steveworkman HTTPS is Hard #fstoco
    Privileged Features
    Geolocation
    Webcam
    Microphone
    Notifications
    Device motion & orientation
    Progressive Web Apps
    Service Worker
    AMP
    

    View Slide

50. @steveworkman HTTPS is Hard #fstoco
    

    View Slide

51. What’s next for Yell?
    • Work with third-party providers to improve TCP performance
    • HTTP/2
    • HTTP Strict Transport Security (HSTS)
    • Create a Content Security Policy (CSP)
    • Ensure server Cookies set with httpsOnly flag
    • Complete the transition and update our CDN
    @steveworkman HTTPS is Hard #fstoco
    

    View Slide

52. @steveworkman HTTPS is Hard #fstoco
    https://observatory.mozilla.org
    

    View Slide

53. Thank you
    Twitter: @steveworkman
    Slides: https://speakerdeck.com/steveworkman/https-is-hard
    Epic Blog post: https://blog.yell.com/2016/03/https-is-hard/
    @steveworkman HTTPS is Hard #fstoco
    

    View Slide