Fastly in Cookpad

Talk at Fastly Yamagoya 2017 https://techplay.jp/event/633461

How Cookpad Global https://cookpad.com/uk uses Fastly CDN.

Transcript

1. Fastly in Cookpad
   Sorah Fukumori
   @ Fastly Yamagoya 2017
   

   View Slide

2. $ whoami
   Sorah Fukumori (׉׵כ
   
   https://sorah.jp/ | GitHub @sorah | Twitter @sora_h
   Site Reliability Engineer at Cookpad Global
   Rubyist, Ruby committer
   Operating AS59128
   Interests: Site Reliability, Networking, Distributed systems
   

   View Slide

3. Agenda
   • About us
   • Latency
   • Fastly usage in Cookpad
   • How we manage/use Fastly
   

   View Slide

4. Cookpad?
   • The world’s largest recipe sharing platform
   • 69 countries, 22 regions
   • Rapidly growing in the world
   • Code is different, unshared between Global and JP
   

   View Slide

5. https://cookpad.com/uk
   

   View Slide

6. https://cookpad.com/
   

   View Slide

7. Cookpad?
   • The world’s largest recipe sharing platform
   • 69 countries, 22 regions
   • Rapidly growing in the world
   • Code is different, unshared between Global and JP
   • But the domain name is shared!
   

   View Slide

8. Infrastructure
   • Everything on AWS
   • JP uses Tokyo (ap-northest-1),

   Global uses US East (N.Virginia, us-east-1)
   • 2 CDNs (Fastly, CloudFront)
   

   View Slide

9. LATENCY
   

   View Slide

10. Fighting with Latency
    • Assume everyone know latency affects user experience
    heavily
    • TCP/TLS handshake…
    

    View Slide

11. Fighting with Latency
    • Q. Apps live in multiple regions, but share the single,
    same domain
    • … What?
    

    View Slide

12. • We chose Route53 Latency Based Routing at first
    • DNS returns IP addresses of the closer region from a
    resolver
    • If a requested service lives in another region, reverse-
    proxy to the alternate region
    Fighting with Latency
    

    View Slide

13. ! "
    

    View Slide

14. location ~ ^/(ae|ar|bh|bo|br|cl|co|cr|
    cu|de|dj|do|dz|ec|eeuu|eg|es|fr|gt|hn|
    hu|id|in|iq|ir|it|jo|km|kr|kw|lb|ly|ma|
    mr|mx|ni|om|pa|pe|ph|pri|ps|py|qa|sa|sd|
    so|sv|sy|th|tn|tw|uk|us|uy|ve|vn|ye)(/|
    $) {
    proxy_pass http://cookpad_use1;
    }
    location / {
    proxy_pass https://cookpad_apne1;
    }
    

    View Slide

15. • This is the minimum
    • 2 regions are not enough
    • We have to serve in Middle East, Europe, UK …
    • They’re too far away from both US east and Japan
    Fighting with Latency
    

    View Slide

16. • But who want to manage many proxy servers in the all
    available AWS regions?
    • tips: AWS doesn’t cover the
    • CDN?
    Fighting with Latency
    

    View Slide

17. Past CDN usage in Cookpad
    • User provided Images
    • Dynamic resizing provided by an internal service
    • Cache resized images
    • Static assets (CSS, JavaScript, Images)
    • No Dynamic Content
    

    View Slide

18. No Dynamic Content
    • No dynamic content
    • (Contents served via CDN are guaranteed static by
    their URL)
    • As an old-school CDN user:
    • Slow deployments, slow purging…
    • It’s too risky when we fail at something
    

    View Slide

19. Meet with Fastly
    • “less-risky?”
    • Fast purge!
    • Fast deployment!
    • VCL!
    • Decided to start using as a replacement of existing
    CDN, to take a try
    

    View Slide

20. The first use case
    • Images and Assets for Global Platform
    

    View Slide

21. The first use case
    • Images and Assets for Global Platform
    • Achieved higher hit rate, better performance

    than our previous CDN provider
    • Shielding <3
    • Few and fast POPs
    

    View Slide

22. now, Fastly in Cookpad
    • Primarily for Cookpad Global
    • 13 services, and 7 in production
    

    View Slide

23. View Slide

24. Fastly in Cookpad
    • Global Platform
    • Static Assets
    • Images (user provided)
    • API
    

    View Slide

25. Fastly in Cookpad
    • And https://cookpad.com/
    

    View Slide

26. Fastly in Cookpad
    • And https://cookpad.com/ (except JP)
    

    View Slide

27. Fastly in Cookpad
    • And https://cookpad.com/ (except JP)
    • Existing traffic in JP is high, but most doesn’t need
    CDN
    • We have many legacy clients, also we determined so
    risky, on TLSv1.0/1.1 and 3DES issue.
    

    View Slide

28. How we
    manage & use
    Fastly
    

    View Slide

29. Tips
    • Managing Fastly services
    • VCL implements a kind of “presigned URL”
    • Retry with Restart
    • Dealing with X-Forwarded-For
    • Use as reverse proxy
    

    View Slide

30. Managing Fastly services
    • In Cookpad, we do codification for (almost) everything
    • i.e. AWS Route 53, EC2 Security Groups, ELB, …
    • Codification makes reviewing and managing history
    very easy
    • Fastly isn’t a exception
    

    View Slide

31. sorah/codily
    • https://github.com/sorah/codily
    • Simple tool to manage Fastly services
    • (Alternate way available now is Terraform, but we’ve
    never used)
    

    View Slide

32. service "global-api.cookpad.com" do
    backend "addr …" do
    hostname …
    error_threshold 0
    first_byte_timeout 21000
    weight 100
    address …
    connect_timeout 1000
    port 443
    between_bytes_timeout 10000
    auto_loadbalance false
    ssl_check_cert true
    ssl_cert_hostname "global-api.cookpad.com"
    max_conn 200
    use_ssl true
    end
    domain "global-api.cookpad.com"
    condition '!req.url' do # for logging
    priority 10
    statement '!req.url'
    type 'RESPONSE'
    end
    vcl "default" do
    content file: 'global-api.cookpad.com.vcl'
    main true
    

    View Slide

33. sorah/codily
    • codily --apply

    --target my-awesome—service

    --activate

    • So easy
    • Useful… especially when testing VCLs

    (Edit in local editor → Run command → Activation done →
    Test → Edit → …)
    

    View Slide

34. Presigned URL
    • For some reason we need to restrict access to some
    dynamically resized image in CDN
    • But, we want to keep cache across users
    

    View Slide

35. Presigned URL
    • Fastly VCL has “Cryptographic- and hashing-related VCL
    functions”
    • Implement HMAC signature based access control in VCL
    

    View Slide

36. if (!table.lookup(keys_table, req.http.X-Tmp-Sign-Key)) {
    error 701 "forbidden";
    }
    if (time.is_after(now,

    std.integer2time(std.atoi(req.http.X-Tmp-Expiry)))) {
    error 701 "forbidden";
    }
    set req.http.X-Tmp-Path = regsub(req.url.path, …)
    if (req.http.X-Tmp-Signature !=

    digest.hmac_sha256(

    table.lookup(keys_table, req.http.X—Tmp-Sign-Key),

    req.http.X-Tmp-Expiry ":" req.http.X-Tmp-Path)) {
    error 701 "forbidden";
    }
    } else {
    error 701 "forbidden";
    }
    } else {
    error 701 "forbidden";
    }
    

    View Slide

37. Presigned URL
    • ✔ ,FFQDBDIFDPWFSBHF
    • ✔ ,FFQTFDVSJUZ
    

    View Slide

38. Retry with Restart
    • Varnish, of course Fastly has “restart”
    

    View Slide

39. Retry with Restart
    • “restart” allows us to restart processing from the
    (almost) beginning
    • What’s important here is “some variables are kept after
    restarts”
    • We can do any magics here, like dispatching multiple
    backend requests, in a single user request,
    intentionally, using “restart”
    • (Original use case is just for retry I guess)
    

    View Slide

40. Retry with Restart
    • We replicates some production data to development
    servers
    • But for images, without copying, we implement “retry
    on production backend” when 404 returned from
    development backend
    

    View Slide

41. sub vcl_recv {
    # …
    set req.backend = F_dev_origin;
    if (req.http.X-Dev-Retry && req.restarts > 0) {
    set req.http.Fastly-Force-Shield = "yes";
    set req.backend = F_prd_origin;
    }
    sub vcl_deliver {
    # …
    if (server.identity ~ "-IAD$" || req.http.Fastly-FF) {
    if (resp.status != 200 && !req.http.X-Dev-Retry) {
    set req.http.X-Dev-Retry = "1";
    restart;
    }
    

    View Slide

42. Retry with Restart
    • (Found this is mentioned in the official doc now)
    • https://docs.fastly.com/guides/performance-tuning/
    checking-multiple-backends-for-a-single-request
    

    View Slide

43. Dealing with X-Forwarded-For
    • Source IP of requests is important.
    • Logging, Analyzing …
    • We have to deal with X-Forwarded-For header when
    requests come from CDN
    

    View Slide

44. Dealing with X-Forwarded-For
    • Most web application framework whitelists “private IP
    range” for a proxy included in X-Forwarded-For header
    • Implementing whitelisting Fastly IPs carefully in every
    app takes maintenance cost
    • And, some load balancers (e.g. AWS ELB) overwrites
    XFF header!
    

    View Slide

45. Dealing with X-Forwarded-For
    • How we did:
    • Send X-Forwarded-For in different name

    (Requires VCL)
    • Overwrite X-Forwarded-For with it, , on our-side
    reverse proxy… when request comes from Fastly IPs
    

    View Slide

46. Dealing with X-Forwarded-For
    # nginx.conf
    geo $trust_xff {
    # list Fastly IPs
    }
    server {
    set $cond “$trust_xff,$http_x_my_xff";
    if ($cond ~ "1,.") {
    set $myxff “$http_x_my_xff, $remote_addr";
    }
    if ($myxff = "") {
    set $myxff $proxy_add_x_forwarded_for;
    }
    proxy_set_header X-My-XFF $myxff;
    proxy_set_header X-Forwarded-For $myxff;
    proxy_set_header X-Raw-Forwarded-For $proxy_add_x_forwarded_for;
    

    View Slide

47. Use as a reverse proxy
    

    View Slide

48. Use as a reverse proxy
    sub vcl_recv {
    #FASTLY recv
    # Never cache by default
    return(pass);
    }
    

    View Slide

49. ˖ 8FˏSFTPDBSFGVMMZQVUUJOHEZOBNJDDPOUFOUPO
    $%/
    ˖ 6TFreturn (fetch);POMZJOTQFDJDDPOEJUJPO
    • .VDITBGFS
    • 8FEPTBNFXIJUFMJTUJOHPOPVS7BSOJTIPO&$
    Use as a reverse proxy
    

    View Slide

50. ˖ 2*T$%/XJUIPVU$BDIJOHFDJFOU
    ˖ "4FFPVSEBUB
    Use as a reverse proxy
    

    View Slide

51. Indonesia
    

    View Slide

52. Argentina
    

    View Slide

53. UAE
    

    View Slide

54. ˖ 5FSNJOBUF5-45$1UPFOEVTFSPOUIFJSDMPTFTU
    TFSWFSJTSFBMMZJNQPSUBOU
    ˖ ,FFQ"MJWFGSPN$%/101BMTPXPSLTHSFBU
    Use as a reverse proxy
    

    View Slide

55. Whew…
    

    View Slide

56. What we require to a CDN
    • Stable
    • Don’t obscure failures, incidents
    • High cache efficiency
    

    View Slide

57. View Slide

58. View Slide

59. What we require to a CDN
    • Stable
    • I think it’s really improved in these years
    • Don’t obscure failures, incidents
    • status.fastly.com always truthful
    • High cache efficiency
    

    View Slide

60. Conclusion
    • Cookpad Global is using Fastly
    • Introduced how we use Fastly
    

    View Slide