Fastly in Cookpad

Transcript

1. 1.

   Fastly in Cookpad Sorah Fukumori <sorah@cookpad.com> @ Fastly Yamagoya 2017

2. 2.

   $ whoami Sorah Fukumori (׉׵כ https://sorah.jp/ | GitHub @sorah |

   Twitter @sora_h Site Reliability Engineer at Cookpad Global Rubyist, Ruby committer Operating AS59128 Interests: Site Reliability, Networking, Distributed systems
3. 3.

   Agenda • About us • Latency • Fastly usage in

   Cookpad • How we manage/use Fastly
4. 4.

   Cookpad? • The world’s largest recipe sharing platform • 69

   countries, 22 regions • Rapidly growing in the world • Code is different, unshared between Global and JP
5. 5.

   https://cookpad.com/uk

6. 6.

   https://cookpad.com/

7. 7.

   Cookpad? • The world’s largest recipe sharing platform • 69

   countries, 22 regions • Rapidly growing in the world • Code is different, unshared between Global and JP • But the domain name is shared!
8. 8.

   Infrastructure • Everything on AWS • JP uses Tokyo (ap-northest-1),


   Global uses US East (N.Virginia, us-east-1) • 2 CDNs (Fastly, CloudFront)
9. 9.

   LATENCY

10. 10.

    Fighting with Latency • Assume everyone know latency affects user

    experience heavily • TCP/TLS handshake…
11. 11.

    Fighting with Latency • Q. Apps live in multiple regions,

    but share the single, same domain • … What?
12. 12.

    • We chose Route53 Latency Based Routing at first •

    DNS returns IP addresses of the closer region from a resolver • If a requested service lives in another region, reverse- proxy to the alternate region Fighting with Latency
13. 13.

    ! "

14. 14.

    location ~ ^/(ae|ar|bh|bo|br|cl|co|cr| cu|de|dj|do|dz|ec|eeuu|eg|es|fr|gt|hn| hu|id|in|iq|ir|it|jo|km|kr|kw|lb|ly|ma| mr|mx|ni|om|pa|pe|ph|pri|ps|py|qa|sa|sd| so|sv|sy|th|tn|tw|uk|us|uy|ve|vn|ye)(/| $) { proxy_pass

    http://cookpad_use1; } location / { proxy_pass https://cookpad_apne1; }
15. 15.

    • This is the minimum • 2 regions are not

    enough • We have to serve in Middle East, Europe, UK … • They’re too far away from both US east and Japan Fighting with Latency
16. 16.

    • But who want to manage many proxy servers in

    the all available AWS regions? • tips: AWS doesn’t cover the • CDN? Fighting with Latency
17. 17.

    Past CDN usage in Cookpad • User provided Images •

    Dynamic resizing provided by an internal service • Cache resized images • Static assets (CSS, JavaScript, Images) • No Dynamic Content
18. 18.

    No Dynamic Content • No dynamic content • (Contents served

    via CDN are guaranteed static by their URL) • As an old-school CDN user: • Slow deployments, slow purging… • It’s too risky when we fail at something
19. 19.

    Meet with Fastly • “less-risky?” • Fast purge! • Fast

    deployment! • VCL! • Decided to start using as a replacement of existing CDN, to take a try
20. 20.

    The first use case • Images and Assets for Global

    Platform
21. 21.

    The first use case • Images and Assets for Global

    Platform • Achieved higher hit rate, better performance
 than our previous CDN provider • Shielding <3 • Few and fast POPs
22. 22.

    now, Fastly in Cookpad • Primarily for Cookpad Global •

    13 services, and 7 in production
23. 23.

    None
24. 24.

    Fastly in Cookpad • Global Platform • Static Assets •

    Images (user provided) • API
25. 25.

    Fastly in Cookpad • And https://cookpad.com/

26. 26.

    Fastly in Cookpad • And https://cookpad.com/ (except JP)

27. 27.

    Fastly in Cookpad • And https://cookpad.com/ (except JP) • Existing

    traffic in JP is high, but most doesn’t need CDN • We have many legacy clients, also we determined so risky, on TLSv1.0/1.1 and 3DES issue.
28. 28.

    How we manage & use Fastly

29. 29.

    Tips • Managing Fastly services • VCL implements a kind

    of “presigned URL” • Retry with Restart • Dealing with X-Forwarded-For • Use as reverse proxy
30. 30.

    Managing Fastly services • In Cookpad, we do codification for

    (almost) everything • i.e. AWS Route 53, EC2 Security Groups, ELB, … • Codification makes reviewing and managing history very easy • Fastly isn’t a exception
31. 31.

    sorah/codily • https://github.com/sorah/codily • Simple tool to manage Fastly services

    • (Alternate way available now is Terraform, but we’ve never used)
32. 32.

    service "global-api.cookpad.com" do backend "addr …" do hostname … error_threshold

    0 first_byte_timeout 21000 weight 100 address … connect_timeout 1000 port 443 between_bytes_timeout 10000 auto_loadbalance false ssl_check_cert true ssl_cert_hostname "global-api.cookpad.com" max_conn 200 use_ssl true end domain "global-api.cookpad.com" condition '!req.url' do # for logging priority 10 statement '!req.url' type 'RESPONSE' end vcl "default" do content file: 'global-api.cookpad.com.vcl' main true
33. 33.

    sorah/codily • codily --apply
 --target my-awesome—service
 --activate
 • So easy

    • Useful… especially when testing VCLs
 (Edit in local editor → Run command → Activation done → Test → Edit → …)
34. 34.

    Presigned URL • For some reason we need to restrict

    access to some dynamically resized image in CDN • But, we want to keep cache across users
35. 35.

    Presigned URL • Fastly VCL has “Cryptographic- and hashing-related VCL

    functions” • Implement HMAC signature based access control in VCL
36. 36.

    if (!table.lookup(keys_table, req.http.X-Tmp-Sign-Key)) { error 701 "forbidden"; } if (time.is_after(now,


    std.integer2time(std.atoi(req.http.X-Tmp-Expiry)))) { error 701 "forbidden"; } set req.http.X-Tmp-Path = regsub(req.url.path, …) if (req.http.X-Tmp-Signature !=
 digest.hmac_sha256(
 table.lookup(keys_table, req.http.X—Tmp-Sign-Key),
 req.http.X-Tmp-Expiry ":" req.http.X-Tmp-Path)) { error 701 "forbidden"; } } else { error 701 "forbidden"; } } else { error 701 "forbidden"; }
37. 37.

    Presigned URL • ✔ ,FFQ
    DBDIF
    DPWFSBHF
    • ✔ ,FFQ
    TFDVSJUZ

38. 38.

    Retry with Restart • Varnish, of course Fastly has “restart”

39. 39.

    Retry with Restart • “restart” allows us to restart processing

    from the (almost) beginning • What’s important here is “some variables are kept after restarts” • We can do any magics here, like dispatching multiple backend requests, in a single user request, intentionally, using “restart” • (Original use case is just for retry I guess)
40. 40.

    Retry with Restart • We replicates some production data to

    development servers • But for images, without copying, we implement “retry on production backend” when 404 returned from development backend
41. 41.

    sub vcl_recv { # … set req.backend = F_dev_origin; if

    (req.http.X-Dev-Retry && req.restarts > 0) { set req.http.Fastly-Force-Shield = "yes"; set req.backend = F_prd_origin; } sub vcl_deliver { # … if (server.identity ~ "-IAD$" || req.http.Fastly-FF) { if (resp.status != 200 && !req.http.X-Dev-Retry) { set req.http.X-Dev-Retry = "1"; restart; }
42. 42.

    Retry with Restart • (Found this is mentioned in the

    official doc now) • https://docs.fastly.com/guides/performance-tuning/ checking-multiple-backends-for-a-single-request
43. 43.

    Dealing with X-Forwarded-For • Source IP of requests is important.

    • Logging, Analyzing … • We have to deal with X-Forwarded-For header when requests come from CDN
44. 44.

    Dealing with X-Forwarded-For • Most web application framework whitelists “private

    IP range” for a proxy included in X-Forwarded-For header • Implementing whitelisting Fastly IPs carefully in every app takes maintenance cost • And, some load balancers (e.g. AWS ELB) overwrites XFF header!
45. 45.

    Dealing with X-Forwarded-For • How we did: • Send X-Forwarded-For

    in different name
 (Requires VCL) • Overwrite X-Forwarded-For with it, , on our-side reverse proxy… when request comes from Fastly IPs
46. 46.

    Dealing with X-Forwarded-For # nginx.conf geo $trust_xff { # list

    Fastly IPs } server { set $cond “$trust_xff,$http_x_my_xff"; if ($cond ~ "1,.") { set $myxff “$http_x_my_xff, $remote_addr"; } if ($myxff = "") { set $myxff $proxy_add_x_forwarded_for; } proxy_set_header X-My-XFF $myxff; proxy_set_header X-Forwarded-For $myxff; proxy_set_header X-Raw-Forwarded-For $proxy_add_x_forwarded_for;
47. 47.

    Use as a reverse proxy

48. 48.

    Use as a reverse proxy sub vcl_recv { #FASTLY recv

    # Never cache by default return(pass); }
49. 49.

    ˖ 8FˏSF
    TP
    DBSFGVMMZ
    QVUUJOH
    EZOBNJD
    DPOUFOU
    PO
    $%/
    ˖ 6TF
    return (fetch);
    POMZ
    JO
    TQFDJD
    DPOEJUJPO
    • .VDI
    TBGFS
    • 8F
    EP
    TBNF
    XIJUFMJTUJOH
    PO
    PVS
    7BSOJTI
    PO
    &$

    Use as a reverse proxy
50. 50.

    ˖ 2
    *T
    $%/
    XJUIPVU
    $BDIJOH
    FDJFOU
    ˖ "
    4FF
    PVS
    EBUB Use as a reverse proxy

51. 51.

    Indonesia

52. 52.

    Argentina

53. 53.

    UAE

54. 54.

    ˖ 5FSNJOBUF
    5-45$1
    UP
    FOE
    VTFS
    PO
    UIFJS
    DMPTFTU
    TFSWFS
    JT
    SFBMMZ
    JNQPSUBOU
    ˖ ,FFQ"MJWF
    GSPN
    $%/
    101
    BMTP
    XPSLT
    HSFBU Use as a reverse proxy

55. 55.

    Whew…

56. 56.

    What we require to a CDN • Stable • Don’t

    obscure failures, incidents • High cache efficiency
57. 57.

    None
58. 58.

    None
59. 59.

    What we require to a CDN • Stable • I

    think it’s really improved in these years • Don’t obscure failures, incidents • status.fastly.com always truthful • High cache efficiency
60. 60.

    Conclusion • Cookpad Global is using Fastly • Introduced how

    we use Fastly