Fastly in Cookpad - Speaker Deck

Slide 1

Slide 1 text

Fastly in Cookpad Sorah Fukumori @ Fastly Yamagoya 2017

Slide 2

Slide 2 text

$ whoami Sorah Fukumori (׉׵כ https://sorah.jp/ | GitHub @sorah | Twitter @sora_h Site Reliability Engineer at Cookpad Global Rubyist, Ruby committer Operating AS59128 Interests: Site Reliability, Networking, Distributed systems

Slide 3

Slide 3 text

Agenda • About us • Latency • Fastly usage in Cookpad • How we manage/use Fastly

Slide 4

Slide 4 text

Cookpad? • The world’s largest recipe sharing platform • 69 countries, 22 regions • Rapidly growing in the world • Code is diﬀerent, unshared between Global and JP

Slide 5

Slide 5 text

https://cookpad.com/uk

Slide 6

Slide 6 text

https://cookpad.com/

Slide 7

Slide 7 text

Cookpad? • The world’s largest recipe sharing platform • 69 countries, 22 regions • Rapidly growing in the world • Code is diﬀerent, unshared between Global and JP • But the domain name is shared!

Slide 8

Slide 8 text

Infrastructure • Everything on AWS • JP uses Tokyo (ap-northest-1),  Global uses US East (N.Virginia, us-east-1) • 2 CDNs (Fastly, CloudFront)

Slide 9

Slide 9 text

LATENCY

Slide 10

Slide 10 text

Fighting with Latency • Assume everyone know latency aﬀects user experience heavily • TCP/TLS handshake…

Slide 11

Slide 11 text

Fighting with Latency • Q. Apps live in multiple regions, but share the single, same domain • … What?

Slide 12

Slide 12 text

• We chose Route53 Latency Based Routing at first • DNS returns IP addresses of the closer region from a resolver • If a requested service lives in another region, reverse- proxy to the alternate region Fighting with Latency

Slide 13

Slide 13 text

! "

Slide 14

Slide 14 text

location ~ ^/(ae|ar|bh|bo|br|cl|co|cr| cu|de|dj|do|dz|ec|eeuu|eg|es|fr|gt|hn| hu|id|in|iq|ir|it|jo|km|kr|kw|lb|ly|ma| mr|mx|ni|om|pa|pe|ph|pri|ps|py|qa|sa|sd| so|sv|sy|th|tn|tw|uk|us|uy|ve|vn|ye)(/| $) { proxy_pass http://cookpad_use1; } location / { proxy_pass https://cookpad_apne1; }

Slide 15

Slide 15 text

• This is the minimum • 2 regions are not enough • We have to serve in Middle East, Europe, UK … • They’re too far away from both US east and Japan Fighting with Latency

Slide 16

Slide 16 text

• But who want to manage many proxy servers in the all available AWS regions? • tips: AWS doesn’t cover the • CDN? Fighting with Latency

Slide 17

Slide 17 text

Past CDN usage in Cookpad • User provided Images • Dynamic resizing provided by an internal service • Cache resized images • Static assets (CSS, JavaScript, Images) • No Dynamic Content

Slide 18

Slide 18 text

No Dynamic Content • No dynamic content • (Contents served via CDN are guaranteed static by their URL) • As an old-school CDN user: • Slow deployments, slow purging… • It’s too risky when we fail at something

Slide 19

Slide 19 text

Meet with Fastly • “less-risky?” • Fast purge! • Fast deployment! • VCL! • Decided to start using as a replacement of existing CDN, to take a try

Slide 20

Slide 20 text

The first use case • Images and Assets for Global Platform

Slide 21

Slide 21 text

The first use case • Images and Assets for Global Platform • Achieved higher hit rate, better performance  than our previous CDN provider • Shielding <3 • Few and fast POPs

Slide 22

Slide 22 text

now, Fastly in Cookpad • Primarily for Cookpad Global • 13 services, and 7 in production

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Fastly in Cookpad • Global Platform • Static Assets • Images (user provided) • API

Slide 25

Slide 25 text

Fastly in Cookpad • And https://cookpad.com/

Slide 26

Slide 26 text

Fastly in Cookpad • And https://cookpad.com/ (except JP)

Slide 27

Slide 27 text

Fastly in Cookpad • And https://cookpad.com/ (except JP) • Existing traﬀic in JP is high, but most doesn’t need CDN • We have many legacy clients, also we determined so risky, on TLSv1.0/1.1 and 3DES issue.

Slide 28

Slide 28 text

How we manage & use Fastly

Slide 29

Slide 29 text

Tips • Managing Fastly services • VCL implements a kind of “presigned URL” • Retry with Restart • Dealing with X-Forwarded-For • Use as reverse proxy

Slide 30

Slide 30 text

Managing Fastly services • In Cookpad, we do codification for (almost) everything • i.e. AWS Route 53, EC2 Security Groups, ELB, … • Codification makes reviewing and managing history very easy • Fastly isn’t a exception

Slide 31

Slide 31 text

sorah/codily • https://github.com/sorah/codily • Simple tool to manage Fastly services • (Alternate way available now is Terraform, but we’ve never used)

Slide 32

Slide 32 text

service "global-api.cookpad.com" do backend "addr …" do hostname … error_threshold 0 first_byte_timeout 21000 weight 100 address … connect_timeout 1000 port 443 between_bytes_timeout 10000 auto_loadbalance false ssl_check_cert true ssl_cert_hostname "global-api.cookpad.com" max_conn 200 use_ssl true end domain "global-api.cookpad.com" condition '!req.url' do # for logging priority 10 statement '!req.url' type 'RESPONSE' end vcl "default" do content file: 'global-api.cookpad.com.vcl' main true

Slide 33

Slide 33 text

sorah/codily • codily --apply  --target my-awesome—service  --activate  • So easy • Useful… especially when testing VCLs  (Edit in local editor → Run command → Activation done → Test → Edit → …)

Slide 34

Slide 34 text

Presigned URL • For some reason we need to restrict access to some dynamically resized image in CDN • But, we want to keep cache across users

Slide 35

Slide 35 text

Presigned URL • Fastly VCL has “Cryptographic- and hashing-related VCL functions” • Implement HMAC signature based access control in VCL

Slide 36

Slide 36 text

if (!table.lookup(keys_table, req.http.X-Tmp-Sign-Key)) { error 701 "forbidden"; } if (time.is_after(now,  std.integer2time(std.atoi(req.http.X-Tmp-Expiry)))) { error 701 "forbidden"; } set req.http.X-Tmp-Path = regsub(req.url.path, …) if (req.http.X-Tmp-Signature !=  digest.hmac_sha256(  table.lookup(keys_table, req.http.X—Tmp-Sign-Key),  req.http.X-Tmp-Expiry ":" req.http.X-Tmp-Path)) { error 701 "forbidden"; } } else { error 701 "forbidden"; } } else { error 701 "forbidden"; }

Slide 37

Slide 37 text

Presigned URL • ✔ ,FFQDBDIFDPWFSBHF • ✔ ,FFQTFDVSJUZ

Slide 38

Slide 38 text

Retry with Restart • Varnish, of course Fastly has “restart”

Slide 39

Slide 39 text

Retry with Restart • “restart” allows us to restart processing from the (almost) beginning • What’s important here is “some variables are kept after restarts” • We can do any magics here, like dispatching multiple backend requests, in a single user request, intentionally, using “restart” • (Original use case is just for retry I guess)

Slide 40

Slide 40 text

Retry with Restart • We replicates some production data to development servers • But for images, without copying, we implement “retry on production backend” when 404 returned from development backend

Slide 41

Slide 41 text

sub vcl_recv { # … set req.backend = F_dev_origin; if (req.http.X-Dev-Retry && req.restarts > 0) { set req.http.Fastly-Force-Shield = "yes"; set req.backend = F_prd_origin; } sub vcl_deliver { # … if (server.identity ~ "-IAD$" || req.http.Fastly-FF) { if (resp.status != 200 && !req.http.X-Dev-Retry) { set req.http.X-Dev-Retry = "1"; restart; }

Slide 42

Slide 42 text

Retry with Restart • (Found this is mentioned in the oﬀicial doc now) • https://docs.fastly.com/guides/performance-tuning/ checking-multiple-backends-for-a-single-request

Slide 43

Slide 43 text

Dealing with X-Forwarded-For • Source IP of requests is important. • Logging, Analyzing … • We have to deal with X-Forwarded-For header when requests come from CDN

Slide 44

Slide 44 text

Dealing with X-Forwarded-For • Most web application framework whitelists “private IP range” for a proxy included in X-Forwarded-For header • Implementing whitelisting Fastly IPs carefully in every app takes maintenance cost • And, some load balancers (e.g. AWS ELB) overwrites XFF header!

Slide 45

Slide 45 text

Dealing with X-Forwarded-For • How we did: • Send X-Forwarded-For in diﬀerent name  (Requires VCL) • Overwrite X-Forwarded-For with it, , on our-side reverse proxy… when request comes from Fastly IPs

Slide 46

Slide 46 text

Dealing with X-Forwarded-For # nginx.conf geo $trust_xff { # list Fastly IPs } server { set $cond “$trust_xff,$http_x_my_xff"; if ($cond ~ "1,.") { set $myxff “$http_x_my_xff, $remote_addr"; } if ($myxff = "") { set $myxff $proxy_add_x_forwarded_for; } proxy_set_header X-My-XFF $myxff; proxy_set_header X-Forwarded-For $myxff; proxy_set_header X-Raw-Forwarded-For $proxy_add_x_forwarded_for;

Slide 47

Slide 47 text

Use as a reverse proxy

Slide 48

Slide 48 text

Use as a reverse proxy sub vcl_recv { #FASTLY recv # Never cache by default return(pass); }

Slide 49

Slide 49 text

˖ 8FˏSFTPDBSFGVMMZQVUUJOHEZOBNJDDPOUFOUPO $%/ ˖ 6TFreturn (fetch);POMZJOTQFDJDDPOEJUJPO • .VDITBGFS • 8FEPTBNFXIJUFMJTUJOHPOPVS7BSOJTIPO&$ Use as a reverse proxy

Slide 50

Slide 50 text

˖ 2*T$%/XJUIPVU$BDIJOHFDJFOU ˖ "4FFPVSEBUB Use as a reverse proxy

Slide 51

Slide 51 text

Indonesia

Slide 52

Slide 52 text

Argentina

Slide 53

Slide 53 text

UAE

Slide 54

Slide 54 text

˖ 5FSNJOBUF5-45$1UPFOEVTFSPOUIFJSDMPTFTU TFSWFSJTSFBMMZJNQPSUBOU ˖ ,FFQ"MJWFGSPN$%/101BMTPXPSLTHSFBU Use as a reverse proxy

Slide 55

Slide 55 text

Whew…

Slide 56

Slide 56 text

What we require to a CDN • Stable • Don’t obscure failures, incidents • High cache eﬀiciency

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

What we require to a CDN • Stable • I think it’s really improved in these years • Don’t obscure failures, incidents • status.fastly.com always truthful • High cache eﬀiciency

Slide 60

Slide 60 text

Conclusion • Cookpad Global is using Fastly • Introduced how we use Fastly