How to seamlessly incorporate CDN into your application stack

Content Delivery Network is a common element of today’s applications. Its configuration should be well aligned with your your backend, not only to cache content, but also to do handle many other things directly at the edge - authenticate your users, route traffic or filter out malicious requests. Keeping things in sync can be a challenge in a multi-origin environment.

We decided to use Jenkins pipelines, Terraform and Chef to automate the entire process and make it painless for the entire team.

Transcript

1. How to seamlessly incorporate CDN into your application stack https://flic.kr/p/7F4bHa
2. Jakub Wądołowski Principal Systems Engineer at Cognifide @jwadolowski
3. • The What • The Why • The How Agenda
4. What we’re dealing with… https://flic.kr/p/eE48CR
5. • Online clothing retailer • Migration from existing system to the new one • Mix of AWS and on-premise infrastructure • Integrations with 3rd-party services • HTTP routing at different levels Platform overview (1/2)
6. • Geographically distributed userbase (EU, US, AU) • 14 content domains • CDN is an integral part of the stack Platform overview (2/2)
8. Content Delivery… what? https://flic.kr/p/pnEAvJ • Large distributed system of servers deployed in multiple data centers across the Internet • Serve content to end-users with high availability and high performance
9. • caching (and invalidation!) • HTTP routing and rewriting • traffic filtering (WAF) • authentication • … CDN responsibilities
10. CDN provider
11. • Powered by Varnish • Highly customizable (VCL) • Pay-as-you-go model • Quality over quantity Fastly overview
12. Fastly customers
15. • Terraform is just a half of the story • Domain-agnostic VCL configuration is stored elsewhere • Multiple environments • Fine-grained control what goes deployed where Is that it?
16. • Terraform repo Git repository layout
17. Logical environment AWS infrastructure CDN config Domain/service specific VCLs
18. • Terraform repo • Fastly config repo (default.vcl) Git repository layout
20. • Terraform repo • Fastly config repo (default.vcl) • Jenkins pipeline repo(s) Git repository layout
21. Pipelines Configurations
 (YAML) Library
 (custom steps)
22. Jenkinsfile
23. cfg.yaml Custom steps
24. Custom steps
25. • Terraform repo • Fastly config repo (default.vcl) • Jenkins pipeline repo(s) • Web server config repo Git repository layout
27. Let’s put all these things together https://flic.kr/p/qm4gV8
28. • Clone Terraform and config repositories • Map CDN config version to Fastly service • Transfer VCL to Terraform repo • Initialize Terraform • Push CDN changes using Terraform CDN deployment process
30. • The entire config stored in Git repository • Configuration contains caching and invalidation rules • Chef fetches Git repo and adjusts it Web server deployments
32. Demo! https://flic.kr/p/gq5UAU