PyConZA 2015: "Python @ CloudFlare" by Gideon Redelinghuys

CloudFlare protects and accelerates any website online. Whether it's protecting large customers from 160Gbps DDOS attacks, dealing with normal Reddit traffic, Universal SSL or providing one of the fastest DNS servers in the world we need to stay up, work fast and deploy fast.

To this end we have two large Python projects we use at CloudFlare. One is the popular configuration manager Salt. We provide patches, features and bug reports on regular intervals to the Salt project. The other is an internal tool called CFSetup. CFSetup allows us to do the following with a few simple commands: spin up Docker containers that reflect machines in production, spin up a tightly coupled stack of Docker containers that reflect our software and hardware stacks, run build environments inside of well defined Docker containers to ensure that the compiled end-product was built using the same libraries that are on the production machines, and various other utility features.

This talk will cover how Salt works, how we've extended it to scale up and how we've had to debug it. It also cover how we've managed to build a layer on top of Docker to make it easier to use for ad-hoc testing and developing and what it's like interacting with Docker using Python.

Transcript

1. Python @ CloudFlare Gideon Redelinghuys
2. Who is CloudFlare?
3. CloudFlare • CDN • Security • DDOS protection
4. Network
5. Customers • Reddit • DigitalOcean • FBI • Donald Trump / Jeb Bush
6. What problems do we face?
7. The Problem? Provide the best product, Don’t break the Internet. #savetheweb
8. Best Product • No product stays the best on its own • Changes need to be ◦ created, ◦ tested ◦ and deployed.
9. • Salt • CFSetup • … both written in Python! Tools
10. Salt
11. What is Salt • Configuration manager • Yaml / Jinja / GPG • Remote commands execution • Thousands of Salt Minions Salt Master Salt Minion Salt Minion Salt Minion 0MQ
12. Problems • Job History • Dynamic scaling up/down • Development • Network • Salt CLI
13. Job History • Store all job history in a postgres DB • … not a file hierarchy for 24 hours. Salt Master Salt Master
14. Dynamic scaling up/down • More than one salt master for backup/staging • Mesos/docker salt masters • Managing salt master keys and GPG encryption keys Salt Master Salt Master Salt Master Salt Minion Salt Minion Salt Minion
15. Development • Writing salt formula’s/states are hard • … especially when you have legacy machines • staging salt masters • CI tests highstates • salt env with git branches • docker containers to test states
16. Network • We have a large and diverse network • See all kinds of package loss and delay • Salt provides zeroMQ, RAET, and Tornado • Salt Syndic Salt Master Salt Minion Salt Syndic Salt Minion Salt Minion The Internet
17. Salt CLI • Slow • Convoluted • Unwanted logic • Hard to automate • Non standard output • Grains resolution Salt Master Internal Salt API Salt CLI deployer saltbot hipchat
18. Future of Salt • Rewrite of the Salt CLI • Key management • Mature network stack
19. CFSetup
20. CFSetup • Python tool that wraps around Docker • Allows developers to spin up Docker containers that mirror production
21. Why Python? • Not sure about the product • Develop fast • Develop and debug on the machine showing the problem.
22. Philosophy • CFSetup - interacts with developer • Makefile - specifies how a project should be built, packaged and tested • Project - contains all the code CFSetup Makefile Project
23. Use Cases - cfsetup spawn - bring up a container that acts as a certain Salt minion type - cfsetup enter - enter and exit containers easily - cfsetup build - spin up a container to build the code and produce packages. - cfsetup test - runs unit tests
24. Freebies • cfsetup apt - own repo’s • cfsetup localdns - discovery
25. Appliances • CFSetup spawns “appliances” • Docker containers that have the necessary requirements for the project • Can specified as a Salt type, or as a Dockerfile
26. Benefits of this wrapper • Projects would modify local file system, or even its own codebase. • One projects may require incompatible dependencies with another project.
27. Stacks • Allows developers to specify a set of machines as one stack. • CFSetup can then orchestrate them together. nginx container postgres container load balancer container
28. Python -> Docker • Docker-py did not work, pre 1.0 • String handling vs data struct/json handling • Shelling out to the docker cli
29. Problems with Docker • Docker version regression • Behaviour changes. • SELinux / AppArmor / firewalld
30. Future of CFSetup • more appliances • prod-like • make more compatible with different python/environments
31. Conclusion • Using salt and docker together we can write, test and deploy faster • Keep up with the Internet.
32. Questions