Don't Let the Weeds Overwhelm the Garden

A year ago, our technical debt was incurring compound interest and we weren’t going to be able to keep making the minimum payments. We had to take a hard look at how we built our infrastructure—we had a code base that had evolved over time, with no design patterns and little structure. Changes in one Chef cookbook would have cascading failures in seemingly unrelated places elsewhere. We had dependencies upon dependencies. We faced a critical decision: fix what we have in place or start over again using what we’d had before as a guide. We chose the latter.

And so, over the past 12 months, we rewrote the majority of our infrastructure code, upgraded the operating system, and rebuilt every single server in our public cache fleet from the ground up—without anyone noticing. Previously, it took Fastly about six people over six weeks to install a new POP. Now, it requires just one person and a single week.

We attribute our ability to rapidly scale, both safely and reliably, to a new approach, which we'll discuss in this talk. We will relive the design and redeployment of our public cache fleet, the challenges faced, lessons learned, and benefits gained from our new architecture.

Transcript

1. Laine Campbell SVP Engineering Marcus Barczak Principal Engineer Fastly presents presents NYC Don't let the weeds overwhelm the garden April 17, 2018
2. https:/ /www.flickr.com/photos/reading-matters/20946940348/
3. presents NYC Don't let the weeds overwhelm the garden commit 6dbb0cb2788e04885ca4bae86b25a83c38dd34c7 Date: Early 2011 Initial commit of the chef-repo diff --git a/metadata.json b/metadata.json
4. 6 years of development 157 contributors 211 cookbooks 160K lines of code 1000's of servers Fastly Chef 2011-2017 presents NYC Don't let the weeds, overwhelm the garden
5. presents NYC Don't let the weeds overwhelm the garden 2011
6. presents NYC Don't let the weeds overwhelm the garden "Cattle not Pets" - Bill Baker: "Scaling SQL Server" 2012
7. presents NYC Don't let the weeds overwhelm the garden Chef Ecosystem Grows 2013
8. presents NYC Don't let the weeds overwhelm the garden Test Kitchen 1.0 2014
9. presents NYC Don't let the weeds overwhelm the garden Cloud Native 2015...
10. ‣ Deploying new PoPs problematic ‣ Code was hard to maintain. ‣ Unexpected side effects from change. ‣ Engineers feared working on the code. Weeds growing presents NYC Don't let the weeds, overwhelm the garden
11. presents NYC Don't let the weeds overwhelm the garden We are here!
12. presents NYC Don't let the weeds overwhelm the garden ‣ Embedded config increasing code complexity. ‣ Config duplication introducing subtle bugs. ‣ Tightly coupled cookbooks. ‣ No consistent design pattern. Grokking the Problem
13. "Witches Brew Architecture" - Neal Ford: "Software Architecture Fundamentals"
14. Establish a design pattern. Rewrite the cookbooks. Upgrade to Ubuntu 16.04. Roll Out. Without production impact! Project Defenestration presents NYC Don't let the weeds, overwhelm the garden
15. presents NYC Don't let the weeds overwhelm the garden Separate Config
16. presents NYC Don't let the weeds overwhelm the garden Separate Config from Code chef server cookbooks infrastructure code server … server server databags node attributes environments
17. presents NYC Don't let the weeds overwhelm the garden chef configly service configuration ipam network configuration server cookbooks infrastructure code server … server server ohai plugin Separate Config from Code
18. presents NYC Don't let the weeds overwhelm the garden chef configly service configuration ipam network configuration server cookbooks infrastructure code server … server server network management tools provisioning tools deployment tools service management tools inventory tools ohai plugin Separate Config from Code
19. presents NYC Don't let the weeds overwhelm the garden ‣ Flexible data model. ‣ Source of truth for all systems. ‣ Separation of concern. ‣ Config changes without chef code changes. Lesson: Separating Config
20. presents NYC Don't let the weeds overwhelm the garden Developer Ergonomics
21. presents NYC Don't let the weeds overwhelm the garden ‣ Library cookbooks - reusable components. ‣ App cookbooks - compose layers from library components. ‣ Tooling - code generation, CI and deployment. Chef Design Pattern
22. presents NYC Don't let the weeds overwhelm the garden module VarnishService class Resource < ILoveSystemdService::Resource resource_name :varnish_service property :cluster, kind_of: String, required: true property :hc_networks, kind_of: Array, default: [] property :limit_memlock, kind_of: Integer, default: 82000 property :limit_nofile, kind_of: Integer, default: 131072 ... Library cookbooks
23. presents NYC Don't let the weeds overwhelm the garden module VarnishService class Resource < ILoveSystemdService::Resource resource_name :varnish_service property :cluster, kind_of: String, required: true property :hc_networks, kind_of: Array, default: [] property :limit_memlock, kind_of: Integer, default: 82000 property :limit_nofile, kind_of: Integer, default: 131072 ... Library cookbooks
24. presents NYC Don't let the weeds overwhelm the garden ... varnish_service 'default' do cluster ipam_pop.upcase hc_networks ipam_access_interface_broadcasts configly_resource_properties('varnish', 'service_properties') action :create only_if { node_is_drained? } end ... Application cookbooks
25. presents NYC Don't let the weeds overwhelm the garden { "varnish": { "service_properties": { "region": "US-East", "limit_memlock": 90000, "limit_nofile": 2000000, "parameters": { "hc_enabled": "on", "lru_interval": 60, ... Configuration data configly_resource_properties('varnish', 'service_properties') Application cookbook code Configly Data
26. presents NYC Don't let the weeds overwhelm the garden ‣ Make the easy way, the right way. ‣ Generators to bootstrap new cookbooks. ‣ Testing built in by default. ‣ Easy integration with CI and deploy processes. ‣ Engineers focus on the functionality. Lesson: Easy Way = Right Way
27. presents NYC Don't let the weeds overwhelm the garden Cookbook Layers
28. presents NYC Don't let the weeds overwhelm the garden App Cookbook Layering Ubuntu 16.04 cache server base hardware cache stack operating sys. infra net
29. presents NYC Don't let the weeds overwhelm the garden App Cookbook Layering Add our users and common OS customization. Ubuntu 16.04 cache server base hardware cache stack operating sys. infra net
30. presents NYC Don't let the weeds overwhelm the garden App Cookbook Layering Add our users and common OS customization. Manages our network stack. Ubuntu 16.04 Manages the kernel and hardware specifics. cache server base hardware cache stack operating sys. infra net
31. presents NYC Don't let the weeds overwhelm the garden App Cookbook Layering Add our users and common OS customization. Manages our network stack. Manages cache software stack. Ubuntu 16.04 Manages the kernel and hardware specifics. cache server base hardware cache stack operating sys. infra net
32. presents NYC Don't let the weeds overwhelm the garden App Cookbook Layering cache server base hardware cache stack operating sys. infra net configly ipam
33. presents NYC Don't let the weeds overwhelm the garden cache server base hardware cache stack operating sys. infra net micro server base hardware cache stack operating sys. micro net swap out network layer configly ipam App Cookbook Layering
34. presents NYC Don't let the weeds overwhelm the garden cache server base hardware cache stack operating sys. infra net micro server base hardware cache stack operating sys. micro net cloud server base cloudy app operating sys. swap out network layer configly ipam remove layers entirely App Cookbook Layering
35. presents NYC Don't let the weeds overwhelm the garden ‣ Layers can be reused without code change. ‣ Environmental differences modeled in configuration. ‣ Layers can be owned and maintained by teams. ‣ Easily testable and understandable. Lesson: Reusability
36. presents NYC Don't let the weeds overwhelm the garden Ship It!
37. 9 months of development 47 code contributors 101 config contributors 30K lines of chef code (81% less code!) 16k lines of config data. Fastly Chef 2018 presents NYC Don't let the weeds, overwhelm the garden
38. presents NYC Don't let the weeds overwhelm the garden ‣ Entire Fastly fleet rebuilt over 3 months. ‣ PoP builds are faster and more efficient. ‣ Increased developer happiness. ‣ Improved infrastructure reliability. Fully Deployed!
39. presents NYC Don't let the weeds overwhelm the garden
40. presents NYC Thanks!