Bloomberg: How We Engineer Infrastructure

NY Chef Meetup

Transcript

1. BLOOMBERG HOW WE ENGINEER INFRASTRUCTURE @johnbellone
2. HOW WE ENGINEER INFRASTRUCTURE WHO IS THIS GUY? ▸ Team Lead, Platform Engineering (The Chef guy) ▸ Responsible for leveling up configuration management at Bloomberg ▸ Define best practices around infrastructure code development, deployment and lifecycle ▸ Provide the tools necessary to grow a DevOps organization
3. HOW WE ENGINEER INFRASTRUCTURE TERMINOLOGY ▸ Application cookbook is a cookbook that installs, configures and manages a particular application on a node (e.g. Redis, NRPE, nginx) ▸ Cluster cookbook is a wrapper cookbook that configures and tunes an application on a node to be part of a cluster (e.g. Consul Cluster, Vault Cluster) ▸ Resource is simply a Chef language primitive that is distributed and exposed by including a cookbook ▸ Service is the application process managed by the operating system (or some other process management framework)
4. HOW WE ENGINEER INFRASTRUCTURE THE TECHNICAL LANDSCAPE ▸ Variety of operating systems: CentOS (RHEL), Ubuntu, Windows, AIX, Solaris ▸ An eclectic compute infrastructure: bare-metal servers, Linux containers, Solaris Zones, AIX WPAR, and virtual servers ▸ Public and private cloud providers: VMware, OpenStack, AWS, Azure ▸ Container management: Spinnaker, Mesos, Docker, Kubernetes
5. HOW WE ENGINEER INFRASTRUCTURE THE PANCEA ▸ Allow for well-designed infrastructure code to be consumed by downstream engineering teams quickly and easily ▸ Support all operating system platforms and versions ▸ Integration testing for all the platforms and versions ▸ Prefer convention over configuration
6. HOW WE ENGINEER INFRASTRUCTURE THE POISE FRAMEWORK ▸ A set of libraries for writing re-usable infrastructure code with Chef by @coderanger (Noah Kantrowitz) ▸ Ability for the infrastructure code to be written once and confidently work on esoteric platforms with little-to-no changes ▸ Much of the Chef Client 12.5 custom resource functionality started here in one form or another
7. HOW WE ENGINEER INFRASTRUCTURE DESIGN METHODOLOGY ▸ The default recipe is always, always convention over configuration ▸ The recipe is the same if installation via package or archive ▸ The recipe is the same if on Ubuntu, RHEL, Windows or Solaris ▸ Provide resources for managing all aspects of an application, e.g. installation, config, process supervision ▸ Advanced usage generally requires wrapper cookbook and attribute tuning
12. CASE STUDY REDIS COOKBOOK GITHUB.COM/BLOOMBERG/REDIS-COOKBOOK
13. REDIS COOKBOOK INSTALLATION RESOURCE ▸ Support installation of the Redis database from archive or packages ▸ Use the operating system packages when applicable (e.g. EPEL, Ubuntu) ▸ Allow for easy overrides when installing behind proxies or firewalls
15. REDIS COOKBOOK SERVICE RESOURCE ▸ Manage the process supervision using native operating system defaults ▸ RHEL 5.x - SysV ▸ RHEL 6.x - Upstart ▸ RHEL 7.x - systemd ▸ Poise Service framework allows for drop-in replacements with monit, runit and supervisord
16. REDIS COOKBOOK CONFIGURATION RESOURCE ▸ Take advantage of resource's property type hints for basic validations ▸ Chef Client convergence now fails if the type is incorrect ▸ Service isn't restarted or reloaded ▸ Render configuration based on the underlying property data type ▸ Coerce property types into their rendered value (e.g. array of strings)
17. BONUS! HOW WE DEPLOY INFRASTRUCTURE
18. HOW WE DEPLOY INFRASTRUCTURE CHEF POLICYFILES ▸ A workflow for managing infrastructure that replaces the "Environment cookbook" pattern ▸ Locks cookbooks based on a cryptographic checksum instead of semantic version ▸ This means no more yolo breaking infrastructure using 
 knife cookbook upload --force
20. HOW WE DEPLOY INFRASTRUCTURE POLICY REPOSITORY ▸ The repository where a team's knowledge about their infrastructure lives ▸ We generate this from a repository (template) in GitHub ▸ Configuration and scripts for provisioning (e.g. Terraform) ▸ Documentation, run books, deployment information
26. THE END THANKS! ▸ Poise and Poise Service ▸ https://github.com/poise/poise ▸ https://github.com/poise/poise-service ▸ Redis Cookbook ▸ https://github.com/bloomberg/redis-cookbook ▸ NRPE Cookbook ▸ https://github.com/johnbellone/nrpe-ng-cookbook