DevOps as Software Development

Exploring how software engineering practices can be applied to operations workflows.

Transcript

1. DevOps as Software Development DevOpsDays Charlotte 2015 Tyler Langlois
2. INTRODUCTIONS • Infrastructure Engineer/DevOps at Elastic • Ex- Qualtrics/Blue Coat/Sandia National Labs • Practices fatigue-driven (Dev|Hug)Ops • Advocate of fine Shia Labeouf gifs
3. DISCLAIMER • Some of this is in practice, some is theory • I Am Not a Thought Leader/Ops Ninja, just a concerned citizen • There are many states of ops-fu; this is a brainstorming session
4. OVERVIEW • Why? • Infrastructure as Code • Adding Software Development Practices • Future Possibilities
5. WHY? Short answer: because “...okay, what version of the app do we need to deploy? Great, yeah, let me spin up a machine really quick. Running puppet… hold on, I’m getting a puppet exec timeout, let me get into the code really quickly and fix something. Okay, things are looking good. Wait, what version of $library do you need? Okay, one sec, let me append $package to the list of packages. Uh, hold on, the other machines have a dependency resolution error. Give me some time, there’s a duplicate resource declaration. No, I have to do that one, the IAM setup doesn’t permit access to route 53, which subdomain do you need? That’s on the old AWS account, give me one sec to add it to the wiki, we still need some automation around provisioning route 53 entries. It looks like there are some errors getting at it from in front of the ELB, are you sure the app is running? Yeah, let me double check to see whether there are any errors in the logs. Okay, it looks like there are some permissions problems, we’ll need to change ownership of $dir for the app, let me take a look at puppet. Okay, the daemon starts up, I get a 200 OK, does the app look good from your side? Which part? Let me check the environment variables. Okay, yeah, it looks like puppet interpolated an empty string, the app is missing the token, let me add some validation and re-run puppet. Does that look okay now? Okay, great.” Can only take you so far before you reach Peak Sysadmin™
6. WHY? Long answer: • Codifying operations as development tasks brings tremendous benefits: Version Control • Repeatable! • Audited! • Documented!
7. WHY? Long answer: • Codifying operations as development tasks brings tremendous benefits: Collaboration • Controlled changes • Clearly defined process • Visibility
8. WHY? Long answer: • Codifying operations as development tasks brings tremendous benefits: LIMITLESS POSSIBILITIES You’re only constrained by what you can code • Infrastructure testing • Automated scaling and provisioning
9. 0. SOME DEFINITIONS, WHY NOT
10. TERMS • Solving operational problems with software engineering • Not necessarily a synonym for system administration (within this context) • Operational tasks, including system administration, is the subject of engineering for automation DEVOPS
11. TERMS ( ) DEVOPS
12. • Reliable - redundancy and stability TERMS Enabling reliable and happy workflows for our developers That ^ entails a lot: OPERATIONS • Enabling - make the tooling easy • Workflows - DevOps as Software Development
13. I. INFRASTRUCTURE AS CODE
14. CODE As long as you’re codifying infrastructure, this makes the DevOps happy
15. CODE A good VCS is essential to reap software development benefits Use what makes sense, but it should be usable and collaborative
16. CODE A good VCS is essential to reap software development benefits Use what makes sense, but it should be usable and collaborative
17. CODE Extensibility is important (you’ re a developer solving operational problems with code, right?) Use something that makes sense within the context of the organization
18. CODE The Developer + Operations Contract (i.e., how to devs express their needs?) Make sure it’s practical: • Can developers iterate without blocking? • Are artifacts easily versioned (immutable, even)? • Minimal app/infra crossover? • Don’t ignore provenance
19. II. ADD SOFTWARE ENGINEERING PROCESSES
20. TESTING I. Make change to nginx rules II. ??? III. :shipit:
21. TESTING
22. TESTING • There are excellent frameworks for this now! ◦ Bats, test-kitchen, etc. ◦ Hook up to existing CI ◦ Limitations of course, but lots of possibilities ... LEADS TO • Defined tickets requirements through failing tests? • Large plumbing switches without fear?
23. INFRA CHANGES AS PULL REQUESTS • Changing packages on systems • Adding users to hosts • Deploying ssh keys • DNS Changes • Creating/destroying hosts • Deploying apps/containers • Capacity scaling • Tickets can become failing tests <3 • CI/CD workflow for system images (packer, etc.) OR, STAGES OF DEVOPS SOFTWARE ENGINEERING EVOLUTION
24. CHANGE TRACKING
25. (BONUS: CHATOPS) • Easy to subscribe to infrastructure activity • Chat bots are now pretty mature • VCS actions/info • PaaS activity (RSS?) • Self-serve ops • Outage/panic-mode service • Automate The Boring Parts™
26. • OS-level declarative configuration ◦ NixOS and guix are good examples • More potential for cattle vs. pets ◦ Datacenter as a resource becoming more and more standard (k8s, mesos) FUTURE WORK • Type-safe infrastructure ◦ Internal project around functional config management
27. Thank you! • github.com/tylerjl • irc/twitter: leothrix • tjll.net