Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOps as Software Development

F43919144cdcddd7ba50e46f71667d08?s=47 Tyler L
November 02, 2015

DevOps as Software Development

Exploring how software engineering practices can be applied to operations workflows.


Tyler L

November 02, 2015


  1. DevOps as Software Development DevOpsDays Charlotte 2015 Tyler Langlois

  2. INTRODUCTIONS • Infrastructure Engineer/DevOps at Elastic • Ex- Qualtrics/Blue Coat/Sandia

    National Labs • Practices fatigue-driven (Dev|Hug)Ops • Advocate of fine Shia Labeouf gifs
  3. DISCLAIMER • Some of this is in practice, some is

    theory • I Am Not a Thought Leader/Ops Ninja, just a concerned citizen • There are many states of ops-fu; this is a brainstorming session
  4. OVERVIEW • Why? • Infrastructure as Code • Adding Software

    Development Practices • Future Possibilities
  5. WHY? Short answer: because “...okay, what version of the app

    do we need to deploy? Great, yeah, let me spin up a machine really quick. Running puppet… hold on, I’m getting a puppet exec timeout, let me get into the code really quickly and fix something. Okay, things are looking good. Wait, what version of $library do you need? Okay, one sec, let me append $package to the list of packages. Uh, hold on, the other machines have a dependency resolution error. Give me some time, there’s a duplicate resource declaration. No, I have to do that one, the IAM setup doesn’t permit access to route 53, which subdomain do you need? That’s on the old AWS account, give me one sec to add it to the wiki, we still need some automation around provisioning route 53 entries. It looks like there are some errors getting at it from in front of the ELB, are you sure the app is running? Yeah, let me double check to see whether there are any errors in the logs. Okay, it looks like there are some permissions problems, we’ll need to change ownership of $dir for the app, let me take a look at puppet. Okay, the daemon starts up, I get a 200 OK, does the app look good from your side? Which part? Let me check the environment variables. Okay, yeah, it looks like puppet interpolated an empty string, the app is missing the token, let me add some validation and re-run puppet. Does that look okay now? Okay, great.” Can only take you so far before you reach Peak Sysadmin™
  6. WHY? Long answer: • Codifying operations as development tasks brings

    tremendous benefits: Version Control • Repeatable! • Audited! • Documented!
  7. WHY? Long answer: • Codifying operations as development tasks brings

    tremendous benefits: Collaboration • Controlled changes • Clearly defined process • Visibility
  8. WHY? Long answer: • Codifying operations as development tasks brings

    tremendous benefits: LIMITLESS POSSIBILITIES You’re only constrained by what you can code • Infrastructure testing • Automated scaling and provisioning

  10. TERMS • Solving operational problems with software engineering • Not

    necessarily a synonym for system administration (within this context) • Operational tasks, including system administration, is the subject of engineering for automation DEVOPS
  11. TERMS ( ) DEVOPS

  12. • Reliable - redundancy and stability TERMS Enabling reliable and

    happy workflows for our developers That ^ entails a lot: OPERATIONS • Enabling - make the tooling easy • Workflows - DevOps as Software Development

  14. CODE As long as you’re codifying infrastructure, this makes the

    DevOps happy
  15. CODE A good VCS is essential to reap software development

    benefits Use what makes sense, but it should be usable and collaborative
  16. CODE A good VCS is essential to reap software development

    benefits Use what makes sense, but it should be usable and collaborative
  17. CODE Extensibility is important (you’ re a developer solving operational

    problems with code, right?) Use something that makes sense within the context of the organization
  18. CODE The Developer + Operations Contract (i.e., how to devs

    express their needs?) Make sure it’s practical: • Can developers iterate without blocking? • Are artifacts easily versioned (immutable, even)? • Minimal app/infra crossover? • Don’t ignore provenance

  20. TESTING I. Make change to nginx rules II. ??? III.


  22. TESTING • There are excellent frameworks for this now! ◦

    Bats, test-kitchen, etc. ◦ Hook up to existing CI ◦ Limitations of course, but lots of possibilities ... LEADS TO • Defined tickets requirements through failing tests? • Large plumbing switches without fear?
  23. INFRA CHANGES AS PULL REQUESTS • Changing packages on systems

    • Adding users to hosts • Deploying ssh keys • DNS Changes • Creating/destroying hosts • Deploying apps/containers • Capacity scaling • Tickets can become failing tests <3 • CI/CD workflow for system images (packer, etc.) OR, STAGES OF DEVOPS SOFTWARE ENGINEERING EVOLUTION

  25. (BONUS: CHATOPS) • Easy to subscribe to infrastructure activity •

    Chat bots are now pretty mature • VCS actions/info • PaaS activity (RSS?) • Self-serve ops • Outage/panic-mode service • Automate The Boring Parts™
  26. • OS-level declarative configuration ◦ NixOS and guix are good

    examples • More potential for cattle vs. pets ◦ Datacenter as a resource becoming more and more standard (k8s, mesos) FUTURE WORK • Type-safe infrastructure ◦ Internal project around functional config management
  27. Thank you! • github.com/tylerjl • irc/twitter: leothrix • tjll.net