DevOps as Software Development DevOpsDays Charlotte 2015 Tyler Langlois 

INTRODUCTIONS ● Infrastructure Engineer/DevOps at Elastic ● Ex- Qualtrics/Blue Coat/Sandia National Labs ● Practices fatigue-driven (Dev|Hug)Ops ● Advocate of fine Shia Labeouf gifs 

DISCLAIMER ● Some of this is in practice, some is theory ● I Am Not a Thought Leader/Ops Ninja, just a concerned citizen ● There are many states of ops-fu; this is a brainstorming session 

OVERVIEW ● Why? ● Infrastructure as Code ● Adding Software Development Practices ● Future Possibilities 

WHY? Short answer: because “...okay, what version of the app do we need to deploy? Great, yeah, let me spin up a machine really quick. Running puppet… hold on, I’m getting a puppet exec timeout, let me get into the code really quickly and fix something. Okay, things are looking good. Wait, what version of $library do you need? Okay, one sec, let me append $package to the list of packages. Uh, hold on, the other machines have a dependency resolution error. Give me some time, there’s a duplicate resource declaration. No, I have to do that one, the IAM setup doesn’t permit access to route 53, which subdomain do you need? That’s on the old AWS account, give me one sec to add it to the wiki, we still need some automation around provisioning route 53 entries. It looks like there are some errors getting at it from in front of the ELB, are you sure the app is running? Yeah, let me double check to see whether there are any errors in the logs. Okay, it looks like there are some permissions problems, we’ll need to change ownership of $dir for the app, let me take a look at puppet. Okay, the daemon starts up, I get a 200 OK, does the app look good from your side? Which part? Let me check the environment variables. Okay, yeah, it looks like puppet interpolated an empty string, the app is missing the token, let me add some validation and re-run puppet. Does that look okay now? Okay, great.” Can only take you so far before you reach Peak Sysadmin™ 

WHY? Long answer: ● Codifying operations as development tasks brings tremendous benefits: Version Control ● Repeatable! ● Audited! ● Documented! 

WHY? Long answer: ● Codifying operations as development tasks brings tremendous benefits: Collaboration ● Controlled changes ● Clearly defined process ● Visibility 

WHY? Long answer: ● Codifying operations as development tasks brings tremendous benefits: LIMITLESS POSSIBILITIES You’re only constrained by what you can code ● Infrastructure testing ● Automated scaling and provisioning 

0. SOME DEFINITIONS, WHY NOT 

TERMS ● Solving operational problems with software engineering ● Not necessarily a synonym for system administration (within this context) ● Operational tasks, including system administration, is the subject of engineering for automation DEVOPS 

TERMS ( ) DEVOPS 

● Reliable - redundancy and stability TERMS Enabling reliable and happy workflows for our developers That ^ entails a lot: OPERATIONS ● Enabling - make the tooling easy ● Workflows - DevOps as Software Development 

I. INFRASTRUCTURE AS CODE 

CODE As long as you’re codifying infrastructure, this makes the DevOps happy 

CODE A good VCS is essential to reap software development benefits Use what makes sense, but it should be usable and collaborative 

CODE A good VCS is essential to reap software development benefits Use what makes sense, but it should be usable and collaborative 

CODE Extensibility is important (you’ re a developer solving operational problems with code, right?) Use something that makes sense within the context of the organization 

CODE The Developer + Operations Contract (i.e., how to devs express their needs?) Make sure it’s practical: ● Can developers iterate without blocking? ● Are artifacts easily versioned (immutable, even)? ● Minimal app/infra crossover? ● Don’t ignore provenance 

II. ADD SOFTWARE ENGINEERING PROCESSES 

TESTING I. Make change to nginx rules II. ??? III. :shipit: 

TESTING 

TESTING ● There are excellent frameworks for this now! ○ Bats, test-kitchen, etc. ○ Hook up to existing CI ○ Limitations of course, but lots of possibilities ... LEADS TO ● Defined tickets requirements through failing tests? ● Large plumbing switches without fear? 

INFRA CHANGES AS PULL REQUESTS ● Changing packages on systems ● Adding users to hosts ● Deploying ssh keys ● DNS Changes ● Creating/destroying hosts ● Deploying apps/containers ● Capacity scaling ● Tickets can become failing tests <3 ● CI/CD workflow for system images (packer, etc.) OR, STAGES OF DEVOPS SOFTWARE ENGINEERING EVOLUTION 

CHANGE TRACKING 

(BONUS: CHATOPS) ● Easy to subscribe to infrastructure activity ● Chat bots are now pretty mature ● VCS actions/info ● PaaS activity (RSS?) ● Self-serve ops ● Outage/panic-mode service ● Automate The Boring Parts™ 

● OS-level declarative configuration ○ NixOS and guix are good examples ● More potential for cattle vs. pets ○ Datacenter as a resource becoming more and more standard (k8s, mesos) FUTURE WORK ● Type-safe infrastructure ○ Internal project around functional config management 

Thank you! ● github.com/tylerjl ● irc/twitter: leothrix ● tjll.net