DISCLAIMER ● Some of this is in practice, some is theory ● I Am Not a Thought Leader/Ops Ninja, just a concerned citizen ● There are many states of ops-fu; this is a brainstorming session
WHY? Short answer: because “...okay, what version of the app do we need to deploy? Great, yeah, let me spin up a machine really quick. Running puppet… hold on, I’m getting a puppet exec timeout, let me get into the code really quickly and fix something. Okay, things are looking good. Wait, what version of $library do you need? Okay, one sec, let me append $package to the list of packages. Uh, hold on, the other machines have a dependency resolution error. Give me some time, there’s a duplicate resource declaration. No, I have to do that one, the IAM setup doesn’t permit access to route 53, which subdomain do you need? That’s on the old AWS account, give me one sec to add it to the wiki, we still need some automation around provisioning route 53 entries. It looks like there are some errors getting at it from in front of the ELB, are you sure the app is running? Yeah, let me double check to see whether there are any errors in the logs. Okay, it looks like there are some permissions problems, we’ll need to change ownership of $dir for the app, let me take a look at puppet. Okay, the daemon starts up, I get a 200 OK, does the app look good from your side? Which part? Let me check the environment variables. Okay, yeah, it looks like puppet interpolated an empty string, the app is missing the token, let me add some validation and re-run puppet. Does that look okay now? Okay, great.” Can only take you so far before you reach Peak Sysadmin™
WHY? Long answer: ● Codifying operations as development tasks brings tremendous benefits: Collaboration ● Controlled changes ● Clearly defined process ● Visibility
WHY? Long answer: ● Codifying operations as development tasks brings tremendous benefits: LIMITLESS POSSIBILITIES You’re only constrained by what you can code ● Infrastructure testing ● Automated scaling and provisioning
TERMS ● Solving operational problems with software engineering ● Not necessarily a synonym for system administration (within this context) ● Operational tasks, including system administration, is the subject of engineering for automation DEVOPS
● Reliable - redundancy and stability TERMS Enabling reliable and happy workflows for our developers That ^ entails a lot: OPERATIONS ● Enabling - make the tooling easy ● Workflows - DevOps as Software Development
CODE Extensibility is important (you’ re a developer solving operational problems with code, right?) Use something that makes sense within the context of the organization
CODE The Developer + Operations Contract (i.e., how to devs express their needs?) Make sure it’s practical: ● Can developers iterate without blocking? ● Are artifacts easily versioned (immutable, even)? ● Minimal app/infra crossover? ● Don’t ignore provenance
TESTING ● There are excellent frameworks for this now! ○ Bats, test-kitchen, etc. ○ Hook up to existing CI ○ Limitations of course, but lots of possibilities ... LEADS TO ● Defined tickets requirements through failing tests? ● Large plumbing switches without fear?
INFRA CHANGES AS PULL REQUESTS ● Changing packages on systems ● Adding users to hosts ● Deploying ssh keys ● DNS Changes ● Creating/destroying hosts ● Deploying apps/containers ● Capacity scaling ● Tickets can become failing tests <3 ● CI/CD workflow for system images (packer, etc.) OR, STAGES OF DEVOPS SOFTWARE ENGINEERING EVOLUTION
● OS-level declarative configuration ○ NixOS and guix are good examples ● More potential for cattle vs. pets ○ Datacenter as a resource becoming more and more standard (k8s, mesos) FUTURE WORK ● Type-safe infrastructure ○ Internal project around functional config management
tylerjl
gonkun
yrhorita
ryomaru0825
linedevth
coconala_engineer
kyonmm
mkwrd
mploed
kenta_hi
nishiuma
yoshiiryo1
lara
jnunemaker
chriscoyier
shpigford
mza
palkan
myddelton
addyosmani
bcantrill
jasonvnalue
searls
zakiyama