Development Operations: Take Back Your Infrastructure

Development Operations (DevOps) brings to infrastructure what Agile has been doing for almost a decade prior with software. There lies a disconnect in many working environments between system administration and software development teams. On the system administration side, little has been changing in terms of process and methodologies employed, while software developers are breaking the mold and finding better efficiency and maintainability of their code-bases. DevOps provides the system administrator with a set of methodologies that can be implemented to take back their infrastructure. Come learn how a Linux system administrator can use a diverse set of tools and process to not only create and maintain better infrastructure, but also close the gap with their software developer counter-parts.

Transcript

1. Development Operations Take Back Your Infrastructure Mark Stanislav <mark.stanislav@gmail.com>
2. Me Senior Linux Administrator for a managed services provider in Michigan Admin experience in start-up, small business, corporate, and education environments Previously a professional web programmer Dev + Ops = Why we are here! ;)
3. The Definition of DevOps “A set of processes, methods and systems for communication, collaboration and integration between departments for Development (Applications/Software Engineering), Technology Operations and Quality Assurance (QA).” -Wikipedia, “DevOps” ...okay, but what is DevOps?
4. DevOps From The Community “So, the Devops movement is characterized by people with a multidisciplinary skill set - people who are comfortable with infrastructure and configuration, but also happy to roll up their sleeves, write tests, debug, and ship features.” - Stephen Nelson-Smith, agilesysadmin.net “DevOps helps to enable IT alignment by aligning development and operations roles and processes in the context of shared business objectives. Both development and operations need to understand that they are part of a unified business process.” -Damon Edwards, dev2ops.org “DevOps is all about trying to avoid that epic failure and working smarter and more efficiently at the same time. It is a framework of ideas and principles designed to foster cooperation, learning and coordination between development and operational groups.” -James Turnbull, kartar.net
5. “If culture isn't the first part of explaining DevOps, you've missed the point.” - John Willis, DTO Solutions
6. This is not DevOps culture.
7. ...nor is this...
8. ...and DevOps culture is certainly not this!
9. DevOps Culture is About... Working in operations because you care about the efficacy of infrastructure and success of your team Considering your ability to have a greater impact on an organization beyond your own job title Communication that has no concern with cubicle walls, geography, or corporate hierarchy Operations and development working more efficiently with less push & pull for agreement Understanding that your job is more than a job, and that your work more than a mere checklist
10. Developers vs. Operations Developers: “Hi Ops, Ruby just rolled a really important new release and we’d like to start testing it in an identical, new staging environment this week.” Operations: “Okay, well we understand the desire to do so, but we can’t just roll out a staging environment that quickly and certainly not one with full parity to the existing one.” Developers: “Hi Ops, we know that you guys have 13,000 servers in five data centers on three continents, but if you could delete /etc/foobar on servers with Apache 2.2.4 only, that would be great! Oh, and in the next 30 minutes or so?” Operations: “Is this April 1st or something? No.”
11. The Agile Manifesto Close, daily co-operation between business people and developers Projects are built around motivated individuals, who should be trusted Working software is delivered frequently Sustainable development, able to maintain a constant pace Regular adaptation to changing circumstances “...software development methodologies based on iterative and incremental development...”
12. What Developers Have Frameworks allowing for consistency & rapid development Versioned repositories of code with the ability to revert Accountability for code that is committed to a project Unit testing of code to watch for regression issues Tiered development environments (dev, test, review, live) Files that adhere to a logical hierarchy (MVC folder structure) Agile practices for team interaction and transparency Organizational software to keep track of project goals/tasks Integrated Development Environments (IDE)
13. Physical Infrastructure Good Bad
14. Physical Infrastructure, cont. Easier server replacement in the event of a system failure Ability to track down a damaged patch cable faster Less distraction from seeing link/error lights on devices Mitigated risk of having cables accidentally unplugged New/potential employees to be excited to come to work Upgrading hardware/components is less of a hassle A lasting satisfaction for all involved that we “did things right” Better Physical Infrastructure Allows...
15. What “Old” Operations Has Lots of BASH/PERL scripts for automation & deployment SSH for-loops (e.g. for i in `cat serverlist.txt`; do ping $i; done) Tarballs of directories for configuration management File comparisons (diff -uNr) to check for inconsistencies Hand editing of monitoring configuration files for new servers “If that servers dies, everything is going to fall apart...” “If this administrator leaves, everything is going to fall apart...” =(
16. Knowing all of this...We need to do better. Configuration management Consistent task automation Portability of environments Revert ‘bad’ infrastructure Accountability for changes An IDE to build infrastructure Behavior Driven Development Auto-generation of monitoring THE ABILITY TO BE AGILE!
17. Configuration Management Infrastructure as Code (Puppet) file { “/etc/resolv.conf”: owner => “root”, mode => “0644”; } package { “openssh”: ensure => latest; } service { “named”: ensure => running, enable => true; } tidy { "/tmp": age => "1w", matches => *.temp"; }
18. Consistent Task Automation Marrionette-Collective (mCollective) redacted RPC-based framework for server orchestration and/or parallel jobs Publish-Subscribe model using ActiveMQ/RabbitMQ for queues Leverages meta-data (ohai, facter, custom files, etc.) to limit scope
19. Portability of Environments a.k.a. The Promise of Cloud Computing Cloud-vendor portability using Puppet, Chef, or otherwise Disaster recovery warm-site with full system parity Easy duplication of environments (e.g. new staging) Treat your infrastructure (non-cloud) as ephemeral Reduction in downtime for failed servers/services “The cloud means nothing unless you can quickly deploy, configure, maintain, and destroy hosts without intervention.” - Me, right now.
20. Version Controlled Infrastructure Infrastructure as code begs to be controlled like other code Accountability [“Who committed this change & when?”] Reversion [“Revert to version 302 of the configuration!”] Differentials [“Which change caused this issue?”] Environments [“Let’s test this change in staging, first.”] Branches [“I’ve got a few new features to push I tested.”] History [“Let’s review what we did this year...”] Developers can submit pull requests to infrastructure!
21. Gepetto: An IDE for Puppet
22. Cucumber-Puppet, BDD Feature: General catalog policy In order to ensure applicability of a host's catalog As a manifest developer I want all catalogs to obey some general rules Scenario Outline: Compile and verify catalog Given a node specified by "features/yaml/<hostname>.example.com.yaml" When I compile its catalog Then compilation should succeed And all resource dependencies should resolve Examples: | hostname | | localhost | Isn’t this awesome? * Yanked from: http://projects.puppetlabs.com/projects/cucumber-puppet/wiki
23. Monitoring Auto-Generation New hosts automatically receive: Nagios service + host checks custom to services running Munin plugins configured for services/network interfaces Removed hosts automatically: Delete all Munin/Nagios checks to clean-up Automagicness!
24. Other Cool DevOpsy Stuff... “The Foreman”: http://theforeman.org Web interface for managing/interacting w/ Puppet hosts Provision servers with Kickstart, Jumpstart and Preseed “Rundeck”: http://rundeck.org/ Distributed command execution via SSH w/ workflows Role-based authentication w/ AD & LDAP support “Vagrant”: http://vagrantup.com/ Automated virtual machine creation using VirtualBox Provisioning of virtual environments using Chef or Puppet
25. The Ability to be Agile... Quickly & consistently deploy replicas of existing environments Identify & revert issues in deployed infrastructure with ease Execute commands in parallel to hosts within a concise scope Develop tests to ensure infrastructure is meeting needs Automatically deploy monitoring of hosts/services easily
26. Developers vs. Operations, Redux Developers: “Hi Ops, Ruby just rolled a really important new release and we’d like to start testing it in an identical, new staging environment this week.” Operations: “Sure! Would you prefer these hosts to be staged in EC2, Rackspace Cloud, or locally with Vagrant? Would you like us to deploy your current codebase as well?” Developers: “Hi Ops, we know that you guys have 13,000 servers in five data centers on three continents, but if you could delete /etc/foobar on servers with Apache 2.2.4 only, that would be great! Oh, and in the next 30 minutes or so?” Operations: “How about the next 3 minutes or so?” # mc-filemgr -W “apache=2.2.4” --file /etc/foobar remove
27. But it does take time & effort! Acceptance that the current methods have room to improve A team commitment to openly engage new practices + tools Learning Puppet DSL or Ruby Planning for implementation Time allocation to deploy new infrastructure and test! Potential overhead costs for infrastructure or service fees
28. Thank You! Questions? mark.stanislav@gmail.com @markstanislav http://www.uncompiled.com/ https://github.com/mstanislav/