Enterprise DevOps: Fast Is As Slow As We Go

http://t.co/noQ4b0fbsN

Presented at ChefConf 2014.

Explore the trials and tribulations of introducing Chef, and DevOps into a well-oiled enterprise machine. This talk will dissect the result of fast business growth, the organization of separate technology stacks, and the problems with management of complex systems. Finally, we will dovetail into how a tiger team of seven works with development, systems, and operations to solve this spiderweb of changes with a little spice.

Transcript

1. ENTERPRISE DEVOPS FAST IS AS SLOW AS WE GO
2. THIS PRESENTATION • Bloomberg • Differential Diagnosis • Introducing DevOps • Web Operations • Battle Stations johnbellone
3. THIS GUY • Technical Lead of Bloomberg Web Operations • New York City / Washington D.C. • Learned C to write code for MUDs johnbellone
4. BLOOMBERG • THE WELL-OILED MACHINE • CORPORATE CULTURE • TECHNOLOGY CROSS-SECTION • WEB PROPERTIES
5. THE WELL-OILED MACHINE • Financial software, data and information delivered over television, radio, internet and terminal • >3,000 engineers primarily located in New York, New Jersey and London • R&D responsible for software and hardware for all aspects of business johnbellone
6. CORPORATE CULTURE • >20% of employees are part of R&D • Efficiency and high availability is our namesake • Time to market is valued very high • Over thirty years of operations results in many homegrown technical solutions johnbellone
7. TECHNOLOGY CROSS-SECTION • Operating systems: Solaris, AIX, RHEL, Windows • CPU architectures: x86_64, POWER, SPARC • Private cloud infrastructure: VMware, OpenStack johnbellone
8. WEB PROPERTIES • Consumer and subscription web verticals • Homegrown service-oriented architecture for data services from “Core” • Applications and services written in Ruby, Node, Java, C++ johnbellone
9. DIFFERENTIAL DIAGNOSIS • BRITTLE FOUNDATION • SUPPORT DIFFICULTIES • SERVER PROCUREMENT • FEAR OF THE UNKNOWN http://commons.wikimedia.org/wiki/File:Magnifying_glass_01.svg
10. BRITTLE FOUNDATION • Core infrastructure dependencies do not have dedicated teams for management • Usage of infrastructure assets is often unknown until business unit changes use case or support • Significant business risk for infrastructure if downtime occurs johnbellone
11. SUPPORT DIFFICULTIES • Non-standardized application infrastructure • Multiple solutions for similar problems • Deadlines often usurp standard procedures johnbellone
12. SERVER PROCUREMENT • Virtual machines treated as physical machines • Provisioning utilizes policies that are not relevant to web technologies • Self-service tools available but poorly understood • Network policies require back and forth between firewall operations johnbellone
13. FEAR OF THE UNKNOWN • “Don’t poke the sleeping bear.” • Policies instituted are often out of place or not relevant for web application teams • Development workflow is much different than “Core” side of the company johnbellone
14. INTRODUCING DEVOPS • CULTURE • CONTINUOUS DELIVERY • CONTINUAL IMPROVEMENT • SHARING AND CARING http://www.jennyjones.com/myblog/wp-content/uploads/2010/09/071309_whoop_21.jpg
15. CULTURE • Stress communication and trust amongst application, operations and infrastructure • Work together towards the ultimate goal of getting quality products out the door • Identify problems and discuss ways to improve johnbellone
16. CONTINUOUS DELIVERY • Wrapper cookbooks building out deployment pipelines quickly and easily • Automatic subscriptions to GitHub Enterprise projects for notification of test failure and successes • Artifact repository for successful builds to allow for quick deployment in upper environments johnbellone
17. CONTINUAL IMPROVEMENT • Measuring infrastructure and build metrics and making them visible • Continuously training support network based on deliverable functionality • Feedback from our customers on how to improve johnbellone
18. SHARING AND CARING • Promote sharing of ideas both internally and externally • Trust that teams and policies are put in place for good reason • Document and make available reasoning for decisions johnbellone
19. TEAM DUCT TAPE • POLYGLOT ENGINEERS • SECURITY MASTERMIND • PLATFORM CZAR • INFRASTRUCTURE NINJAS http://phoosh.deviantart.com/art/Duct-Tape-Texture-109343067
20. POLYGLOT ENGINEERS • Technical lead responsible for backlog • Security engineer to oversee policy and architecture • Platform engineer with deep knowledge in corporate integration and orchestration layer • Infrastructure engineers familiar Ruby and application development inefficiencies johnbellone
21. SECURITY MASTERMIND • Responsible for overseeing security policy decisions for web infrastructure • Intimate knowledge of existing process • Model existing web application infrastructure using Chef johnbellone
22. PLATFORM CZAR • Responsible for the provisioning and orchestration on cloud infrastructure • Knowledge with existing web application platform • Work on platform services for self-service tooling johnbellone
23. INFRASTRUCTURE NINJAS • Previous experience as application developers • Knowledge of pain points and inefficiencies • Responsible for building “glue” for application infrastructure and enterprise integration johnbellone
24. BATTLE STATIONS • LOW HANGING FRUIT • BREAKING BAD HABITS • FAST BUSINESS GROWTH • RAPID TECHNOLOGY PIVOTS • DOCUMENTATION http://i1.sndcdn.com/artworks-000051141856-zppe05-original.jpg?3eddc42
25. LOW HANGING FRUIT • Reusable patterns for building packages and infrastructure • Solve immediate inefficiencies for maximum gain • Streamline legacy support with automation • Ownership of a standard workstation environment johnbellone
26. BREAKING BAD HABITS • GitHub Enterprise enforces pull-request model for code reviews • Standardize application software stacks using wrapper cookbooks • Applications are no longer unique snowflakes • Support “chain of command” for escalation johnbellone
27. FAST BUSINESS GROWTH • Self-service panel for launching feature branches from GitHub • Consolidate patterns and infrastructure • Provide “best practices” for provisioning and orchestrating virtual machines johnbellone
28. RAPID TECHNOLOGY PIVOTS • Omnibus “full-stack” installers for Ruby, Node, Java, etc. • Internal mirrors of necessary build dependencies • VMware, VirtualBox and OpenStack disk images built with Packer • Self-service environment to launch infrastructure johnbellone
29. DOCUMENTATION • GitHub Wiki and Pages available for all projects • Setup instructions for Vagrant, OpenStack and other DevOps utilities • Steps for configuration of desktop workstation for on boarding new employees johnbellone
30. BREEDING SUPERHEROS • SUPPORT NETWORK • FIRST LINE OF DEFENSE • LEVELING UP • SHARING http://www.eschrade.com/wp-content/uploads/2011/12/superhero-white-bkg.png
31. SUPPORT NETWORK • Identified by technical contributions to the discovery process • Representatives are responsible for answering infrastructure questions • Escalation to an area “Champion” which bulldozes barriers for getting the solution johnbellone
32. FIRST LINE OF DEFENSE • Technical leads have most experience with application problems • Full scope of infrastructure deployments and knowledge of their affected areas • Stops questions from going directly to operations and infrastructure teams johnbellone
33. LEVELING UP • Training sessions focused on infrastructure built for web application teams • Feedback on technologies and services that are still a work-in-progress • Creating a community around common infrastructure amongst web verticals johnbellone
34. SHARING • All infrastructure code available through GitHub Enterprise • Pull-requests accepted from any member in web organizations • Ultimate goal of providing the most holistic solution for problems johnbellone
35. SO LONG… AND THANKS FOR ALL THE FISH! http://www.ozsticker.com/312-662-thickbox/dont-panic-thumb.jpg johnbellone