DODDFW2017 - Recipe for DevOps Success, Capital One Style

We will take you through the challenges and successes of Capital One’s DevOps Transformation. We are going to talk about the culture changes that have been necessary and how we have adapted our processes and tools to support this culture change.

This talk will go through how we organizationally changed how we did work and how we adapted to a devops model. We will talk about where we have failed and have had to regroup and also what fundamental process and tools. This will include how we enabled an automated deployment model while maintaining change controls and how we rallied around fully automated deployments.

Transcript

1. Recipe for DevOps Success: Capital One Style
2. • Tell you about Capital One’s DevOps journey • Describe the successes and challenges • Give you at least 1 actionable item to take away Goals
3. • Mark Andersen (mark.andersen2@capitalone.com) • Alma Mater: University of Illinois – Urbana Champaign • At Capital One for last 2 years • Director of Devops for Capital One Auto Finance • Capital One cloud account lead for Plano Who am I?
4. • More than just credit cards (Auto Lending, Home Lending, Retail Banking, Commercial Banking, etc.) • Startup in bank years • All in on devops • All in on the public cloud • Open sourcing some of our own products (Hygieia / Cloud Custodian) Who is Capital One?
6. • Inconsistent releases • Many handoffs between many teams for even the smallest change • Manual steps documented in spreadsheet • Environments took many months to setup and had considerable drift Capital One Life Before DevOps
7. The Capital One DevOps Transformation Phase 1 On Prem Use of Configuration Management Automation Tools (Chef) Automation of Building out Middleware and Application Software Phase 2 Cloud Journey – Full Infrastructure Automation into Delivery Pipelines Current Implementing Robust Pipelines with Increased Quality Checks and Automation Pre-Approved Releases
8. • Started with 2 legacy applications. Got a focused SWAT Team together in 1 room for 2 months. • SWAT Team had developers, infrastructure, and production support. Forced cross team / cross responsibility collaboration. • Added a few professional services resources to address knowledge gaps. • Automated the application build, infrastructure deployment, and application deployment. • Tools: Jenkins, AWS CloudFormation, Chef • Moved on to address 4 more applications of increasing complexity. • Completed an additional 4 applications before having the application teams drive themselves. (retired the SWAT Team) The Capital One DevOps Transformation
9. Automation Really Does Work and Provides Value • When testing our first application, we had a bad performance test. Changed a property to make the servers 1 size bigger. Ran the deployment job. 20 minutes later we were resized and had a successful test. SWAT Team Cross- Functional Collaboration Provided Huge Dividends • Removed many handoffs between teams • Developed shared goals for deliverables – More “We” instead of “They” • People learned other roles and learned to respect the complexity – i.e. App Teams creating infrastructure automation Speed of Delivery Increased • Less waiting on other teams • Faster feedback loop for teams • Test and Fail What worked?
10. Tried to Automate Everything all at once • Created long pipelines with long deploy cycles. Also required too many new problems to be solved at the same time. Automation handoff to application team had issues • As automation was returned back to applications team, they didn’t have some of the training / knowledge to grow it and support it. Tools were setup to support non-devops environment • Tools needed to be updated to support devops processes and availability needs. What didn’t work well
11. 10. Don’t just have the devops engineer know the automation. DevOps engineers need to teach the POD how things work and how to fix things. • The goal is for the developers to be self sufficient. • The more the developers understand, the more then can fix their own issues without help and waiting on someone. • Treat the automation like a car. Teach the application team to drive it, change the oil, change the windshield fluid, etc. Not how to build the engine. 9. Prepare yourself for on prem / legacy software to be challenging to automate. You will have to be creative. • Cluster discovery tends to be tougher • Consider the software you are trying to automate and deploy Top 10 Ingredients for DevOps Success
12. 8. Don’t try to do everything at once. Focus on one or two thing first. • If you try to do CI / CD / Environments / Cloud all at once, it will be too much. • Smaller batches (where have I heard that?) • Smaller chunks means you get to celebrate more. Do it • Celebrate like Dude Perfect Top 10 Ingredients for DevOps Success (cont.)
13. 7. If you are picking (or building) tools, make sure they have API’s. • API’s are the enabler for devops (for cloud too) • Even if it has a bad API, it is better than no API. • Be API first for your tools / scripts. Build your UI after that. 6. Speed matters. Long feedback loops are bad feedback loops. • When you automate everything the first time, you’ll find some things take a really long time. • Look to parallelize everything (especially functional tests) • If you pipeline takes 1 hour to complete, are you going to wait for it for feedback? Top 10 Ingredients for DevOps Success (cont.)
14. 5. Unit testing is still needed. • Don’t just focus on functional tests. They tend to be really slow and brittle. • Unit tests are fast and stable. • Consider mocking out your functional dependencies for speed and stability. 4. Networking and security is hard. It is even harder for developers. Training and tools are needed. • Centralize your log collection for everything (applications and infrastructure). Let everyone see them. Create dashboards in “developer” language for the information they need. Top 10 Ingredients for DevOps Success (cont.)
15. 3. Avoid creating a Devops silo to replace your infrastructure silo. 2. Focus on removing / reducing the handoffs. Create self service for things you can’t give developers direct access to. • If you can’t give developers access to create / modify them (LDAP groups, Security Groups, etc.), give them a tool to do it the right way Top 10 Ingredients for DevOps Success (cont.)
16. 1. Don’t let great get in the way of being very good. • Deliver something, get feedback, do some more. • If you are trying to wait until it is perfect, you’ll never deliver. • Unicorns don’t really exist (or they are tough to find). Strive to be a horse. Top 10 Ingredients for DevOps Success (cont.)
17. Thank You
18. Appendix
19. Main site Open Source Project Site = https://developer.capitalone.com/ Hygieia (DevOps visualization tool) = http://www.capitalone.io/Hygieia/getting_started.html Cloud Custodian (cloud management) = https://developer.capitalone.com/opensource- projects/cloud-custodian/ My email = mark.andersen2@capitalone.com Capital One Open Source Projects