Ops at Emcien

Lessons learned in 18 months doing DevOps at a small software shop. Presented at the January 2014 meeting of the DevOpsATL meetup group.

Transcript

1. Ops at Emcien Lessons learned in 18 months doing DevOps at a small software shop
2. About Me Grew up on a Commodore 64 1998: Learned Perl to write a web forum application Founded a neighborhood news site for Downtown L.A. that grew into a weekly print paper While working at Southern California Public Radio, learned Node.js to write a new audio server for the station’s live streams Moved to Atlanta and joined Emcien in June of 2012
3. Meet Emcien A typical, small software shop with servers and apps in the cloud
4. About Emcien We create software for automated pattern discovery, helping businesses find value in their data Founded in 2002 ~20 people; six in Engineering Four main products, mostly Ruby on Rails around an engine written in C
5. What We Had Two dozen or so EC2 instances, all built up by hand or by homegrown attempts at automation No consistent method for managing services No centralized reporting for server and app health All Rails environment and deployment configuration stored inside the app repositories Environments that all assumed they controlled the entire server
6. What We Wanted Bulletproof deployments Better monitoring and visibility into operations Elastic architecture to scale up and down with load … and make it all cheaper
7. What Didn’t We Have? Dedicated sysadmins DBAs Any ability to slow down development and devote resources to infrastructure
8. How Do You Start? Dream big: Think about the system you really want Start small: Identify pain points and prioritize Look at the business case Find tools that fit with your existing technology stack Don’t be afraid to write your own glue Just get started
9. Emcien’s First Baby Steps Step One: Repeatability
10. Focus on Pain Points No repeatable way to build new environments No way for non-developers to deploy Deployments were an adventure Did we restart Delayed Jobs? Did we install Phantom JS?
11. Baby Steps: Chef Start to experiment with Chef Build a base AMI off of Ubuntu 12.04, with versions of Ruby and nginx/Passenger built and stashed Chef cookbooks to take that base AMI and set it up to run our apps Server gets a key, uses it to look up a databag item that says what environments should be built out
12. Baby Steps: “hubot deploy” Hubot script to check out repository and run Capistrano deployment commands Hardcoded apps and environments Fully built repo / bundle required for deploy
13. Progress! Chef cookbooks and our pre-built AMI make sure that our servers get built out the same every time Non-developers could now deploy to dev and testing environments via Hubot Deployments are more reliable
14. Phase One: Six Months First ideas to new process for production servers
15. What did we learn? Versions you bake onto an AMI go out of date quickly Forking your Chef cookbooks starts you down a path toward heartache Management loves reliable deployments
16. The Plateau First DevOps Level-Up complete. Now what?
17. What Now? Refer back to your wish list. What needs to be updated from your new experience? Assess your pain points. What are the little things you can clean up? Use the goodwill from the first phase to get buy-in for a little bit more this time around
18. Pain Points: Configuration Configuration is two places: in Chef databags and in the app repository Adding a new environment means creating commits in the repo, which makes the log dirty UI for manipulating environments in a databag is no fun
19. Pain Points: Server Roles Each server’s Chef run list had to explicitly include the apps that we wanted to run Initial AMI / recipes had multiple Ruby versions installed, but could only use one at a time Phusion Passenger limitation lifted with 4.0
20. Pain Points: Visibility Asking “what code is deployed on server X?” was still a manual check, so no at-a-glance view of status When values are scattered, it’s easy to get out-of-sync Management and the services group wants a dashboard for what code is deployed in production
21. Bringing It All Together Don’t be afraid to write your own glue code
22. Common Themes Need a system to manage configurations and state What environments should be deployed on each server? What code and configuration should be deployed on each environment? Need a common application config file format
23. Selling the Business Case First pitched at a November 2012 “Lunch and Learn”: One-stop-shop for metrics about how Emcien’s servers and applications are performing Making provisioning environments a simpler (and eventually self-service) task frees up engineering time Throw-away environments allow for better testing and better development cycles
24. Building MCP Centralized configuration management for Emcien’s applications and servers
25. For good, not evil Work with others, not against them
26. Use the Existing Foundation Configuration Management: Chef Use the parts you like (cookbooks and recipes), but replace the parts you don’t (data bags, lack of reporting in Open Source server) Deployment: Capistrano Build cap configs on-the-fly, leverage existing knowledge and deploy tasks
27. One Step at a Time December, 2012: Got started on MCP as part of an effort to move off of Amazon’s RDS Needed a system to administer database backups Running our own database servers was going to save money, so I had a business benefit to use as a starting point
28. One Step at a Time March, 2013: Built endpoints for distributing SSH public keys to servers March, 2013: Start to build out concepts of App and Environment July, 2013: First deployments via MCP
29. Today: Emcien’s Dashboard MCP is in charge of: Environment configuration Deployment Database Backups Chef Cookbook Uploads Chef Run Logging CI Server
30. Iterate Chef Cookbooks Query app / environment information from MCP, not databag Report back as environments are built Keep track of what environments we see
31. Problem: Hardcoded != Agile Creating a new environment still involves editing 4-5 files inside the repository
32. Solution: MasterConfig Centralize configuration for a better quality of life
33. Take Advantage of DevOps Tools and apps need to meet in the middle MCP: Build a tool around our existing apps MasterConfig: Alter the apps to make life easier for operations
34. One Config File == Agile Load environment, database and New Relic config from a single JSON file New environments no longer mean changes in the repository
35. Final Product: Provisioning
36. How is it Working? Wins from having MCP in production
37. Visibility into Past Deploys Being able to see what revisions were deployed and when, as well as all deploy output, is a win
38. Pre-Deployment Views Easy to see what commits will be deployed, and what needs to happen
39. Easy Environment Setup One home for configuration makes management simple
40. Chat integration is still cool A passive window into operations is a great way to keep tabs on the big picture
41. Big Picture Lessons Don’t try to boil the ocean. Identify your pain points and limit the scope of your solutions to what you can do successfully. Don’t be afraid to mix your tools. There’s nothing wrong with picking and choosing the parts you like. Find the right mix for your company and your team. Look at the business case. Find spots where DevOps can create value. Demonstrate little wins to get more buy-in for later projects. Find ways to get other groups excited about what you’re building.