DevOps at CM

Presenting the Monitoring POC at Critical Mass

Transcript

1. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 1 DEVOPS AT Critical Mass JUNE 17t h , 2016
2. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 2 MONITORING POC March 2015 Open-source and Free All pieces coded in Chef Works on various platforms
3. 3 TWO PARALLEL PIPELINES SYSTEM METRICS MONITORING & LOG MONITORING
5. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 5 ACTIONABLE ALERTS
6. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 6 The time-series pipeline consists of several pieces, each of which fulfill a role in outlined in the monitoring system overview. This follows a push model, where the monitoring collection programs will push to the storage server. TIME SERIES DATA
7. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 7 The log aggregation pipeline relies heavily on the Elasticsearch, Logstash, and Kibana stack. This fulfills the collection, storage, and graphing pieces of the monitoring system overview. Checking and alerting will be built atop this stack. LOG DATA
8. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED BUT HOW HARD ARE ALL THESE TO SET UP? 8
9. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED JUST RUN THE INFRASTRUCTURE CODE 9
10. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 10 CHEF
11. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED IT’S ENTIRELY AUTOMATED 11
12. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED Collection - collectd -  filebeat -  statsd -  http_collector Storage -  influxdb -  elasticsearch -  logstash -  sensu_client Visualization -  grafana -  kibana 12 THE CHEF COOKBOOKS ENGINEERING - DEVOPS Checking -  sensu_server -  sensu_client -  sensu_checks -  sensu_handlers
13. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED THE STANDARD PACKAGE without any additional configuration 13
14. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED System Level: CPU Load Memory Disk (utilization & saturation) Network Swap Context Switches Application Level*: Request Count Connection Count Session Count Response Time Response Codes Navigation Timing API Data Thread Count Database Connection Count File Descriptor Counts Business Level*: Key Performance Indicator(s) Other custom metrics as needed 14 DEFAULT TIME SERIES METRICS ENGINEERING - DEVOPS *The pipeline processes this data by default, but your application needs to provide it.
15. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 15
16. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 16
17. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 17 -  System Load -  Disk Usage -  Swap Usage -  Memory Usage -  HTTP Service Check -  Log Alerting DEFAULT ALERTING ENGINEERING - DEVOPS
18. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 18
19. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 19 -  Lots of Dependencies -  Used to take several days to set up. Onboarding was not easy -  Different developers used to have slightly different environments -  Code works on dev’s machine, not on integration server. AEM DEVELOPMENT ENVIRONMENT SETUP
20. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED NOW IT’S ENTIRELY AUTOMATED 20
21. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 21 CHEF
22. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 22 DEMO!
23. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 23 -  It takes time initially to code everything but it pays off in the long run. -  You do not need monitoring intelligence every day, but this will save your time when you need it most -  You need to put effort to spread the use of these new tools. People might not be using it first, but once they know about these, they will love you. -  Pick your automation project carefully, work with the people who are suffering from the lack of it and figure their preferences and style of working LEARNINGS ENGINEERING - DEVOPS
24. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 24 QUESTIONS?
25. © 2015 CRITICAL MASS, INC. ALL RIGHTS RESERVED 25 C M E N G I N E E R I N G THANKS