Monitoring at SugarCRM - Build vs Buy

Talk given at DevOps Minneapolis group March 16, 2015. Talked about our journey from #monitoringsucks to #monitoringlove.

Transcript

1. © SugarCRM Inc. All rights reserved. 06/08/ 2015 MONITORING AT SUGARCRM! DevOps Minneapolis, March 16, 2015
2. © SugarCRM Inc. All rights reserved. 06/08/ 2015 About Me! •  Ops @ SugarCRM •  Lived in MN for ~1.5 years •  Occasional taker of photos •  Maker of beef jerky
3. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Overview! •  Build vs Buy •  Tools and Techniques •  Reflections and What’s Next
4. © SugarCRM Inc. All rights reserved. 06/08/ 2015 About SugarCRM! •  Founded in 2004, headquarters in Cupertino, CA •  400+ employees worldwide •  9 member Ops staff – all remote •  CRM == “Customer Relationship Management” •  Highly customizable CRM •  Diverse technology stack
5. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Tech Stack!
6. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Related Context! •  Running on several platforms –  Dedicated hardware –  Xen –  Rackspace Cloud –  AWS •  “Growing up” startup •  Migration out of physical datacenter to AWS •  Main product re-architect underway
7. © SugarCRM Inc. All rights reserved. 06/08/ 2015 BUILD VS BUY!
8. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Where We Started! •  Late 2013, old solution no longer cutting it –  API breakage with every upgrade –  Dashboards difficult to build and inflexible –  Building new plugins was painful –  Integration with configuration management was difficult, things got missed
9. © SugarCRM Inc. All rights reserved. 06/08/ 2015 #monitoringsucks
10. © SugarCRM Inc. All rights reserved. 06/08/ 2015 SaaS Solutions! •  Started here •  Most were young or undergoing massive changes •  Issues with agent deployment •  Many were platform specific •  None supported monitoring of network devices •  Required us to build plugins
11. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Self-hosted Commercial Solutions! •  None we found addressed original issues •  No budget for new hardware (and software) •  Either complicated or difficult to extend •  Required us to build plugins
12. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Build Using Open Source Components! •  Needed to architect for high availability •  More services and servers to maintain •  Required us to build plugins
13. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Realizations! All paths lead to.. •  building plugins •  implementing and / or buying multiple systems •  many hours spent building dashboards / reports
14. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Two Questions! We’re going to be build something, do we want to pay someone for the privilege? How much time will it really take to build and maintain the server-side components?
15. © SugarCRM Inc. All rights reserved. 06/08/ 2015 What to Build?! •  Graphite stood out immediately •  Collectd seemed like a reasonable choice •  Nagios vs Sensu
16. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Proof of Concept! Collectd: •  Simple to build •  Lots of builtin plugins •  Python and curl_json plugins Graphite: •  Steep learning curve •  Really powerful •  Metrics + simple events API Sensu: •  Pre-built packages •  Official Puppet module •  Loads of community plugins
17. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Expanded Proof of Concept! •  Graphite split into cluster –  Using ELB + carbon_relay –  Use Carbonate! (https://github.com/jssjr/carbonate) –  Add Grafana for dashboards –  zomg SSDs •  Sensu –  RabbitMQ clustering behind ELB –  RabbitMQ Federation to each datacenter –  Check dependencies reduced noise
18. © SugarCRM Inc. All rights reserved. 06/08/ 2015 TOOLS AND TECHNIQUES!
19. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Playbooks (aka runbooks, etc)! •  Section in wiki page for a given service •  Describes the check –  What –  Where –  Why –  Who –  Optional: ‘Normal’ values
20. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Click-to-Create Tickets! •  Automatic ticket creation didn’t quite work •  We’re using JIRA for incident tracking, click-to- create link in every email •  Tracking of issues much better
21. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Graphite Event Handler! •  Monitoring the monitor! •  Track oncall events over time •  Opted to do this at Sensu level
22. © SugarCRM Inc. All rights reserved. 06/08/ 2015 No-Handler Checks! •  Sensu checks which trigger no action •  Used for systems which already provide notification •  Events show up in dashboard •  Dashboard becomes central place for events
23. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Apdex! •  Reduces performance metrics to single value •  Useful in communicating customer satisfaction with other departments Apdext = (Satisfied Count + Tolerating Count / 2) / Total Samples
24. © SugarCRM Inc. All rights reserved. 06/08/ 2015 REFLECTIONS AND FUTURE!
25. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Reflections! •  Initial buildout of backend systems was hard •  Ongoing maintenance, not so bad •  Scaling out Graphite is a pain –  Carbonate makes it tolerable •  Smaller components make upgrades easier •  On the other hand, always upgrading •  Integration between tools is poor, but improving
26. © SugarCRM Inc. All rights reserved. 06/08/ 2015 What’s Next?! •  Push application metrics •  Client-side app monitoring (again, build vs buy)
27. © SugarCRM Inc. All rights reserved. 06/08/ 2015 Thank You!! @wyattwalter wwalter@sugarcrm.com p.s. we’re hiring!
28. © SugarCRM Inc. All rights reserved. 06/08/ 2015