Efficient monitoring with open source tools (Time series databases)

Ozgur Web Gunleri 2016

Transcript

1. Effecient monitoring with Open source tools Osman Ungur, github.com/o
2. Who i am? • software developer with system-administration background over 10 years • mostly writes Java and PHP • also working about infrastructure design, system automation, deployment and monitoring • obsessed about clean, well structured, maintainable and scalable architectures. • loves open source github.com/o
3. My career path • in 2002, i started to learn fundamentals of Linux network and security. After that, for years i sold and managed dedicated servers and shared web hosting • after the Linux administration story, in 2005 dived into PHP and learned principles of object-oriented-programming • in 2010, i'd started a company which is uses Java, Spring Framework and SOA architecture. Ported thousands of line PHP code to Java and experienced with very large traffic. Slowly i'd embraced Java, NoSQL, RESTful and micro-services architectures • Since August 2015, i'm working as a freelance consultant, trainer and developer. I'm an active contributor and author of open-source projects.
5. Today • Why i need? • Best practices • Time-series databases • Agents • Dashboards • Alerting
6. What is going on?
7. • What is your application doing right now? • Do you will be notifed when a server fails?
8. How about fixing things?
9. • Fixing problems is difficult without logs and monitoring • Sleep better by automation and monitoring
10. Customers and Boss
11. • Don't tolerate software errors • Everyone hates "500 Server Error" • Don't like slow websites
12. Loss of • productivity • money • reputation • time • customer • trust
13. What kind of problems? What to monitor?!?!
14. • Can the users hit my page? • What is %95th page load time? • Is our revenue increased? • What are mostly occured exceptions in last hour?
15. • I didn't change the code, something wonky? • Which part of system is unaccesible? • Do i need to scale up / down my servers? • Is my servers works over capacity?
16. • Is (rdbms|mq|cache) running healthy? • What are (mem|cpu|disk|io) usage of servers? • Is (app server|web server|lb) is up? • Current (bandwidth|network) usage comparing with last weeks?
17. If something fails?
19. You need to get it up and running ASAP
20. Our objective is reducing • time to detect • time to repair
21. Time series databases
22. A time series database (TSDB) is a software system that is optimized for handling time series data, arrays of numbers indexed by time (a datetime or a datetime range). In some fields these time series are called profiles, curves, or traces. A time series of stock prices might be called a price curve. A time series of energy consumption might be called a load profile. A log of temperature values over time might be called a temperature trace. Wikipedia
23. RRDTool • Round robin database tool (File based) • Successor of MRTG • Used by Nagios, Munin, Cacti, pfSense, Ganglia • Storing and graphing capability • Outdated data model, only command line interface
24. Graphite • Whisper database library (File based) • Very popular, simple to operate • Tons of tools that work with graphite • Comes with dashboard, nice functions • Outdated data model, doesn't scale
25. InfluxDB • Time Structured Merge Tree (TSM) • Easy to operate, highly customisable • Also supports events • Good performance, InfluxQL • Clustering removed from open source edition
26. Prometheus • Local file per time series • Pull based metric collectors, PromQL • Easy to operate, good data model • Effecient storage, good performance • Also supports alerting
27. OpenTSDB • Hadoop backed • Scales very well, moderate performance • JSON over HTTP • One of the first databases to use metric lables in its data model • Painful to operate
28. RiakTS • Riak backed • Very easy to operate • Moderate performance • Highly resilient • Good data model, querying like SQL
29. DalmatinerDB • Riak backed • Very high performance • Clustering and fault tolerance • Works with ZFS, Postgres • Limited client support
30. KairosDB • Cassandra storage • Fast writes • Good data model • Ineffecient storage • Slow to query
31. Blueflood • Cassandra storage • Good performance • Highly scalable • Outdated data model • Metric processing system behind Rackspace Metrics
32. Others • Druid • Netflix Atlas • Chronix Server
33. Complex monitoring suites • Nagios • Sensu • Ganglia
34. Log based • Graylog • ELK • Splunk
35. Agents • Collectd • Diamond • Metrics
36. Alerting • Riemann • Seyren • Icinga
37. Questions? github.com/o