Monitoring_With_Riemann.pdf

This is presentation for the talk I gave in Clojured Berlin 2019 on Riemann.

Transcript

1. Monitoring With Riemann
2. About (def about-me {:name “Abhishek Anand Amralkar” :shortname “@aaa” :from “Talentica Software Pvt. Ltd” :social { :blog “https://medium.com/@aamralkar” :twitter “https://twitter.com/aamralkar” :github “https://github.com/abhishekamralkar”} })
3. Agenda • What Riemann is? • Why Riemann? • Concepts of Riemann ◦ Indexes ◦ Streams ◦ Events • Configuration walk through
4. Why Monitoring? • To understand the Business risks. • To plan for capacity. • To catch the issue early in infrastructure/Applications. • To compli SLA’s promised to customers. • To know what part of infrastructure/application broken rather customer telling letting us know.
5. Why Monitoring? • Make sure application is stable and performance is under SLA and available across the globe. • Systems are getting complex. Humans needs Sleep.
6. What needs to be Monitored? • Cloud • Infrastructure • Operating Systems • Business Applications
7. How Many Types of Monitoring? • Push • Pull • Customers -- The Good ones who send email/chat to support and tell them “Dear XYZ your application foobar module is not working. Please check”
8. Challenges in Distributed Systems Monitoring? • Multiple distributed machines emitting hundreds of thousands of metrics every second. • What metrics need to be monitored? • What metrics should be alerted? • How frequently alerted?
9. Challenges in Distributed Systems Monitoring? • Storage of useful metrics • Real Time metrics so that we know when the problem occurs. • Real Time alerting if something fails in system. • Sending out metrics to Enterprise Monitoring Solutions is costly. • Informative Dashboards across all environment?
10. What Riemann is? In one word “ Riemann is Event Aggregator” • A monitoring tool which aggregates events from servers and applications running on those servers. • Riemann uses powerful stream processing language written in Clojure to aggregates the events.
11. Why Riemann? • Low latency events processing monitoring engine. • Can process millions events per second. • Events and Events Stream processing, which makes its highly adaptable/ideal for dynamic and Distributed Systems. • Riemann configuration file is Clojure Program. • Riemann is Highly Configurable.
12. Why Riemann? • Full control on your Infrastructure/Application monitoring and alerting. • Can monitor anything. • Comes with its own Instrumentation to measures own performance. • Can send alerts to email, chats and SMS and many more.. • Can get connected to backend time series databases InfluxDB and Graphite to store metrics for historical data.
13. Riemann in our case
14. Riemann Events • Events in Riemann are the base construct. • Riemann receives events and process them • Events are structs (records) which Riemann treats as map. (immutable) • Events fields are referred by Keywords in config like :host, :service, :tags. • Apart from the standard fields, custom fields can also be sent in the event.
15. How Event Looks Like? A Clojure map (immutable for sure)
16. Riemann Streams • Streams are Clojure functions that we can define. • Streams are defined in stream section of the Riemann config file. • Streams can have child stream. • Events get passed to the streams for aggregation, modification and alerting. • Riemann config can have as many streams. • Riemann comes with a Powerful Stream Processing Language.
18. Riemann Indexes • Table of current state of all services tracked by Riemann. • Each event is uniquely indexed by its host and service. The index just keeps track of the most recent event for a given (host, service) pair. • Index can have TTL (time to leave). • Events in Index expires after the TTL.
19. Riemann Config • Below is valid Clojure expression where we are calling logging namespace and init function which takes a map {:key value}.
20. Riemann Config • Clojure let a lexical scope where its taking vector of one or more binding in our case a vector of riemann host, followed by functions call like tcp-server, udp-server
22. In built Filtering Streams • where • match • tagged • tagged-all • expired
23. Rate Events Streams • rollup • throttle
24. Coalesce The coalesce stream remembers the last events from each host and service, and sends them all as a vector to its children. We can map that vector of events to a single event--the one with the largest metric--using folds/maximum. Then we just set the service and host, since this event pertains to the system as a whole.
26. Events Grouping • moving-time-window • moving-event-window • fixed-time-window • fixed-event-window
27. Time for Action • We will demonstrate how to send Zookeeper metrics to Riemann and InfluxDB. • Also the Riemann/Grafana Dashboard for the same.
28. Thank You All!
29. Questions If Any?