Managing Zabbix Hosts & Templates with Puppet (Configuration Management)

This presentation will walk through a setup that uses the Puppet Config Management tool to use custom types and providers, to call the Zabbix API and add new hosts to be monitored and apply the correct Zabbix Templates.

Transcript

1. MANAGING ZABBIX WITH PUPPET EXPORTED RESOURCES IN PUPPET AND API'S IN ZABBIX @mattiasgeniar
2. WHAT'S THIS TALK ABOUT? A quick intro into Zabbix An equally quick intro into Puppet Exported resources Providers and types in Puppet Zabbix API Ruby code Q & A
3. WHO AM I? Mattias Geniar System Engineer / Support Manager / ... @ Nucleus.be Wrote MoZBX and gave talk at ZabbixConf in Riga in 2012 Blogger at http://ma.ttias.be WHO IS NUCLEUS? Uptime-as-a-Service Hosting, hosting & hosting Many servers, many automations
4. WHO ARE YOU? Who uses Zabbix? Who uses Puppet? Who has written their own types/providers in Puppet? Who knows exported resources?
5. QUICK ZABBIX INTRODUCTION Monitoring system WebUI = PHP + database (MySQL/PostgreSQL/...) Server-side components: pollers, syncers, escalators, alerters, ... Server collects data, stored in RDBMS, viewed by WebUI
6. DATA GATHERING Done by Zabbix Host or Proxies Passive vs. Active gathering (get vs. push data) External checks (no agent required --> is port open?) Agent checks: CPU load, diskspace, custom parameters, ... (like Nagios NRPE) Agent = very extendable (our configs = 75% custom parameters) UserParameter = key.name, your command here UserParameter = varnish.cache_hits, varnishstat -1 | grep '^cache_hit' | head -n 1 | awk '{print $2}'
7. ZABBIX HAS AN API Who doesn't? Read + Write to API "Simple" JSON-RPC Multiple library wrappers available (Ruby, PHP, ...)
8. ZABBIX IN A NUTSHELL You create an "item" on a "host" (ie: cpu load): system.cpu.load[ , avg1] You write a trigger (ie: if load > 5 for 10 minutes in a row) {hostname : system.cpu.load [ , avg1 ].min ( 600 ) } > 5
9. ZABBIX IN A NUTSHELL (PART 2) You define an action to be taken (ie: wake up poor server guy) (actions don't have a cool syntax, just a GUI) Graphs and screens for displaying data You can make templates from items + trigger combinations
10. PUPPET IN A NUTSHELL Declarative configuration management You tell puppet "what" to do, not "how" to do it Manages Files, Services, Packes, ... anything you want Easy to learn, hard to master Client - Server model (standalone mostly for testing)
11. A TYPICAL PUPPET RUN Client starts the Puppet Agent run Puppet Server validates certificate, grants or denies the run Client downloads latest list of plugins from master Agent collects "facts" about itself (key/value pairs) $ facter -p is_virtual => true kernel => Linux ... fqdn => ... Sends the facts to the Puppet Master Puppet Master uses facts to generate the server config (catalog) Master sends catalog to the client Client receives catalog and performs desired configuration (no logic) After client run, it sends a report back to the server
12. TYPES AND PROVIDERS Types are the "what" Providers are the "how" Providers are mostly invisible to the Puppet-user Behind the scenes stuff
13. TYPES AND PROVIDERS EXAMPLES The P a c k a g e resource is a type There are multiple providers for each OS (yum, apt, ...) You can write them yourself: ruby code In our case: 1 "zabbix" type (visible), 1 "zabbix" provider (invisible)
14. RESOURCES VS. EXPORTED RESOURCES Usually, resources are applied "locally" f i l e { " / t m p / t e s t " : e n s u r e = > p r e s e n t , } They can be exported @ @ f i l e { " / t m p / t e s t " : e n s u r e = > p r e s e n t , } And collected on other hosts F i l e < < | | > >
15. A MORE MEANINGFUL EXPORTED RESOURCE EXAMPLE We have a z a b b i x _ m o n i t o r _ h o s t resource Each host exports their Zabbix Configuration @ @ z a b b i x _ m o n i t o r _ h o s t { $ : : f q d n : e n s u r e = > p r e s e n t , h o s t = > $ : : f q d n , i p = > $ : : i p a d d r e s s , h o s t n a m e = > $ : : f q d n , } And collected on our monitoring host Z a b b i x _ m o n i t o r _ h o s t < < | | > > The monitoring host executes those resources (in our case: API calls)
16. EACH MODULE GETS THEIR OWN EXPORTED RESOURCE We have modules for Nginx, Apache, PHP, Postfix, ... c l a s s n g i n x { p a c k a g e { ' n g i n x ' : e n s u r e = > i n s t a l l e d , } }
17. Each module makes sure their monitoring template is applied @ @ z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y { $ : : f q d n : e n s u r e = > p r e s e n t , t e m p l a t e = > ' N g i n x N g i n x ' , h o s t = > $ : : f q d n , r e q u i r e = > Z a b b i x _ m o n i t o r _ h o s t [ $ : : f q d n ] , } Monitoring host applies template Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y < < | | > >
18. PEANUT BUTTER DEMO TIME (PLEASE PLEASE PLEASE PLEASE WORK)
19. LIVE DEMO: BACKUP SCREENSHOTS (1) We start with a clean Zabbix install: no hosts in the inventory, just pre-configged templates
20. LIVE DEMO: BACKUP SCREENSHOTS (2) Each host starts exporting their Zabbix Configurations - host1.tld: nginx, mysql, ... - host2.tld: varnish, postfix, ... (there's not really something to show here, this happens behinds the scenes) Our monitoring host collects those exported resources, and starts applying them Example for "rdns1.nucleus.be" N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ h o s t [ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d b e p r e s e n t ( n o o p ) N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y [ z b x _ a p p l i c a t i o n _ c h e c k s u m s _ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d b e p r e s e n t ( n o o p ) N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y [ z b x _ a p p l i c a t i o n _ f w d _ r e v e r s e _ d n s _ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d b e p r e s e n t ( n o o p ) N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y [ z b x _ a p p l i c a t i o n _ g e n e r a l _ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d
21. LIVE DEMO: BACKUP SCREENSHOTS (3) After a (long) time, more & more hosts are added
22. LIVE DEMO: BACKUP SCREENSHOTS (4) Each with all (puppet-managed) services monitored
23. THE CODE Forked from https://github.com/vadv/zabbixapi which was then moved to https://github.com/express42/zabbixapi My version at https://github.com/mattiasgeniar/zabbixapi You're better of implementing the express42 examples/code, I think Thanks to the groundwork, extensions are easy to make Requires the z a b b i x a p i and i p a d d r e s s ruby gem
24. LESSONS LEARNED API calls trigger audit logs in Zabbix (+100GB mysql tables, oops) z a b b i x _ * are bad type names, should have been m o n i t o r _ * (type vs. provider) Each Puppet run = 64k API calls 300 hosts in Zabbix = 60 minutes to run Puppet Agent for initial import Notice: Finished catalog run in 3996.18 seconds then 20 minutes for each following run Notice: Finished catalog run in 1065.66 seconds
25. Q & A
26. THANK YOU! Twitter: @mattiasgeniar Blog: http://ma.ttias.be (zabbix, puppet, open source, php, ... ) Looking for colleagues @ https://www.nucleus.be ;-)