Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing Zabbix Hosts & Templates with Puppet (Configuration Management)

Managing Zabbix Hosts & Templates with Puppet (Configuration Management)

This presentation will walk through a setup that uses the Puppet Config Management tool to use custom types and providers, to call the Zabbix API and add new hosts to be monitored and apply the correct Zabbix Templates.

6050b4913a2f5cfb09a1cdb31ea489ed?s=128

Mattias Geniar

December 16, 2014
Tweet

Transcript

  1. MANAGING ZABBIX WITH PUPPET EXPORTED RESOURCES IN PUPPET AND API'S

    IN ZABBIX @mattiasgeniar
  2. WHAT'S THIS TALK ABOUT? A quick intro into Zabbix An

    equally quick intro into Puppet Exported resources Providers and types in Puppet Zabbix API Ruby code Q & A
  3. WHO AM I? Mattias Geniar System Engineer / Support Manager

    / ... @ Nucleus.be Wrote MoZBX and gave talk at ZabbixConf in Riga in 2012 Blogger at http://ma.ttias.be WHO IS NUCLEUS? Uptime-as-a-Service Hosting, hosting & hosting Many servers, many automations
  4. WHO ARE YOU? Who uses Zabbix? Who uses Puppet? Who

    has written their own types/providers in Puppet? Who knows exported resources?
  5. QUICK ZABBIX INTRODUCTION Monitoring system WebUI = PHP + database

    (MySQL/PostgreSQL/...) Server-side components: pollers, syncers, escalators, alerters, ... Server collects data, stored in RDBMS, viewed by WebUI
  6. DATA GATHERING Done by Zabbix Host or Proxies Passive vs.

    Active gathering (get vs. push data) External checks (no agent required --> is port open?) Agent checks: CPU load, diskspace, custom parameters, ... (like Nagios NRPE) Agent = very extendable (our configs = 75% custom parameters) UserParameter = key.name, your command here UserParameter = varnish.cache_hits, varnishstat -1 | grep '^cache_hit' | head -n 1 | awk '{print $2}'
  7. ZABBIX HAS AN API Who doesn't? Read + Write to

    API "Simple" JSON-RPC Multiple library wrappers available (Ruby, PHP, ...)
  8. ZABBIX IN A NUTSHELL You create an "item" on a

    "host" (ie: cpu load): system.cpu.load[ , avg1] You write a trigger (ie: if load > 5 for 10 minutes in a row) {hostname : system.cpu.load [ , avg1 ].min ( 600 ) } > 5
  9. ZABBIX IN A NUTSHELL (PART 2) You define an action

    to be taken (ie: wake up poor server guy) (actions don't have a cool syntax, just a GUI) Graphs and screens for displaying data You can make templates from items + trigger combinations
  10. PUPPET IN A NUTSHELL Declarative configuration management You tell puppet

    "what" to do, not "how" to do it Manages Files, Services, Packes, ... anything you want Easy to learn, hard to master Client - Server model (standalone mostly for testing)
  11. A TYPICAL PUPPET RUN Client starts the Puppet Agent run

    Puppet Server validates certificate, grants or denies the run Client downloads latest list of plugins from master Agent collects "facts" about itself (key/value pairs) $ facter -p is_virtual => true kernel => Linux ... fqdn => ... Sends the facts to the Puppet Master Puppet Master uses facts to generate the server config (catalog) Master sends catalog to the client Client receives catalog and performs desired configuration (no logic) After client run, it sends a report back to the server
  12. TYPES AND PROVIDERS Types are the "what" Providers are the

    "how" Providers are mostly invisible to the Puppet-user Behind the scenes stuff
  13. TYPES AND PROVIDERS EXAMPLES The P a c k a

    g e resource is a type There are multiple providers for each OS (yum, apt, ...) You can write them yourself: ruby code In our case: 1 "zabbix" type (visible), 1 "zabbix" provider (invisible)
  14. RESOURCES VS. EXPORTED RESOURCES Usually, resources are applied "locally" f

    i l e { " / t m p / t e s t " : e n s u r e = > p r e s e n t , } They can be exported @ @ f i l e { " / t m p / t e s t " : e n s u r e = > p r e s e n t , } And collected on other hosts F i l e < < | | > >
  15. A MORE MEANINGFUL EXPORTED RESOURCE EXAMPLE We have a z

    a b b i x _ m o n i t o r _ h o s t resource Each host exports their Zabbix Configuration @ @ z a b b i x _ m o n i t o r _ h o s t { $ : : f q d n : e n s u r e = > p r e s e n t , h o s t = > $ : : f q d n , i p = > $ : : i p a d d r e s s , h o s t n a m e = > $ : : f q d n , } And collected on our monitoring host Z a b b i x _ m o n i t o r _ h o s t < < | | > > The monitoring host executes those resources (in our case: API calls)
  16. EACH MODULE GETS THEIR OWN EXPORTED RESOURCE We have modules

    for Nginx, Apache, PHP, Postfix, ... c l a s s n g i n x { p a c k a g e { ' n g i n x ' : e n s u r e = > i n s t a l l e d , } }
  17. Each module makes sure their monitoring template is applied @

    @ z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y { $ : : f q d n : e n s u r e = > p r e s e n t , t e m p l a t e = > ' N g i n x N g i n x ' , h o s t = > $ : : f q d n , r e q u i r e = > Z a b b i x _ m o n i t o r _ h o s t [ $ : : f q d n ] , } Monitoring host applies template Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y < < | | > >
  18. PEANUT BUTTER DEMO TIME (PLEASE PLEASE PLEASE PLEASE WORK)

  19. LIVE DEMO: BACKUP SCREENSHOTS (1) We start with a clean

    Zabbix install: no hosts in the inventory, just pre-configged templates
  20. LIVE DEMO: BACKUP SCREENSHOTS (2) Each host starts exporting their

    Zabbix Configurations - host1.tld: nginx, mysql, ... - host2.tld: varnish, postfix, ... (there's not really something to show here, this happens behinds the scenes) Our monitoring host collects those exported resources, and starts applying them Example for "rdns1.nucleus.be" N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ h o s t [ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d b e p r e s e n t ( n o o p ) N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y [ z b x _ a p p l i c a t i o n _ c h e c k s u m s _ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d b e p r e s e n t ( n o o p ) N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y [ z b x _ a p p l i c a t i o n _ f w d _ r e v e r s e _ d n s _ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d b e p r e s e n t ( n o o p ) N o t i c e : / S t a g e [ m a i n ] / Z a b b i x _ m o n i t o r / Z a b b i x _ m o n i t o r _ t e m p l a t e _ a p p l y [ z b x _ a p p l i c a t i o n _ g e n e r a l _ r d n s 1 . n u c l e u s . b e ] / e n s u r e : c u r r e n t _ v a l u e a b s e n t , s h o u l d
  21. LIVE DEMO: BACKUP SCREENSHOTS (3) After a (long) time, more

    & more hosts are added
  22. LIVE DEMO: BACKUP SCREENSHOTS (4) Each with all (puppet-managed) services

    monitored
  23. THE CODE Forked from https://github.com/vadv/zabbixapi which was then moved to

    https://github.com/express42/zabbixapi My version at https://github.com/mattiasgeniar/zabbixapi You're better of implementing the express42 examples/code, I think Thanks to the groundwork, extensions are easy to make Requires the z a b b i x a p i and i p a d d r e s s ruby gem
  24. LESSONS LEARNED API calls trigger audit logs in Zabbix (+100GB

    mysql tables, oops) z a b b i x _ * are bad type names, should have been m o n i t o r _ * (type vs. provider) Each Puppet run = 64k API calls 300 hosts in Zabbix = 60 minutes to run Puppet Agent for initial import Notice: Finished catalog run in 3996.18 seconds then 20 minutes for each following run Notice: Finished catalog run in 1065.66 seconds
  25. Q & A

  26. THANK YOU! Twitter: @mattiasgeniar Blog: http://ma.ttias.be (zabbix, puppet, open source,

    php, ... ) Looking for colleagues @ https://www.nucleus.be ;-)