Pro Yearly is on sale from $80 to $50! »

What is your configuration management doing?

A428c190cba8985990a410b0450b680f?s=47 Bernd Ahlers
April 22, 2015
41

What is your configuration management doing?

How to integrate Chef and Graylog. OSDC 2015

A428c190cba8985990a410b0450b680f?s=128

Bernd Ahlers

April 22, 2015
Tweet

Transcript

  1. Bernd Ahlers – Graylog, Inc. bernd@graylog.com What is your configuration

    management doing? Bernd Ahlers Graylog, Inc.
  2. Bernd Ahlers – Graylog, Inc. bernd@graylog.com • About me •

    Introduction • Tools • Getting events from Chef into Graylog • Demo • QA Agenda
  3. Bernd Ahlers – Graylog, Inc. bernd@graylog.com • Bernd Ahlers •

    Working as engineer at Graylog, Inc. in Hamburg/Germany • bernd@graylog.com • twitter.com/berndahlers • github.com/bernd About me
  4. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Configuration management tools like

    Puppet or Chef enable us to manage our computing resources in an effective and consistent way Introduction
  5. Bernd Ahlers – Graylog, Inc. bernd@graylog.com They make it easy

    to run hundreds or thousands of machines without having to manually execute the same tasks over and over again Introduction
  6. Bernd Ahlers – Graylog, Inc. bernd@graylog.com By using shared modules/cookbooks

    it is pretty easy to end up with hundreds of managed resources like files, packages and services per node Introduction
  7. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Nodes can be configured

    to run at a regular interval to check for updates and to apply new changes automatically Introduction
  8. Bernd Ahlers – Graylog, Inc. bernd@graylog.com This allows us to

    roll out changes to lots of nodes very easily but also makes it possible to quickly break our infrastructure resulting in outages that might affect our business Introduction
  9. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Being able to collect,

    analyze, and monitor all events that happen during a Chef or Puppet run sounds like a good thing to me Introduction
  10. Bernd Ahlers – Graylog, Inc. bernd@graylog.com It would make it

    possible to quickly analyze if an incident has been caused by changes to the configuration management system Introduction
  11. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Having the exceptions thrown

    during a configuration management run available in a central place can help debugging problems Introduction
  12. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Or just graphing the

    duration of Puppet or Chef runs could help finding unusual behavior Introduction
  13. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Do you know what

    your configuration management is doing? Introduction
  14. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Building an example implementation

    with Chef, Icinga2, Graylog Introduction
  15. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Chef (logo) www.getchef.com Tools:

    Chef
  16. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Popular open source configuration

    management tool that uses Ruby to provide a powerful DSL for defining and managing resources Tools: Chef
  17. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Backed by Chef, Inc.

    which has several commercial offerings available Tools: Chef
  18. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Icinga2 (logo) www.icinga2.org Tools:

    Icinga2
  19. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Free and open source

    monitoring tool Use it, it is good Tools: Icinga2
  20. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Graylog (logo) www.graylog.org Tools:

    Graylog
  21. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Free and open source

    tool for centralized log/event management and analysis Tools: Graylog
  22. Bernd Ahlers – Graylog, Inc. bernd@graylog.com • Started by Lennart

    Koopmann in his free time in 2010 (Graylog2 at that time) • TORCH GmbH founded as company behind Graylog in late 2012 • Big rewrite that got released as 0.20 in Feb 2014 Tools: Graylog
  23. Bernd Ahlers – Graylog, Inc. bernd@graylog.com • New US based

    company Graylog, Inc. founded in Jan 2015 • Renamed from Graylog2 to Graylog • Graylog 1.0 release in Feb 2015 Tools: Graylog
  24. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Getting events from Chef into Graylog
  25. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Building a consumer for the internal event bus More than 60 event callbacks available
  26. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog run_start run_started run_completed run_failed ohai_completed skipping_registration registration_start registration_completed registration_failed node_load_start node_load_failed run_list_expand_failed node_load_completed cookbook_resolution_start cookbook_resolution_failed cookbook_resolution_complete cookbook_clean_start removed_cookbook_file cookbook_clean_complete cookbook_sync_start synchronized_cookbook updated_cookbook_file cookbook_sync_failed cookbook_sync_complete library_load_start library_file_loaded library_file_load_failed library_load_complete lwrp_load_start lwrp_file_loaded lwrp_file_load_failed lwrp_load_complete attribute_load_start attribute_file_loaded attribute_file_load_failed attribute_load_complete definition_load_start definition_file_loaded definition_file_load_failed definition_load_complete recipe_load_start recipe_file_loaded recipe_file_load_failed recipe_not_found recipe_load_complete converge_start converge_complete resource_action_start resource_failed_retriable resource_failed resource_skipped resource_completed resource_current_state_loaded resource_current_state_load_bypassed resource_bypassed resource_up_to_date resource_update_applied resource_updated stream_opened stream_closed stream_output handlers_start handler_executed handlers_completed provider_requirement_failed whyrun_assumption msg
  27. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Output show during a Chef run is also based on that
  28. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog (paste snippet of chef-client output here)
  29. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Consuming Chef events requires a Start Handler that listens for events
  30. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Implement all the callbacks?
  31. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Too noisy – for now only Success Failure Resource modification
  32. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Sending Chef events as GELF messages over HTTP
  33. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog GELF JSON based format to send structured events to a Graylog server
  34. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Callbacks have access to lots of details about the Chef run Will be added to the events
  35. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog (add GELF JSON for success)
  36. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog (add GELF JSON for failure)
  37. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog (add GELF JSON for resource updated)
  38. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog GELF HTTP input in Graylog to receive events via HTTP
  39. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Using HTTP makes it easy to implement the same for other configuration management systems
  40. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Install gem and configure Start Handler in chef-client configuration file /etc/chef/client.rb
  41. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog begin require 'chef/handler/graylog/gelf_start_handler' # IP address and port of the GELF HTTP input on # your Graylog server. graylog_server_url = 'http://10.0.2.2:12201/gelf' options = {} start_handlers << Chef::Handler::Graylog::GelfStartHandler.new( graylog_server_url, options) rescue Object => e Chef::Log.error( "Loading Graylog start handler failed: #{e.message}") end
  42. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Getting events from Chef

    into Graylog Chef now sends events as GELF over HTTP to Graylog
  43. Bernd Ahlers – Graylog, Inc. bernd@graylog.com DEMO DEMO

  44. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Finish Source code for

    the event handler and the demo setup available on GitHub https://github.com/Graylog2/chef-handler-graylog https://github.com/Graylog2/osdc-2015-demo Contributions welcome!
  45. Bernd Ahlers – Graylog, Inc. bernd@graylog.com Thank you! Thank you

    for your attention!
  46. Bernd Ahlers – Graylog, Inc. bernd@graylog.com QA QA Ask me

    anything! bernd@graylog.com @berndahlers