Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What is your configuration management doing?

Bernd Ahlers
April 22, 2015
58

What is your configuration management doing?

How to integrate Chef and Graylog. OSDC 2015

Bernd Ahlers

April 22, 2015
Tweet

Transcript

  1. Bernd Ahlers – Graylog, Inc. [email protected] • About me •

    Introduction • Tools • Getting events from Chef into Graylog • Demo • QA Agenda
  2. Bernd Ahlers – Graylog, Inc. [email protected] • Bernd Ahlers •

    Working as engineer at Graylog, Inc. in Hamburg/Germany • [email protected] • twitter.com/berndahlers • github.com/bernd About me
  3. Bernd Ahlers – Graylog, Inc. [email protected] Configuration management tools like

    Puppet or Chef enable us to manage our computing resources in an effective and consistent way Introduction
  4. Bernd Ahlers – Graylog, Inc. [email protected] They make it easy

    to run hundreds or thousands of machines without having to manually execute the same tasks over and over again Introduction
  5. Bernd Ahlers – Graylog, Inc. [email protected] By using shared modules/cookbooks

    it is pretty easy to end up with hundreds of managed resources like files, packages and services per node Introduction
  6. Bernd Ahlers – Graylog, Inc. [email protected] Nodes can be configured

    to run at a regular interval to check for updates and to apply new changes automatically Introduction
  7. Bernd Ahlers – Graylog, Inc. [email protected] This allows us to

    roll out changes to lots of nodes very easily but also makes it possible to quickly break our infrastructure resulting in outages that might affect our business Introduction
  8. Bernd Ahlers – Graylog, Inc. [email protected] Being able to collect,

    analyze, and monitor all events that happen during a Chef or Puppet run sounds like a good thing to me Introduction
  9. Bernd Ahlers – Graylog, Inc. [email protected] It would make it

    possible to quickly analyze if an incident has been caused by changes to the configuration management system Introduction
  10. Bernd Ahlers – Graylog, Inc. [email protected] Having the exceptions thrown

    during a configuration management run available in a central place can help debugging problems Introduction
  11. Bernd Ahlers – Graylog, Inc. [email protected] Or just graphing the

    duration of Puppet or Chef runs could help finding unusual behavior Introduction
  12. Bernd Ahlers – Graylog, Inc. [email protected] Do you know what

    your configuration management is doing? Introduction
  13. Bernd Ahlers – Graylog, Inc. [email protected] Popular open source configuration

    management tool that uses Ruby to provide a powerful DSL for defining and managing resources Tools: Chef
  14. Bernd Ahlers – Graylog, Inc. [email protected] Backed by Chef, Inc.

    which has several commercial offerings available Tools: Chef
  15. Bernd Ahlers – Graylog, Inc. [email protected] Free and open source

    monitoring tool Use it, it is good Tools: Icinga2
  16. Bernd Ahlers – Graylog, Inc. [email protected] Free and open source

    tool for centralized log/event management and analysis Tools: Graylog
  17. Bernd Ahlers – Graylog, Inc. [email protected] • Started by Lennart

    Koopmann in his free time in 2010 (Graylog2 at that time) • TORCH GmbH founded as company behind Graylog in late 2012 • Big rewrite that got released as 0.20 in Feb 2014 Tools: Graylog
  18. Bernd Ahlers – Graylog, Inc. [email protected] • New US based

    company Graylog, Inc. founded in Jan 2015 • Renamed from Graylog2 to Graylog • Graylog 1.0 release in Feb 2015 Tools: Graylog
  19. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Getting events from Chef into Graylog
  20. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Building a consumer for the internal event bus More than 60 event callbacks available
  21. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog run_start run_started run_completed run_failed ohai_completed skipping_registration registration_start registration_completed registration_failed node_load_start node_load_failed run_list_expand_failed node_load_completed cookbook_resolution_start cookbook_resolution_failed cookbook_resolution_complete cookbook_clean_start removed_cookbook_file cookbook_clean_complete cookbook_sync_start synchronized_cookbook updated_cookbook_file cookbook_sync_failed cookbook_sync_complete library_load_start library_file_loaded library_file_load_failed library_load_complete lwrp_load_start lwrp_file_loaded lwrp_file_load_failed lwrp_load_complete attribute_load_start attribute_file_loaded attribute_file_load_failed attribute_load_complete definition_load_start definition_file_loaded definition_file_load_failed definition_load_complete recipe_load_start recipe_file_loaded recipe_file_load_failed recipe_not_found recipe_load_complete converge_start converge_complete resource_action_start resource_failed_retriable resource_failed resource_skipped resource_completed resource_current_state_loaded resource_current_state_load_bypassed resource_bypassed resource_up_to_date resource_update_applied resource_updated stream_opened stream_closed stream_output handlers_start handler_executed handlers_completed provider_requirement_failed whyrun_assumption msg
  22. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Output show during a Chef run is also based on that
  23. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog (paste snippet of chef-client output here)
  24. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Consuming Chef events requires a Start Handler that listens for events
  25. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Too noisy – for now only Success Failure Resource modification
  26. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Sending Chef events as GELF messages over HTTP
  27. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog GELF JSON based format to send structured events to a Graylog server
  28. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Callbacks have access to lots of details about the Chef run Will be added to the events
  29. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog (add GELF JSON for resource updated)
  30. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog GELF HTTP input in Graylog to receive events via HTTP
  31. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Using HTTP makes it easy to implement the same for other configuration management systems
  32. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Install gem and configure Start Handler in chef-client configuration file /etc/chef/client.rb
  33. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog begin require 'chef/handler/graylog/gelf_start_handler' # IP address and port of the GELF HTTP input on # your Graylog server. graylog_server_url = 'http://10.0.2.2:12201/gelf' options = {} start_handlers << Chef::Handler::Graylog::GelfStartHandler.new( graylog_server_url, options) rescue Object => e Chef::Log.error( "Loading Graylog start handler failed: #{e.message}") end
  34. Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef

    into Graylog Chef now sends events as GELF over HTTP to Graylog
  35. Bernd Ahlers – Graylog, Inc. [email protected] Finish Source code for

    the event handler and the demo setup available on GitHub https://github.com/Graylog2/chef-handler-graylog https://github.com/Graylog2/osdc-2015-demo Contributions welcome!