What is your configuration management doing?

Bernd Ahlers – Graylog, Inc. [email protected] What is your configuration
management doing? Bernd Ahlers Graylog, Inc.

Bernd Ahlers – Graylog, Inc. [email protected] • About me •
Introduction • Tools • Getting events from Chef into Graylog • Demo • QA Agenda

Bernd Ahlers – Graylog, Inc. [email protected] • Bernd Ahlers •
Working as engineer at Graylog, Inc. in Hamburg/Germany • [email protected] • twitter.com/berndahlers • github.com/bernd About me

Bernd Ahlers – Graylog, Inc. [email protected] Configuration management tools like
Puppet or Chef enable us to manage our computing resources in an effective and consistent way Introduction

Bernd Ahlers – Graylog, Inc. [email protected] They make it easy
to run hundreds or thousands of machines without having to manually execute the same tasks over and over again Introduction

Bernd Ahlers – Graylog, Inc. [email protected] By using shared modules/cookbooks
it is pretty easy to end up with hundreds of managed resources like files, packages and services per node Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Nodes can be configured
to run at a regular interval to check for updates and to apply new changes automatically Introduction

Bernd Ahlers – Graylog, Inc. [email protected] This allows us to
roll out changes to lots of nodes very easily but also makes it possible to quickly break our infrastructure resulting in outages that might affect our business Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Being able to collect,
analyze, and monitor all events that happen during a Chef or Puppet run sounds like a good thing to me Introduction

Bernd Ahlers – Graylog, Inc. [email protected] It would make it
possible to quickly analyze if an incident has been caused by changes to the configuration management system Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Having the exceptions thrown
during a configuration management run available in a central place can help debugging problems Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Or just graphing the
duration of Puppet or Chef runs could help finding unusual behavior Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Do you know what
your configuration management is doing? Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Building an example implementation
with Chef, Icinga2, Graylog Introduction

Bernd Ahlers – Graylog, Inc. [email protected] Chef (logo) www.getchef.com Tools:
Chef

Bernd Ahlers – Graylog, Inc. [email protected] Popular open source configuration
management tool that uses Ruby to provide a powerful DSL for defining and managing resources Tools: Chef

Bernd Ahlers – Graylog, Inc. [email protected] Backed by Chef, Inc.
which has several commercial offerings available Tools: Chef

Bernd Ahlers – Graylog, Inc. [email protected] Icinga2 (logo) www.icinga2.org Tools:
Icinga2

Bernd Ahlers – Graylog, Inc. [email protected] Free and open source
monitoring tool Use it, it is good Tools: Icinga2

Bernd Ahlers – Graylog, Inc. [email protected] Graylog (logo) www.graylog.org Tools:
Graylog

Bernd Ahlers – Graylog, Inc. [email protected] Free and open source
tool for centralized log/event management and analysis Tools: Graylog

Bernd Ahlers – Graylog, Inc. [email protected] • Started by Lennart
Koopmann in his free time in 2010 (Graylog2 at that time) • TORCH GmbH founded as company behind Graylog in late 2012 • Big rewrite that got released as 0.20 in Feb 2014 Tools: Graylog

Bernd Ahlers – Graylog, Inc. [email protected] • New US based
company Graylog, Inc. founded in Jan 2015 • Renamed from Graylog2 to Graylog • Graylog 1.0 release in Feb 2015 Tools: Graylog

Bernd Ahlers – Graylog, Inc. [email protected] Getting events from Chef
into Graylog Getting events from Chef into Graylog

into Graylog Building a consumer for the internal event bus More than 60 event callbacks available

into Graylog run_start run_started run_completed run_failed ohai_completed skipping_registration registration_start registration_completed registration_failed node_load_start node_load_failed run_list_expand_failed node_load_completed cookbook_resolution_start cookbook_resolution_failed cookbook_resolution_complete cookbook_clean_start removed_cookbook_file cookbook_clean_complete cookbook_sync_start synchronized_cookbook updated_cookbook_file cookbook_sync_failed cookbook_sync_complete library_load_start library_file_loaded library_file_load_failed library_load_complete lwrp_load_start lwrp_file_loaded lwrp_file_load_failed lwrp_load_complete attribute_load_start attribute_file_loaded attribute_file_load_failed attribute_load_complete definition_load_start definition_file_loaded definition_file_load_failed definition_load_complete recipe_load_start recipe_file_loaded recipe_file_load_failed recipe_not_found recipe_load_complete converge_start converge_complete resource_action_start resource_failed_retriable resource_failed resource_skipped resource_completed resource_current_state_loaded resource_current_state_load_bypassed resource_bypassed resource_up_to_date resource_update_applied resource_updated stream_opened stream_closed stream_output handlers_start handler_executed handlers_completed provider_requirement_failed whyrun_assumption msg

into Graylog Output show during a Chef run is also based on that

into Graylog (paste snippet of chef-client output here)

into Graylog Consuming Chef events requires a Start Handler that listens for events

into Graylog Implement all the callbacks?

into Graylog Too noisy – for now only Success Failure Resource modification

into Graylog Sending Chef events as GELF messages over HTTP

into Graylog GELF JSON based format to send structured events to a Graylog server

into Graylog Callbacks have access to lots of details about the Chef run Will be added to the events

into Graylog (add GELF JSON for success)

into Graylog (add GELF JSON for failure)

into Graylog (add GELF JSON for resource updated)

into Graylog GELF HTTP input in Graylog to receive events via HTTP

into Graylog Using HTTP makes it easy to implement the same for other configuration management systems

into Graylog Install gem and configure Start Handler in chef-client configuration file /etc/chef/client.rb

into Graylog begin require 'chef/handler/graylog/gelf_start_handler' # IP address and port of the GELF HTTP input on # your Graylog server. graylog_server_url = 'http://10.0.2.2:12201/gelf' options = {} start_handlers << Chef::Handler::Graylog::GelfStartHandler.new( graylog_server_url, options) rescue Object => e Chef::Log.error( "Loading Graylog start handler failed: #{e.message}") end

into Graylog Chef now sends events as GELF over HTTP to Graylog

Bernd Ahlers – Graylog, Inc. [email protected] DEMO DEMO

Bernd Ahlers – Graylog, Inc. [email protected] Finish Source code for
the event handler and the demo setup available on GitHub https://github.com/Graylog2/chef-handler-graylog https://github.com/Graylog2/osdc-2015-demo Contributions welcome!

Bernd Ahlers – Graylog, Inc. [email protected] Thank you! Thank you
for your attention!

Bernd Ahlers – Graylog, Inc. [email protected] QA QA Ask me
anything! [email protected] @berndahlers

What is your configuration management doing?

What is your configuration management doing?

More Decks by Bernd Ahlers

Featured

Transcript