Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Owning Your (Puppet) Infrastructure

Nic Benders
September 23, 2014

Owning Your (Puppet) Infrastructure

Presented at PuppetConf 2014

Nic Benders

September 23, 2014
Tweet

More Decks by Nic Benders

Other Decks in Technology

Transcript

  1. I work at New Relic. ! This is a story

    about treating your infrastructure the same way you treat your applications. ! This is also a story about how we use New Relic, 
 at New Relic. http://nicbenders.com/presentations/puppetconf-2014
  2. Managing Our Applications • Source Control • Continuous Deployment •

    Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  3. Managing Our Infrastructure • Source Control • Continuous Deployment •

    Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  4. DON’T PANIC • These example are for Puppet Enterprise 3.1,

    but this will work with Puppet Open Source, or with newer versions. You’ll just need to make a few changes here and there. • If you don’t have a New Relic account, you can sign-up for a free “Lite” account. • All of the examples are up in a more useful format at the URL below.
 (The URL will also be shown at the end of the presentation.) http://nicbenders.com/presentations/puppetconf-2014
  5. Instrumenting the Puppet Master • Install newrelic_rpm using the Puppet’s

    Ruby interpreter • Modify the config.ru in the Puppet Master root dir • Create a newrelic.yml in the same directory #  /var/opt/lib/pe-­‐puppetmaster/config.ru   ! #  Setup  the  New  Relic  Ruby  Agent  before  anything  else     require  'rubygems'   require  'newrelic_rpm'   #  END  New  Relic  Ruby  Agent  setup #  /var/opt/lib/pe-­‐puppetmaster/newrelic.yml   ! common:  &default_settings      license_key:  '000-­‐your-­‐license-­‐key-­‐here-­‐000'          app_name:  "Puppet  Master"          log_level:  info      log_file_path:  '/var/log/pe-­‐puppet/'          ssl:  false          capture_params:  true          transaction_tracer:          enabled:  true          error_collector:          enabled:  true          ignore_errors:  "ActionController::RoutingError"       production:      <<:  *default_settings      monitor_mode:  true  
  6. Getting good transaction names • Use set_transaction_name in a Rack

    Middleware to tell the Ruby Agent how we are going to name the “web transactions” for the Puppet Master • Use add_custom_parameters to pass the environment and other useful data along #     #  We  don't  want  every  transaction  to  just  say  "call",  so  define  a     #  Transaction  "namer"  for  the  Ruby  Agent  based  on  the  URL  structure     #     class  CoolNamer   def  initialize(app,  opts  =  {})    @app  =  app   end       def  call(env)    req  =  Rack::Request.new(env)    _,  environment,  kind_of_thing,  resource  =  req.path.split('/',4)    NewRelic::Agent.set_transaction_name(kind_of_thing,        :category  =>  :rack)    NewRelic::Agent.add_custom_parameters(:puppet_env  =>  environment)    NewRelic::Agent.add_custom_parameters(:puppet_resource  =>  resource)    @app.call(env)   end   end   use  CoolNamer   #  END  Transaction  "namer"    
  7. Getting more detail • The Ruby Agent’s MethodTracer allows us

    to get details of what is going on inside a Transaction. • By hooking into Puppet’s built-in Profiler calls, we can quickly instrument all the most important things. • This example is for Puppet 3.3.1. In newer versions it is much easier. require  "puppet/util/profiler"   module  Puppet::Util::Profiler      include  ::NewRelic::Agent::MethodTracer      def  self.profile(message,  &block)          if  /^Processed  request  /  ===  message              #  Top  level  trace,  skip  for  now                yield          else              metric_name  =  self.message_to_metric(message)              trace_execution_scoped([metric_name])  do                  yield            end        end      end          def  self.message_to_metric(message)          message.chomp!          message.gsub!(/^(Compiled  catalog)  for  .*/,  '\1')          message.gsub!(/^(Filtered  result  for  catalog)  .*/,  '\1')          message.gsub!(/\(\d+\)  /,  '')          message.gsub!(/^(Evaluated  resource)  /,  '\1/')          message.gsub!(/\[.*\]/,'')          message.gsub!(/\:\s/,'/')          message.gsub!(/^(Called)  /,  '\1/')          "Custom/Puppet/#{message}"      end   end
  8. Instrumenting the Puppet DB • Download the Java Agent from

    your “Account Settings” page. • Create an /opt/puppet/newrelic dir and place the newrelic.jar and newrelic.yaml file into it. • Modify the newrelic.yaml • Modify JAVA_ARGS #  /opt/puppet/newrelic/newrelic.yml   ! common:  &default_settings      license_key:  '000-­‐your-­‐license-­‐key-­‐here-­‐000'   !    app_name:  'Puppet  DB'          log_level:  info      log_file_path:  /var/log/pe-­‐puppetdb/          ssl:  false          transaction_tracer:          enabled:  true          transaction_threshold:  apdex_f          record_sql:  obfuscated          stack_trace_threshold:  0.2          explain_enabled:  true          explain_threshold:  0.2          browser_monitoring:          auto_instrument:  false         production:      <<:  *default_settings #  /etc/syslog/pe-­‐puppetdb   ! #  Enable  the  New  Relic  Java  Agent  by  adding  this   #  to  the  bottom  of  the  file     JAVA_ARGS="-­‐javaagent:/opt/puppet/newrelic/newrelic.jar  ${JAVA_ARGS}"
  9. Puppet Agent already goes into Syslog • Agent syslogs have

    a ton of useful information. We just need to get it to a useful place. • Our application and syslog data is collected by a Heka process on each server. It then makes its way to ElasticSearch, where it can be queried by Kibana.
  10. Send run reports to the same place • Add your

    log system as a 
 Report processor! • We use https://github.com/ logstash/puppet-logstash-reporter • To get the data into Heka, we had to write some Lua, but it was pretty straightforward.
  11. #  Logstash-­‐style  report  processer  through  Heka      class  {

     'logstash_reporter':          logstash_port  =>  '22253',          logstash_host  =>  hiera('logstash_reporter::logstash_host'),      }   pe_puppet::master::config_hash:      reports:  'http,puppetdb,logstash'   ! logstash_reporter::logstash_host:  'our-­‐heka-­‐server'  
  12. Managing Our Infrastructure • Source Control • Continuous Deployment •

    Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana