Cooking Up Elasticsearch with Chef at elastic{on}

9ad2a5355d8cfa842e24b7a4322b2535?s=47 Martin Smith
February 19, 2016

Cooking Up Elasticsearch with Chef at elastic{on}

It’s not written by Julia Child or Gordon Ramsey, but the elasticsearch cookbook is worth it. Come hear Martin Smith — co-author of the Chef Community Cookbook for Elasticsearch — present a deep dive on automating Elasticsearch installation and configuration with Chef, and re-writing the open source Chef cookbook the last 12 months.

9ad2a5355d8cfa842e24b7a4322b2535?s=128

Martin Smith

February 19, 2016
Tweet

Transcript

  1. 1 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19

    February, 2016 @martinb3 Cooking up Elasticsearch with Chef
  2. 2 About the Chef Ecosystem Multiple workflows, what works best

    for you.
  3. 3 Chef Ecosystem Desired State Centralized Resource Primitives • Idempotency

    • Declarative • Client-server model • Data & secret storage • Management tools • Ruby DSL • Multiple implementations
  4. 4 an operation that will produce the same results if

    executed once or multiple times Idempotency … desired state
  5. 5 5

  6. 6 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do

    action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb
  7. 7 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do

    version node[‘curl_version’] action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb default[‘curl_version’] = ‘1.2.3’ attributes/curl.rb
  8. 8 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do

    action :install end file ‘/etc/motd’ do content ‘Curl is installed’ end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb
  9. 9 Elasticsearch cookbook A short history

  10. 10 Elasticsearch cookbook • 2011 – Initial commits from Karel

    Minarik, ES @ 0.18.5 • 2013 – Capable of installing monit, nginx, EBS via Fog, Discovery • 2014 – Began discussing rewrites • 2015 – GCE support, and the kitchen sink… • 2015 July – released v1.0.0 A short history Source: Gray Arial 10pt
  11. 11

  12. 12 Technical debt Attribute soup No multiple ES instances Unmaintained

    features Lack of testing Problems with the ‘current’ cookbook
  13. 13 - Focus on basic features - Respecting Chef standards

    - Testability Guiding principles for rewrite/refactor - October 2014
  14. 14 Simple, declarative resources Examples for package, service, file package

    ‘curl’ service ’mysql' do action [:enable, :start] end file ’/etc/motd' do content ‘Hello World!’ end
  15. 15 Where we ended up Simple resources with no surprises

    Add appropriate user and group Install, uninstall by package or archive file (tar.gz) Configure elasticsearch.yml, parity with packages Create and start system service Install and remove plugins via bin/plugin
  16. 16 Simple, declarative resources Examples using elasticsearch cookbook elasticsearch_install 'elasticsearch'

    do type :package end elasticsearch_configure 'elasticsearch' do logging(:action => 'INFO') allocated_memory '123m' configuration('node.name' => 'arbitrary_name') end elasticsearch_service 'my_elasticsearch_node' do service_actions [:enable, :start] end
  17. 17 Where we ended up Simple resources with no surprises

    Defaults follow packages, which have evolved Recipe installs most defaults, otherwise use resources Missing the more exotic features (EBS, S3, Discovery) Still cannot handle complex cases with plugins Does support multiple instances
  18. 18 More pure Ruby testing Cleanup logic Versioning Duplicate environment

    vars Where we go from here
  19. 19 FOLLOWED of all Chef cookbooks 27th 1% 70% Elasticsearch

    cookbook TOP COOKBOOKS out of 2719 on Community ‘Supermarket’ CHEF CLIENTS are Fortune 1000 companies
  20. 20 How to get involved Logstash, Kibana, ELK stack Elastic

    stack elastic/cookbook-elasticsearch lusis/chef-logstash lusis/chef-kibana rackspace-cookbooks/elkstack
  21. 21

  22. 22 Option 3 Martin B. Smith, DevOps Engineer Rackspace Hosting,

    Inc. 19 February, 2016 @martinb3 Questions outside by AMA booth
  23. 23 Except where otherwise noted, this work is licensed under

    http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.