Cooking Up Elasticsearch with Chef at elastic{on}

It’s not written by Julia Child or Gordon Ramsey, but the elasticsearch cookbook is worth it. Come hear Martin Smith — co-author of the Chef Community Cookbook for Elasticsearch — present a deep dive on automating Elasticsearch installation and configuration with Chef, and re-writing the open source Chef cookbook the last 12 months.

Transcript

1. 1
   Martin B. Smith, DevOps Engineer
   Rackspace Hosting, Inc.
   19 February, 2016
   @martinb3
   Cooking up
   Elasticsearch with Chef
   

   View Slide

2. 2
   About the Chef
   Ecosystem
   Multiple workflows, what
   works best for you.
   

   View Slide

3. 3
   Chef Ecosystem
   Desired State Centralized Resource
   Primitives
   • Idempotency
   • Declarative
   • Client-server model
   • Data & secret storage
   • Management tools
   • Ruby DSL
   • Multiple implementations
   

   View Slide

4. 4
   an operation that will produce
   the same results if executed
   once or multiple times
   Idempotency
   … desired state
   

   View Slide

5. View Slide

6. 6
   Recipes and attributes
   Examples from ‘mycookbook’
   package ‘curl’ do
   action :install
   end
   include_recipe ‘mycookbook::curl’
   recipes/curl.rb
   recipes/default.rb
   

   View Slide

7. 7
   Recipes and attributes
   Examples from ‘mycookbook’
   package ‘curl’ do
   version node[‘curl_version’]
   action :install
   end
   include_recipe ‘mycookbook::curl’
   recipes/curl.rb
   recipes/default.rb
   default[‘curl_version’] = ‘1.2.3’
   attributes/curl.rb
   

   View Slide

8. 8
   Recipes and attributes
   Examples from ‘mycookbook’
   package ‘curl’ do
   action :install
   end
   file ‘/etc/motd’ do
   content ‘Curl is installed’
   end
   include_recipe ‘mycookbook::curl’
   recipes/curl.rb
   recipes/default.rb
   

   View Slide

9. 9
   Elasticsearch
   cookbook
   A short history
   

   View Slide

10. 10
    Elasticsearch cookbook
    • 2011 – Initial commits from Karel Minarik, ES @ 0.18.5
    • 2013 – Capable of installing monit, nginx, EBS via Fog, Discovery
    • 2014 – Began discussing rewrites
    • 2015 – GCE support, and the kitchen sink…
    • 2015 July – released v1.0.0
    A short history
    Source: Gray Arial 10pt
    

    View Slide

11. View Slide

12. 12
    Technical debt
    Attribute soup
    No multiple ES instances
    Unmaintained features
    Lack of testing
    Problems with the ‘current’ cookbook
    

    View Slide

13. 13
    - Focus on basic features
    - Respecting Chef standards
    - Testability
    Guiding principles for rewrite/refactor
    - October 2014
    

    View Slide

14. 14
    Simple, declarative resources
    Examples for package, service, file
    package ‘curl’
    service ’mysql' do
    action [:enable, :start]
    end
    file ’/etc/motd' do
    content ‘Hello World!’
    end
    

    View Slide

15. 15
    Where we ended up
    Simple resources with no surprises
    Add appropriate user and group
    Install, uninstall by package or archive file (tar.gz)
    Configure elasticsearch.yml, parity with packages
    Create and start system service
    Install and remove plugins via bin/plugin
    

    View Slide

16. 16
    Simple, declarative resources
    Examples using elasticsearch cookbook
    elasticsearch_install 'elasticsearch' do
    type :package
    end
    elasticsearch_configure 'elasticsearch' do
    logging(:action => 'INFO')
    allocated_memory '123m'
    configuration('node.name' => 'arbitrary_name')
    end
    elasticsearch_service 'my_elasticsearch_node' do
    service_actions [:enable, :start]
    end
    

    View Slide

17. 17
    Where we ended up
    Simple resources with no surprises
    Defaults follow packages, which have evolved
    Recipe installs most defaults, otherwise use resources
    Missing the more exotic features (EBS, S3, Discovery)
    Still cannot handle complex cases with plugins
    Does support multiple instances
    

    View Slide

18. 18
    More pure Ruby testing
    Cleanup logic
    Versioning
    Duplicate environment vars
    Where we go from here
    

    View Slide

19. 19
    FOLLOWED
    of all Chef
    cookbooks
    27th 1% 70%
    Elasticsearch cookbook
    TOP
    COOKBOOKS
    out of 2719 on
    Community
    ‘Supermarket’
    CHEF
    CLIENTS
    are Fortune 1000
    companies
    

    View Slide

20. 20
    How to get involved
    Logstash, Kibana, ELK stack Elastic stack
    elastic/cookbook-elasticsearch
    lusis/chef-logstash
    lusis/chef-kibana
    rackspace-cookbooks/elkstack
    

    View Slide

21. View Slide

22. 22
    Option 3
    Martin B. Smith, DevOps Engineer
    Rackspace Hosting, Inc.
    19 February, 2016
    @martinb3
    Questions outside by AMA booth
    

    View Slide

23. 23
    Except where otherwise noted, this work is licensed under
    http://creativecommons.org/licenses/by-nd/4.0/
    Creative Commons and the double C in a circle are
    registered trademarks of Creative Commons in the United States and other countries.
    Third party marks and brands are the property of their respective holders.
    

    View Slide