Cooking Up Elasticsearch with Chef at elastic{on}

It’s not written by Julia Child or Gordon Ramsey, but the elasticsearch cookbook is worth it. Come hear Martin Smith — co-author of the Chef Community Cookbook for Elasticsearch — present a deep dive on automating Elasticsearch installation and configuration with Chef, and re-writing the open source Chef cookbook the last 12 months.

Transcript

1. 1 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19 February, 2016 @martinb3 Cooking up Elasticsearch with Chef
2. 2 About the Chef Ecosystem Multiple workflows, what works best for you.
3. 3 Chef Ecosystem Desired State Centralized Resource Primitives • Idempotency • Declarative • Client-server model • Data & secret storage • Management tools • Ruby DSL • Multiple implementations
4. 4 an operation that will produce the same results if executed once or multiple times Idempotency … desired state
5. 5 5
6. 6 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb
7. 7 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do version node[‘curl_version’] action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb default[‘curl_version’] = ‘1.2.3’ attributes/curl.rb
8. 8 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do action :install end file ‘/etc/motd’ do content ‘Curl is installed’ end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb
9. 9 Elasticsearch cookbook A short history
10. 10 Elasticsearch cookbook • 2011 – Initial commits from Karel Minarik, ES @ 0.18.5 • 2013 – Capable of installing monit, nginx, EBS via Fog, Discovery • 2014 – Began discussing rewrites • 2015 – GCE support, and the kitchen sink… • 2015 July – released v1.0.0 A short history Source: Gray Arial 10pt
12. 12 Technical debt Attribute soup No multiple ES instances Unmaintained features Lack of testing Problems with the ‘current’ cookbook
13. 13 - Focus on basic features - Respecting Chef standards - Testability Guiding principles for rewrite/refactor - October 2014
14. 14 Simple, declarative resources Examples for package, service, file package ‘curl’ service ’mysql' do action [:enable, :start] end file ’/etc/motd' do content ‘Hello World!’ end
15. 15 Where we ended up Simple resources with no surprises Add appropriate user and group Install, uninstall by package or archive file (tar.gz) Configure elasticsearch.yml, parity with packages Create and start system service Install and remove plugins via bin/plugin
16. 16 Simple, declarative resources Examples using elasticsearch cookbook elasticsearch_install 'elasticsearch' do type :package end elasticsearch_configure 'elasticsearch' do logging(:action => 'INFO') allocated_memory '123m' configuration('node.name' => 'arbitrary_name') end elasticsearch_service 'my_elasticsearch_node' do service_actions [:enable, :start] end
17. 17 Where we ended up Simple resources with no surprises Defaults follow packages, which have evolved Recipe installs most defaults, otherwise use resources Missing the more exotic features (EBS, S3, Discovery) Still cannot handle complex cases with plugins Does support multiple instances
18. 18 More pure Ruby testing Cleanup logic Versioning Duplicate environment vars Where we go from here
19. 19 FOLLOWED of all Chef cookbooks 27th 1% 70% Elasticsearch cookbook TOP COOKBOOKS out of 2719 on Community ‘Supermarket’ CHEF CLIENTS are Fortune 1000 companies
20. 20 How to get involved Logstash, Kibana, ELK stack Elastic stack elastic/cookbook-elasticsearch lusis/chef-logstash lusis/chef-kibana rackspace-cookbooks/elkstack
22. 22 Option 3 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19 February, 2016 @martinb3 Questions outside by AMA booth
23. 23 Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.