1 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19 February, 2016 @martinb3 Cooking up Elasticsearch with Chef

2 About the Chef Ecosystem Multiple workflows, what works best for you.

3 Chef Ecosystem Desired State Centralized Resource Primitives • Idempotency • Declarative • Client-server model • Data & secret storage • Management tools • Ruby DSL • Multiple implementations

4 an operation that will produce the same results if executed once or multiple times Idempotency … desired state

5 5

6 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb

7 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do version node[‘curl_version’] action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb default[‘curl_version’] = ‘1.2.3’ attributes/curl.rb

8 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do action :install end file ‘/etc/motd’ do content ‘Curl is installed’ end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb

9 Elasticsearch cookbook A short history

10 Elasticsearch cookbook • 2011 – Initial commits from Karel Minarik, ES @ 0.18.5 • 2013 – Capable of installing monit, nginx, EBS via Fog, Discovery • 2014 – Began discussing rewrites • 2015 – GCE support, and the kitchen sink… • 2015 July – released v1.0.0 A short history Source: Gray Arial 10pt

11

12 Technical debt Attribute soup No multiple ES instances Unmaintained features Lack of testing Problems with the ‘current’ cookbook

13 - Focus on basic features - Respecting Chef standards - Testability Guiding principles for rewrite/refactor - October 2014

14 Simple, declarative resources Examples for package, service, file package ‘curl’ service ’mysql' do action [:enable, :start] end file ’/etc/motd' do content ‘Hello World!’ end

15 Where we ended up Simple resources with no surprises Add appropriate user and group Install, uninstall by package or archive file (tar.gz) Configure elasticsearch.yml, parity with packages Create and start system service Install and remove plugins via bin/plugin

16 Simple, declarative resources Examples using elasticsearch cookbook elasticsearch_install 'elasticsearch' do type :package end elasticsearch_configure 'elasticsearch' do logging(:action => 'INFO') allocated_memory '123m' configuration('node.name' => 'arbitrary_name') end elasticsearch_service 'my_elasticsearch_node' do service_actions [:enable, :start] end

17 Where we ended up Simple resources with no surprises Defaults follow packages, which have evolved Recipe installs most defaults, otherwise use resources Missing the more exotic features (EBS, S3, Discovery) Still cannot handle complex cases with plugins Does support multiple instances

18 More pure Ruby testing Cleanup logic Versioning Duplicate environment vars Where we go from here

19 FOLLOWED of all Chef cookbooks 27th 1% 70% Elasticsearch cookbook TOP COOKBOOKS out of 2719 on Community ‘Supermarket’ CHEF CLIENTS are Fortune 1000 companies

20 How to get involved Logstash, Kibana, ELK stack Elastic stack elastic/cookbook-elasticsearch lusis/chef-logstash lusis/chef-kibana rackspace-cookbooks/elkstack

21

22 Option 3 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19 February, 2016 @martinb3 Questions outside by AMA booth

23 Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.