Slide 1

Slide 1 text

1 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19 February, 2016 @martinb3 Cooking up Elasticsearch with Chef

Slide 2

Slide 2 text

2 About the Chef Ecosystem Multiple workflows, what works best for you.

Slide 3

Slide 3 text

3 Chef Ecosystem Desired State Centralized Resource Primitives • Idempotency • Declarative • Client-server model • Data & secret storage • Management tools • Ruby DSL • Multiple implementations

Slide 4

Slide 4 text

4 an operation that will produce the same results if executed once or multiple times Idempotency … desired state

Slide 5

Slide 5 text

5 5

Slide 6

Slide 6 text

6 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb

Slide 7

Slide 7 text

7 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do version node[‘curl_version’] action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb default[‘curl_version’] = ‘1.2.3’ attributes/curl.rb

Slide 8

Slide 8 text

8 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do action :install end file ‘/etc/motd’ do content ‘Curl is installed’ end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb

Slide 9

Slide 9 text

9 Elasticsearch cookbook A short history

Slide 10

Slide 10 text

10 Elasticsearch cookbook • 2011 – Initial commits from Karel Minarik, ES @ 0.18.5 • 2013 – Capable of installing monit, nginx, EBS via Fog, Discovery • 2014 – Began discussing rewrites • 2015 – GCE support, and the kitchen sink… • 2015 July – released v1.0.0 A short history Source: Gray Arial 10pt

Slide 11

Slide 11 text

11

Slide 12

Slide 12 text

12 Technical debt Attribute soup No multiple ES instances Unmaintained features Lack of testing Problems with the ‘current’ cookbook

Slide 13

Slide 13 text

13 - Focus on basic features - Respecting Chef standards - Testability Guiding principles for rewrite/refactor - October 2014

Slide 14

Slide 14 text

14 Simple, declarative resources Examples for package, service, file package ‘curl’ service ’mysql' do action [:enable, :start] end file ’/etc/motd' do content ‘Hello World!’ end

Slide 15

Slide 15 text

15 Where we ended up Simple resources with no surprises Add appropriate user and group Install, uninstall by package or archive file (tar.gz) Configure elasticsearch.yml, parity with packages Create and start system service Install and remove plugins via bin/plugin

Slide 16

Slide 16 text

16 Simple, declarative resources Examples using elasticsearch cookbook elasticsearch_install 'elasticsearch' do type :package end elasticsearch_configure 'elasticsearch' do logging(:action => 'INFO') allocated_memory '123m' configuration('node.name' => 'arbitrary_name') end elasticsearch_service 'my_elasticsearch_node' do service_actions [:enable, :start] end

Slide 17

Slide 17 text

17 Where we ended up Simple resources with no surprises Defaults follow packages, which have evolved Recipe installs most defaults, otherwise use resources Missing the more exotic features (EBS, S3, Discovery) Still cannot handle complex cases with plugins Does support multiple instances

Slide 18

Slide 18 text

18 More pure Ruby testing Cleanup logic Versioning Duplicate environment vars Where we go from here

Slide 19

Slide 19 text

19 FOLLOWED of all Chef cookbooks 27th 1% 70% Elasticsearch cookbook TOP COOKBOOKS out of 2719 on Community ‘Supermarket’ CHEF CLIENTS are Fortune 1000 companies

Slide 20

Slide 20 text

20 How to get involved Logstash, Kibana, ELK stack Elastic stack elastic/cookbook-elasticsearch lusis/chef-logstash lusis/chef-kibana rackspace-cookbooks/elkstack

Slide 21

Slide 21 text

21

Slide 22

Slide 22 text

22 Option 3 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19 February, 2016 @martinb3 Questions outside by AMA booth

Slide 23

Slide 23 text

23 Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.