Wrangling Elasticsearch with Ansible A love story…. 

kinda... 

Who am I and why should you listen to me? ● Director of Engineering Operations ● Fedora Linux Contributor ● WireGuard VPN Packager ● Joined Kenna in 2015 as a Sr. Systems Engineer ● @jdoss on Twitter, Github, Copr and freenode.net ● Manage 5 production and 1 centralized logging clusters ● Over 4 billion documents with 200 million processed daily 

It was a dark timeline in 2015 ● Running Elasticsearch 1.7.x on Ubuntu managed with Chef ● Elasticsearch was falling over every few days ○ 500 million documents (1 million processed daily) ○ 6TB of data ○ 20 data nodes ○ 80+ shards ● We used Elasticsearch in weird ways (11mb documents, all searchable data was stored on a single document) ● No method to upgrade Elasticsearch versions between devel and prod 

It was not great but... 

There was hope! ● Created by Michael DeHaan ● Initial released in 2012 ● Written in mostly Python ● Configuration management, infrastructure as code, Orchestration engine ● Agentless ● Acquired by Red Hat in 2015 

Design Goals ● Secure ● Reproducible ● Maintainable ● Usable in development and production 

Defining Elasticsearch Clusters in Code Kenna US Production 

Defining Elasticsearch Clusters in Code Kenna GCP Production 

Defining Elasticsearch Clusters in Code Kenna Development 

Defining Elasticsearch Clusters in Code Kenna LAME (Logging, Alerting, Metrics, Events) 

Ansible Inventory with Static Groups of Dynamic Groups 

Ansible Inventory with Static Groups of Dynamic Groups 

Ansible Group_Vars -- Kenna US Production 

Ansible Group_Vars -- Kenna GCP Production 

Ansible Group_Vars -- Kenna LAME 

Ansible Group_Vars -- Kenna Development Role Defaults! 

Ansible Roles! Roles are ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users. 

Ansible Roles Quick Overview ● defaults - default variables for the role ● files - contains files which can be deployed via this role ● handlers - contains handlers, which may be used by this role or even anywhere outside this role. ● vars - other variables for the role ● tasks - contains the main list of tasks to be executed by the role ● templates - contains templates which can be deployed via this role ● meta - defines some meta data for this role 

Kenna’s Ansible Elasticsearch Role -- Defaults 

Kenna’s Ansible Elasticsearch Role -- Tasks 

Kenna’s Ansible Elasticsearch Role -- Templates 

Ansible Playbook 

TL; DR 

If you take six points home... ● Define your cluster in code! ● Use tags/labels to define instance types for your inventory! ● Static Groups of Dynamic Groups! ● Use group_vars on those static groups! ● Have sane default variables! ● Use Ansible to manage your Elasticsearch clusters!