Wrangling Elasticsearch With Ansible

How Kenna Security uses Ansible to build, maintain, and deploy all of their Elasticsearch clusters.

Transcript

1. Wrangling Elasticsearch with Ansible A love story….
2. kinda...
3. Who am I and why should you listen to me? • Director of Engineering Operations • Fedora Linux Contributor • WireGuard VPN Packager • Joined Kenna in 2015 as a Sr. Systems Engineer • @jdoss on Twitter, Github, Copr and freenode.net • Manage 5 production and 1 centralized logging clusters • Over 4 billion documents with 200 million processed daily
4. It was a dark timeline in 2015 • Running Elasticsearch 1.7.x on Ubuntu managed with Chef • Elasticsearch was falling over every few days ◦ 500 million documents (1 million processed daily) ◦ 6TB of data ◦ 20 data nodes ◦ 80+ shards • We used Elasticsearch in weird ways (11mb documents, all searchable data was stored on a single document) • No method to upgrade Elasticsearch versions between devel and prod
5. It was not great but...
6. There was hope! • Created by Michael DeHaan • Initial released in 2012 • Written in mostly Python • Configuration management, infrastructure as code, Orchestration engine • Agentless • Acquired by Red Hat in 2015
7. Design Goals • Secure • Reproducible • Maintainable • Usable in development and production
8. Defining Elasticsearch Clusters in Code Kenna US Production
9. Defining Elasticsearch Clusters in Code Kenna GCP Production
10. Defining Elasticsearch Clusters in Code Kenna Development
11. Defining Elasticsearch Clusters in Code Kenna LAME (Logging, Alerting, Metrics, Events)
12. Ansible Inventory with Static Groups of Dynamic Groups
13. Ansible Inventory with Static Groups of Dynamic Groups
14. Ansible Group_Vars -- Kenna US Production
15. Ansible Group_Vars -- Kenna GCP Production
16. Ansible Group_Vars -- Kenna LAME
17. Ansible Group_Vars -- Kenna Development Role Defaults!
18. Ansible Roles! Roles are ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.
19. Ansible Roles Quick Overview • defaults - default variables for the role • files - contains files which can be deployed via this role • handlers - contains handlers, which may be used by this role or even anywhere outside this role. • vars - other variables for the role • tasks - contains the main list of tasks to be executed by the role • templates - contains templates which can be deployed via this role • meta - defines some meta data for this role
20. Kenna’s Ansible Elasticsearch Role -- Defaults
21. Kenna’s Ansible Elasticsearch Role -- Tasks
22. Kenna’s Ansible Elasticsearch Role -- Templates
23. Ansible Playbook
24. TL; DR
25. If you take six points home... • Define your cluster in code! • Use tags/labels to define instance types for your inventory! • Static Groups of Dynamic Groups! • Use group_vars on those static groups! • Have sane default variables! • Use Ansible to manage your Elasticsearch clusters!