Introduction into Open Distro from Elasticsearch

In this session I will give an introduction into what is Elasticsearch, going back through from its origins in the Lucene project, and then walk you through what Open Distro for Elasticsearch is; why we created it, what it looks like and then a quick demo of its capabilities.

Transcript

1. MUNICH
2. MUNICH Open Distro for Elasticsearch Ricardo Sueiras Principal Evangelist at AWS
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The history of Elasticsearch Source: DB-Engines.com, April 2019 Elasticsearch, Logstash, and Kibana Sometimes referred to as the “ELK Stack” Distributed search and analytics engine Build on Apache Lucene Easy ingestion and visualization Other partner solutions Splunk, Sumo Logic, Logz.io, and Loggly
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Machine data driving Elasticsearch growth Machine-generated data is growing 10x faster vs. business data… logs, logs, and more logs Source: insideBigData—The Exponential Growth of Data, February 16, 2017 IT and DevOps databases, servers, storage, networking Increase in IoT and mobile devices gaming, sensors, web content Cloud-based architectures
5. It is a database Application Data Server, application, network, AWS, and other logs 1 Elasticsearch Cluster 2 Application users, analysts, DevOps, security 3 1 Send data as JSON via REST APIs 2 Data is indexed— all fields searchable, including nested JSON 3 Queries, via REST APIs, allow fielded matching, Boolean expressions, include sorting and analysis
6. Elasticsearch concepts and terminology Relational DB concepts Elasticsearch concepts Database Rows/Records Table Column Name Unique key Schema Index Document Type Field ID Mapping
7. Elasticsearch concepts and terminology An index is split into shards Each shard runs its own instance of lucene Shard may be on different nodes in a cluster 1 2 3 Index Shard Shard Shard
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Elasticsearch runs on a cluster of instances Data Nodes Master Nodes Client Nodes (optional) Request processing Cluster state HTTP/s API requests
9. Interacting with Elasticsearch Restful endpoints You interact with Elasticsearch via HTTP(s) requests and JSON data Client APIs There are many client libraries that make this even easier across many languages and platforms Analytics and Visualizations Tools like Kibana and others enable easy access to data to explore and perform analytics
10. You use the indexing APIs to send data POST endpoint/index/type/id { Document } * Your ingestion tools will probably automate this
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Elasticsearch cluster Matches Query Engine You use the query APIs to retrieve data Ranked results Scoring/Sorting
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Elasticsearch cluster Matches Query Engine You use aggregations to analyze log data • Histogram • Numeric sum, min, max • Terms bucketing • Nesting Analysis Engine (Aggregations)
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kibana is a lightweight, real-time visualization tool
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Behind the scenes logs_11.26.201 8 logs_11.25.201 8 logs_11.24.201 8
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Introducing…
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How we think about OSS licensing and distribution? Base OSS free of proprietary code Keep commercial software on top of OSS separate Allow anyone to innovate on OSS Don’t change licensing or distribution midstream
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. An Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise- grade security, alerting, SQL, and more
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Open Distro for Elasticsearch BENEFITS 100% open source Providing you the freedoms, so you can freely view, use, change, and distribute the code Enterprise-grade Delivering security and advanced capabilities such as alerting, SQL, and cluster diagnostics Community-driven Providing individuals and organizations the freedom to easily contribute changes to the distro
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Open Distro for Elasticsearch SQL FEATURES Security Achieve encryption in-flight, role-based access control, audit logging, and compliance Alerting Monitor your data and send automatic alerts on any changes in your data SQL Easily interact with your Elasticsearch cluster and extract insights using the familiar SQL query syntax Performance Analyzer Get deep visibility into system bottlenecks even when your Elasticsearch cluster is under duress
20. Security Encryption Keep your data secure when in transit Authentication Leverage your existing authentication infrastructure RBAC Granular access control to control the user actions on your cluster Audit logging Track and record all user actions and meet HIPAA, PCI compliance KEEP YOUR DATA SECURE
21. Access control flow for RBAC Authc Via basic HTTP auth, LDAP, AD, SAML, web tokens, SSL Authz Backend identities mapped to Open Distro roles Permissions Allow a role to perform an action against a cluster/index/document/field Action groups Groups of permissions Request with credentials Authc Request with user/backend roles Authz Response Roles and permissions Authc provider
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enabled out of the box Kibana multi- tenancy Group A Group B Group B permissions Group A permissions Index 1 Index 2 Dashboard B Dashboard A
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Audit Logs Cluster access by authenticated user Request path—expose who did what Combine with the Alerting plugin for security notifications Cluster access by authenticated user Request path— expose who did what Combine with the Alerting plugin for security notifications
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alerting Create monitors Query the data you want to and receive alerts on it Customize alert conditions Define alerting threshold and severity for multiple trigger conditions Get notifications Built-in integrations for webhook and Slack to get notified on the channels you use View alerts All alert executions are indexed for easy tracking and visualization RECEIVE ALERTS ON YOUR DATA
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. SQL support Comprehensive SQL support Supports over 40 functions, data types, and commands including join support Translate SQL to JSON Create JSON using SQL to configure sophisticated access control policies Use existing tools Provides a JDBC driver so you can use a variety of business intelligence, analytics, and ETL tools QUERY DATA WITH SQL
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Analyzer Identify bottlenecks across the stack Provides a powerful REST API for querying Elasticsearch metrics to diagnose issues across stack Runs independent of your cluster Perform diagnostics even if the cluster is under duress Analyze hundreds of data points Supports over 60 metrics across 10 dimensions for instrumentation of your cluster health GET DEEP DIAGNOSTIC INSIGHTS INTO YOUR CLUSTER
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PerfTop CLI Provides pre-configured dashboards for analyzing cluster, node, and shard performance Custom JSON templates to create the dashboards to diagnose your cluster performance
28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simple to get started 1 Visit the website 2 Download the Elasticsearch and Kibana packages 3 Load and query data
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Flexible deployment options Docker RPM Debian
30. Community and contributions Open Distro for Elasticsearch’s success is driven by the community’s participation, contributions, and innovation to the project. You can follow project discussions, engage with fellow community members, contribute PRs, file bugs or request a feature at: Discussion forums https://discuss.opendistrocommunity.dev/ Community https://github.com/opendistro-for-elasticsearch/community/issues
31. Useful links Project website and technical documentation https://opendistro.github.io/for-elasticsearch/ Source code https://github.com/opendistro-for-elasticsearch
32. MUNICH Danke!