Slide 1

Slide 1 text

MUNICH

Slide 2

Slide 2 text

MUNICH Open Distro for Elasticsearch Ricardo Sueiras Principal Evangelist at AWS

Slide 3

Slide 3 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The history of Elasticsearch Source: DB-Engines.com, April 2019 Elasticsearch, Logstash, and Kibana Sometimes referred to as the “ELK Stack” Distributed search and analytics engine Build on Apache Lucene Easy ingestion and visualization Other partner solutions Splunk, Sumo Logic, Logz.io, and Loggly

Slide 4

Slide 4 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Machine data driving Elasticsearch growth Machine-generated data is growing 10x faster vs. business data… logs, logs, and more logs Source: insideBigData—The Exponential Growth of Data, February 16, 2017 IT and DevOps databases, servers, storage, networking Increase in IoT and mobile devices gaming, sensors, web content Cloud-based architectures

Slide 5

Slide 5 text

It is a database Application Data Server, application, network, AWS, and other logs 1 Elasticsearch Cluster 2 Application users, analysts, DevOps, security 3 1 Send data as JSON via REST APIs 2 Data is indexed— all fields searchable, including nested JSON 3 Queries, via REST APIs, allow fielded matching, Boolean expressions, include sorting and analysis

Slide 6

Slide 6 text

Elasticsearch concepts and terminology Relational DB concepts Elasticsearch concepts Database Rows/Records Table Column Name Unique key Schema Index Document Type Field ID Mapping

Slide 7

Slide 7 text

Elasticsearch concepts and terminology An index is split into shards Each shard runs its own instance of lucene Shard may be on different nodes in a cluster 1 2 3 Index Shard Shard Shard

Slide 8

Slide 8 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Elasticsearch runs on a cluster of instances Data Nodes Master Nodes Client Nodes (optional) Request processing Cluster state HTTP/s API requests

Slide 9

Slide 9 text

Interacting with Elasticsearch Restful endpoints You interact with Elasticsearch via HTTP(s) requests and JSON data Client APIs There are many client libraries that make this even easier across many languages and platforms Analytics and Visualizations Tools like Kibana and others enable easy access to data to explore and perform analytics

Slide 10

Slide 10 text

You use the indexing APIs to send data POST endpoint/index/type/id { Document } * Your ingestion tools will probably automate this

Slide 11

Slide 11 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Elasticsearch cluster Matches Query Engine You use the query APIs to retrieve data Ranked results Scoring/Sorting

Slide 12

Slide 12 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Elasticsearch cluster Matches Query Engine You use aggregations to analyze log data • Histogram • Numeric sum, min, max • Terms bucketing • Nesting Analysis Engine (Aggregations)

Slide 13

Slide 13 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kibana is a lightweight, real-time visualization tool

Slide 14

Slide 14 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Behind the scenes logs_11.26.201 8 logs_11.25.201 8 logs_11.24.201 8

Slide 15

Slide 15 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Introducing…

Slide 16

Slide 16 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How we think about OSS licensing and distribution? Base OSS free of proprietary code Keep commercial software on top of OSS separate Allow anyone to innovate on OSS Don’t change licensing or distribution midstream

Slide 17

Slide 17 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. An Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise- grade security, alerting, SQL, and more

Slide 18

Slide 18 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Open Distro for Elasticsearch BENEFITS 100% open source Providing you the freedoms, so you can freely view, use, change, and distribute the code Enterprise-grade Delivering security and advanced capabilities such as alerting, SQL, and cluster diagnostics Community-driven Providing individuals and organizations the freedom to easily contribute changes to the distro

Slide 19

Slide 19 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Open Distro for Elasticsearch SQL FEATURES Security Achieve encryption in-flight, role-based access control, audit logging, and compliance Alerting Monitor your data and send automatic alerts on any changes in your data SQL Easily interact with your Elasticsearch cluster and extract insights using the familiar SQL query syntax Performance Analyzer Get deep visibility into system bottlenecks even when your Elasticsearch cluster is under duress

Slide 20

Slide 20 text

Security Encryption Keep your data secure when in transit Authentication Leverage your existing authentication infrastructure RBAC Granular access control to control the user actions on your cluster Audit logging Track and record all user actions and meet HIPAA, PCI compliance KEEP YOUR DATA SECURE

Slide 21

Slide 21 text

Access control flow for RBAC Authc Via basic HTTP auth, LDAP, AD, SAML, web tokens, SSL Authz Backend identities mapped to Open Distro roles Permissions Allow a role to perform an action against a cluster/index/document/field Action groups Groups of permissions Request with credentials Authc Request with user/backend roles Authz Response Roles and permissions Authc provider

Slide 22

Slide 22 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enabled out of the box Kibana multi- tenancy Group A Group B Group B permissions Group A permissions Index 1 Index 2 Dashboard B Dashboard A

Slide 23

Slide 23 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Audit Logs Cluster access by authenticated user Request path—expose who did what Combine with the Alerting plugin for security notifications Cluster access by authenticated user Request path— expose who did what Combine with the Alerting plugin for security notifications

Slide 24

Slide 24 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alerting Create monitors Query the data you want to and receive alerts on it Customize alert conditions Define alerting threshold and severity for multiple trigger conditions Get notifications Built-in integrations for webhook and Slack to get notified on the channels you use View alerts All alert executions are indexed for easy tracking and visualization RECEIVE ALERTS ON YOUR DATA

Slide 25

Slide 25 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. SQL support Comprehensive SQL support Supports over 40 functions, data types, and commands including join support Translate SQL to JSON Create JSON using SQL to configure sophisticated access control policies Use existing tools Provides a JDBC driver so you can use a variety of business intelligence, analytics, and ETL tools QUERY DATA WITH SQL

Slide 26

Slide 26 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Analyzer Identify bottlenecks across the stack Provides a powerful REST API for querying Elasticsearch metrics to diagnose issues across stack Runs independent of your cluster Perform diagnostics even if the cluster is under duress Analyze hundreds of data points Supports over 60 metrics across 10 dimensions for instrumentation of your cluster health GET DEEP DIAGNOSTIC INSIGHTS INTO YOUR CLUSTER

Slide 27

Slide 27 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PerfTop CLI Provides pre-configured dashboards for analyzing cluster, node, and shard performance Custom JSON templates to create the dashboards to diagnose your cluster performance

Slide 28

Slide 28 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simple to get started 1 Visit the website 2 Download the Elasticsearch and Kibana packages 3 Load and query data

Slide 29

Slide 29 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Flexible deployment options Docker RPM Debian

Slide 30

Slide 30 text

Community and contributions Open Distro for Elasticsearch’s success is driven by the community’s participation, contributions, and innovation to the project. You can follow project discussions, engage with fellow community members, contribute PRs, file bugs or request a feature at: Discussion forums https://discuss.opendistrocommunity.dev/ Community https://github.com/opendistro-for-elasticsearch/community/issues

Slide 31

Slide 31 text

Useful links Project website and technical documentation https://opendistro.github.io/for-elasticsearch/ Source code https://github.com/opendistro-for-elasticsearch

Slide 32

Slide 32 text

MUNICH Danke!