‹#› Edition: April 24, 2017 Elastic Stack/X-Pack 5.0 for IT Security Workshop

Objective • Be an expert with broader knowledge to the Elastic Stack and the X-Pack • Secure your cluster • Leverage realtime alerting capability for daily IT security routine 2 "Take full advantage of the Elastic Stack and the X-Pack to maximize your IT security analytics."

3 Dealing with Time-series Data

Curl vs Console with Kibana 4 $ curl -XGET "https://ES_HOST:ES_PORT/_search" -H "Content-type:application/json" \ -u ES_USER:ES_PASSWORD -d' { "query": { "match_all": {} } }'

CRUD 5 PUT my-log-2017.03.02/my-type/1 { "@timestamp": "2017-03-02T14:12:00", "host": "server-01", "message": "user elastic logged in" } GET my-log-2017.03.02/my-type/1 PUT my-log-2017.03.02/my-type/1 { "@timestamp": "2017-03-02T14:12:00", "host": "server-01", "message": "user elastic logged off" } DELETE my-log-2017.03.02/my-type/1

Search Basics 6 GET my-log-*/_search?q=* GET my-log-*/_search { "size": 10, "query": { "match_all": {} } } URI search with Query String Query (Lucene Syntax) Search with Query DSL

Range Query with Date Math 7 GET my-log-*/_search { "query": { "range": { "@timestamp": { "gte": "now-10m" } } } } Search the events happened within the last 10 minutes.

Cardinality Aggregation 8 GET packetbeat-*/_search { "size": 0, "query": { "range": { "@timestamp": { "gte": "now-10m" } } }, "aggs": { "1": { "cardinality": { "field": "ip" } } } }
 { "took": 2, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 3262, "max_score": 0, "hits": [] }, "aggregations": { "1": { "value": 4 } } }

Date Histogram Aggregation 9 GET my-log-2017.02.28/_search { "size": 0, "aggs": { "1": { "date_histogram": { "field": "@timestamp", "interval": "minute" } } } } { "hits": { "total": 280646, "max_score": 0, "hits": [] }, "aggregations": { "1": { "buckets": [ { "key_as_string": "2017-02-28T05:21:00.000Z", "key": 1488259260000, "doc_count": 686 }, { "key_as_string": "2017-02-28T05:22:00.000Z", "key": 1488259320000, "doc_count": 1387 }, { "key_as_string": "2017-02-28T05:23:00.000Z", "key": 1488259380000, "doc_count": 1384 },

Cardinality Aggregation over Minutes 10 GET packetbeat-*/_search { "size": 0, "query": { "range": { "@timestamp": { "gte": "now-10m" } } }, "aggs": { "1": { "date_histogram": { "field": "@timestamp", "interval": "minute" }, "aggs": { "2": { "cardinality": { "field": "ip" } } } } } }
 { "aggregations": { "1": { "buckets": [ { "2": { "value": 20 }, "key_as_string": "2017-03-28T04:55:00.000Z", "key": 1490676900000, "doc_count": 97 }, { "2": { "value": 18 }, "key_as_string": "2017-03-28T04:56:00.000Z", "key": 1490676960000, "doc_count": 243 }, { "2": { "value": 14 }, "key_as_string": "2017-03-28T04:57:00.000Z", "key": 1490677020000, "doc_count": 382 },

11 Timelion

Pull Data from Elasticsearch 12 .es(index=metricbeat-*) Show document counts .es(index=metricbeat-*,metric=avg:system.cpu.user.pct) Plot aggregated values (avg, sum, min, max or cardinality) .es(index=metricbeat-*,metric=avg:system.cpu.user.pct).movingaverage(window=3) Moving Average Aggregation .es(index=metricbeat-*,q=metricset.module:system) Query with Lucene syntax

Styles 13 .es(index=metricbeat-*,metric=avg:system.cpu.user.pct).bars()
 .es(index=metricbeat-*,metric=avg:system.cpu.user.pct).lines()
 .es(index=metricbeat-*,metric=avg:system.cpu.user.pct).points()

Colors 14 .es(index=metricbeat-*,metric=avg:system.cpu.user.pct).bars().color(lightblue)

Operations 15 $avg=.es(index=metricbeat-*,metric=avg:system.cpu.user.pct) Assignment ($avg).add($avg) ($avg).multiply(2) ($avg).subtract($avg) ($avg).divide($avg) Arithmetic ($avg).if(gt,0.2,$avg,null) Conditional

Code Helper is Your Friend 16 Type in "." to show useful tips

17 Types, Analyzers and Mappings

Text vs Keyword Types 18 PUT my-index { "mappings": { "_default_": { "properties" : { "mail_address" : { "type": "text" } } } } } PUT my-index { "mappings": { "_default_": { "properties" : { "mail_address" : { "type": "keyword" } } } } } "IT-Admin@example.com" to be tokenized into ["it", "admin", "example", "com"] "IT-Admin@example.com" remains

Defining and Testing Custom Analyzer 19 PUT my-index { "settings": { "analysis": { "analyzer": { "lowercase-keyword": { "type": "custom", "tokenizer": "keyword", "filter": [ "lowercase" ] } } } } } GET my-index/_analyze { "text": ["IT-Admin@elastic.co"], "analyzer": "lowercase-keyword" }

path_hierarchy Tokenizer 20 POST _analyze { "tokenizer": "path_hierarchy", "text": "/one/two/three" }
 { "tokens": [ { "token": "/one", "start_offset": 0, "end_offset": 4, "type": "word", "position": 0 }, { "token": "/one/two", "start_offset": 0, "end_offset": 8, "type": "word", "position": 0 }, { "token": "/one/two/three", "start_offset": 0, "end_offset": 14, "type": "word", "position": 0 } ] }

21 Packetbeat

Packetbeat 22 Protocols HTTP, Thrift-RPC, DNS, MySQL, PostgreSQL, Redis, Memcached, MongoDB, ICMP, AMQP and Cassandra Realtime Monitoring Passively works. Zero latency overhead. A lightweight real-time network packet analyzer that you can use with Elasticsearch to provide an application monitoring and performance analytics system.

Ready Made Dashboards 23

Configurations 24 packetbeat.interfaces.device: en0 packetbeat.protocols.icmp: enabled: true packetbeat.protocols.http: ports: [80, 8080, 8000, 5000, 8002] … output.elasticsearch: hosts: ["ES_HOST:9200"] protocol: "https" username: "elastic" password: "changeme" Protocols and output $ sudo packetbeat -e —c packetbeat.yml Running from the command line

25 X-Pack: Security

Security Features 26 Access Control Role-base access control against indices, documents and fields. Native, LDAP, AD, PKI and custom realms are supported. Encrypting Communications Enable SSL/TLS against endpoints and cluster-internal communications. IP Filtering Deny/allow access from specific hosts and IP addresses. Auditing Security Events Record security events on index and log file.

‹#›

Built-in kibana_user Role 28

Creating Read-only Role 29

Creating User with Read-only Role 30

31 X-Pack: Alerting

Your Watch E.g. • Send e-mail to web admins when the number of access/min is 120% greater than the moving average. Check it every minute. • Slack on #it-sec when the number of login failures/minute per ip is greater than 5. Check it every 5 seconds. • Generate a report from a dashboard as always. Check it 8am on Mondays. 32 Can be described in a natural language as: [Action] when [input] is [condition]. Check it [trigger].

Watch APIs 33 PUT _xpack/watcher/watch/my-watch { … } GET _xpack/watcher/watch/my_watch DELETE _xpack/watcher/watch/my_watch PUT _xpack/watcher/watch/my_watch/_activate PUT _xpack/watcher/watch/my_watch/_deactivate

Watch Definition 34 trigger Determines how frequently the watch is checked. (hourly, daily, weekly, monthly, yearly, cron or interval) input Loads data into the watch payload. What alert on. Typically an Elasticsearch query. (simple, search, http, chain) condition Decides whether to take actions. (always, never, compare, array_compare, script) transform Processes the watch payload. Both the watch level and the action level are available. actions Specifies actions to take when the condition is met. (email, webhook, index, logging and etc.) metadata Defines optional static metadata. PUT _xpack/watcher/watch/my-watch { "trigger": {…}, "input": {…}, "condition": {…}, "transform": {…}, "actions": {…} "metadata": {…} }

Watch History 35 GET .watcher-history-*/_search watch_id The name of the watch that was triggered. trigger_event How the watch was triggered (manual or schedule) and the watch’s scheduled time and actual trigger time. input The input type (http, search, or simple) and definition. condition The condition type (always, never, or script) and definition. state The state of the watch execution (execution_not_needed, executed, throttled). result The results of each phase of the watch execution. Shows the input payload, condition status, transform status (if defined), and actions status

Watch Context 36 ctx.watch_id The id of the watch that is currently executing. ctx.execution_time The time execution of this watch started. ctx.trigger.triggered_time The time this watch was triggered. ctx.trigger.scheduled_time The time this watch was supposed to be triggered. ctx.metadata.* Any metadata associated with the watch. ctx.payload.* The payload data loaded by the watch’s input.

Trigger - Interval 37 { "trigger" : { "schedule" : { "interval" : "5m" } } } Runs triggers every five minutes.

Input - Search 38 { "input": { "search": { "request": { "indices": [ "logs" ], "body": { "query": { "match_all": {} } } }, "extract": [ "hits.total" ] } } } Run query/aggregation upon a local Elasticsearch cluster.

Watch Payload 39 ctx.payload.hits All the search hits. ctx.payload.hits.total Number of documents of being hit. ctx.payload.hits.hits.0 The first document of the hits. ctx.payload.hits.hits..fields. A field value of a particular hit. ctx.payload.aggregations..buckets...value An aggregated value of a specific bucket.

Conditions 40 { "condition" : { "compare" : { "ctx.payload.hits.total" : { "gte" : 5 } } } { "condition": { "always": {} } } { "condition": { "never": {} } } "always" forces the watch actions to be executed unless they are throttled. Never execute actions. Frequently used for comparing the value in the watch payload with a threshold. Available operators: eq, not_eq, gt, gte, lt and lte.

Action - Email Setup 41 xpack.notification.email.account: gmail_account: profile: gmail smtp: auth: true starttls.enable: true host: smtp.gmail.com port: 587 user: Configure an email account in elasticsearch.yml.

Action - Email 42 { "actions":{ "send_email":{ "email":{ "to":"@", "subject":"Watcher Notification", "body":"{{ctx.payload.hits.total}} error logs found", "attachments":{ "dashboard.pdf":{ "reporting":{ "url":"http://example.org:5601/api/reporting/generate/dashboard/Error-Monitoring" } } } } } } } The subject and the body can contain static text and the watch context as Mustache templates. http, data and reporting type attachments are supported.

Action - Webhook 43 "actions" : { "create_github_issue" : { "webhook" : { "method" : "POST", "url" : "https://api.github.com/repos///issues", "body" : "{ \"title\": \"Found errors in 'contact.html'\", \"body\": \"Found {{ctx.payload.hits.total}} errors in the last 5 minutes\", \"assignee\": \"web-admin\", \"labels\": [ \"bug\", \"sev2\" ] }", "auth" : { "basic" : { "username" : "", "password" : "" } } } } } Performs a HTTP/HTTPS request to any third party’s web service.

Action - Index Single Document 44 "actions" : { "index_payload" : { "index" : { "index" : "my-index", "doc_type" : "my-type" , "execution_time_field": "@timestamp" } } Index ctx.payload into an Elasticsearch index as a single document.

Action - Time Based Throttling 45 "actions" : { "email_administrator" : { "throttle_period": "15m", "email" : { … "throttle_period" : "15m", "actions" : { "email_administrator" : { "email" : { … "notify_pager" : { The watch level and action level throttling is available. The action will not be taken while throttled (default 5 sec).

Execute Watch API 46 PUT _xpack/watcher/watch/my-watch/_execute PUT _xpack/watcher/watch/_execute { "watch" : { "trigger": { … }, "input": { … }, "condition": { … }, "actions": { … }, "meta": { … }, "throttle_period": { … } } } Execute a watch inline without registering for debugging. Forces execution of a stored watch outside of its triggering logic.

Alerting Idea - Minute by Minute Roll-up 47 "input": { "search": { "request": { "indices": ["flight-track—*"], "body": { "query": { "range": { "@timestamp": {"gte": "now-1m" } } }, "aggs": { "1": { "stats": {"field": "speed"} } … "actions": { "index_payload": { "transform": { "script": { "lang": "painless", "inline": "return ctx.payload.aggregations.1" } }, "index": { "index": "rollup-speed", "doc_type": "metric", "execution_time_field": "@timestamp" Strategy: Run stats aggregation upon a specific field every minute and index.

Alerting Idea - Alert with Moving Average Aggregation 48 { "condition":{ "script":{ "lang":"painless", "inline":"return ctx.payload.aggregations.agg_day.buckets.29.agg_bytes.value > ctx.payload.aggregations.agg_day.buckets.29.agg_moving_avg.value * params.gap", "params":{ "gap":1.2 } } } } Strategy: Run moving_avg aggregation upon the target index. Compare the value on the last bucket with the actual value. This example runs upon 30 days with "interval": "day" setting thus, the 30 buckets will be returned.

Alerting Idea - Measure Time Differences 49 "aggs":{ "agg_session_id":{ "terms":{ "field":"session_id.keyword" }, "aggs":{ "agg_user":{ "terms":{ "field":"user.keyword" } }, "agg_start":{ "min":{ "field":"@timestamp" } }, "agg_end":{ "max":{"field":"@timestamp" } }, "agg_duration":{ "bucket_script":{ "buckets_path":{ "min":"agg_start", "max":"agg_end" }, "script":{ "lang":"painless", "inline":"return params.max - params.min" } } } Strategy: Run terms aggregation upon a field which specifies a time series event and calculate max - min timestamp.

Restrictions on Elastic Cloud 50 • Email is delivered from the Elastic Cloud as the Email action is taken. So use of own SMTP server is not possible. • The default throttle period is not configurable. Specify a throttle period per watch/action, however.