$30 off During Our Annual Pro Sale. View Details »

Shield Your cluster - Security with Elasticsearch

Shield Your cluster - Security with Elasticsearch

This talk provides an overview of the different ways to secure an Elasticsearch cluster and an ELK environment. Some standard use cases are covered, such as putting HTTP proxies in front of Elasticsearch, then illustrating the advantages and obstacles of each option.

This talk explores the how and why Elasticsearch Shield was built, including how it helps you secure both your data and communication. Use case examples for Shield will also be on offer.

Elastic Co

June 18, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Shield your cluster
    Security with Elasticsearch
    Alexander Reelsen
    @spinscale
    [email protected]

    View Slide

  2. Agenda
    Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  3. About
    2012
    Elasticsearch got founded
    Series A investment
    Trainings
    Supports subscriptions

    View Slide

  4. About
    2012
    Series B investment
    Kibana
    Elasticsearch for Apache Hadoop Integration
    Logstash
    Elasticsearch Clients
    2013
    Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

    View Slide

  5. About
    2012
    Series C investment
    Marvel released
    2013 2014

    View Slide

  6. About
    2012
    Shield goes GA
    First user conference & rebrand
    Found acquired
    Packetbeat joins
    Watcher in beta
    2013 2014 2015

    View Slide

  7. About
    2012 2013 2014 2015
    Joined in March 2013
    Working on Elasticsearch & Shield
    Development, Trainings, Conferences, Support, Blog posts
    We're hiring...

    View Slide

  8. Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  9. Why?
    Elasticsearch: No security OOTB
    No encrypted communication
    No Authorization
    No Authentication
    No Audit Logging

    View Slide

  10. ES
    nginx
    client
    Filter by HTTP method, URI or IP
    User management via basic auth
    Use aliases & filters
    nginx in front

    View Slide

  11. ES
    nginx
    client
    How to solve multi index operations?
    nginx in front
    GET /logs-2015.10.10,evil,logs-2015.10.11
    {
    "query" : { "match_all": {} }
    }

    View Slide

  12. ES
    nginx
    client
    How to solve bulk/multi operations?
    nginx in front
    { "index" : { "_index" : "test1", "_type" : "type1", "_id" : "1" } }
    { "field1" : "value1" }
    { "delete" : { "_index" : "test2", "_type" : "type1", "_id" : "2" } }
    { "create" : { "_index" : "test3", "_type" : "type1", "_id" : "3" } }
    { "field1" : "value3" }
    { "update" : {"_id" : "1", "_type" : "type1", "_index" : "test4"} }
    { "doc" : {"field2" : "value2"} }

    View Slide

  13. ES
    nginx
    client
    Prevent unwanted accesses
    nginx in front
    HTTP/Transport

    View Slide

  14. ES
    nginx
    client
    nginx in front
    Firewall

    View Slide

  15. Data
    ACL
    client
    Configuration scattered across systems
    operational overhead
    IP Filtering

    View Slide

  16. Data
    ACL
    client
    operational overhead
    IP Filtering Directory
    Configuration scattered across systems

    View Slide

  17. Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  18. How?
    Elasticsearch modular & pluggable
    Security as a plugin
    HTTP + Transport protocols
    Integration into the ELK stack!

    View Slide

  19. How?
    Elasticsearch
    Elasticsearch
    auth_token Authentication
    Authorization

    View Slide

  20. How?
    Elasticsearch
    Elasticsearch
    auth_token
    200 OK
    Authentication
    Authorization

    View Slide

  21. How?
    Elasticsearch
    Elasticsearch
    auth_token
    401 Unauthorized
    Authentication
    Authorization

    View Slide

  22. How?
    Getting up and running is easy
    Install elasticsearch 1.6
    bin/plugin install elasticsearch/license/latest
    bin/plugin install elasticsearch/shield/latest

    View Slide

  23. Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  24. What?
    IP Filtering
    Encrypted communication
    Authentication
    Authorization
    Audit Trail

    View Slide

  25. Configurable in elasticsearch.yml
    Can be updated dynamically via
    cluster update settings API
    IP Filtering
    shield.transport.filter:
    allow: "192.168.0.1"
    deny: "192.168.0.0/24"

    View Slide

  26. keystore required
    different config for HTTP and
    transport protocol (+profiles)
    Encrypted communication
    shield.ssl.keystore.path: /path/to/keystore.jks
    shield.ssl.keystore.password: secret
    shield.transport.ssl: true
    shield.http.ssl: true

    View Slide

  27. Authentication
    "Who are you?"
    Auth mechanisms are called realms
    Available: esusers, ldap, ad, pki
    Realms can be chained
    Support for caching & API for clearing

    View Slide

  28. Authentication
    shield.authc:
    realms:
    esusers:
    type: esusers
    order: 0
    ldap1:
    type: ldap
    order: 1
    enabled: false
    url: 'url_to_ldap1'
    ...
    ad1:
    type: active_directory
    order: 3
    url: 'url_to_ad'

    View Slide

  29. ESusers realm
    Local files, can be changed via CLI
    Elasticsearch watches file changes &
    reloads
    config/shield/users
    config/shield/users_roles

    View Slide

  30. ESusers realm
    bin/shield/esusers useradd alex
    bin/shield/esusers roles alex -a
    admin -r user
    bin/shield/esusers list
    bin/shield/esusers userdel alex

    View Slide

  31. Fallback to configurable user
    Disabled by default
    Anonymous access
    shield.authc:
    anonymous:
    username: anonymous_user
    roles: role1, role2

    View Slide

  32. Authorization
    "Are you allowed to do that?"
    File: config/shield/roles.yml
    admin:
    cluster: all
    indices:
    '*': all

    View Slide

  33. Role Based Access Control
    role
    named set of permissions
    permission
    set of cluster wide privileges
    set of indices/aliases specific privileges
    privilege
    set of one or more action names
    /_search ⬌ indices:data/read/search

    View Slide

  34. Role Based Access Control
    admin:
    cluster: all
    indices:
    '*': all
    role permission

    View Slide

  35. Authorization
    user:
    indices:
    '*': read
    events_user:
    indices:
    'events_*': read

    View Slide

  36. Authorization
    get_user:
    indices:
    'events_index': 'indices:data/read/get'
    logfile_user_readonly:
    indices:
    "logstash-201?-*": read

    View Slide

  37. Audit Trail
    Writes an own audit log file
    Implemented as logger
    Logs different types of event based
    on log level
    (ip filtering, tampered requests, access denied, auth failed)
    shield.audit.enabled: true

    View Slide

  38. Integration
    Transport Client
    Logstash
    Kibana 3/4
    Watcher
    Marvel

    View Slide

  39. Transport Client
    TransportClient client = new TransportClient(builder()
    .put("cluster.name", "myClusterName")
    .put("shield.user", "test_user:changeme")
    .put("shield.ssl.keystore.path", "/path/to/client.jks")
    .put("shield.ssl.keystore.password", "password")
    .put("shield.transport.ssl", "true"))
    .addTransportAddress(new
    InetSocketTransportAddress("localhost", 9300));

    View Slide

  40. Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  41. Who?
    Use-case 1: Monitoring application
    No write access
    Cluster Health
    Nodes stats/info
    Indices Stats

    View Slide

  42. Use-case 2: Logstash
    No read access (unless input is used)
    Indices: Indexing
    Cluster: Index templates

    View Slide

  43. Use-case 3: Marvel
    marvel_user:
    cluster: cluster:monitor/nodes/info,
    cluster:admin/plugin/license/get
    indices:
    '.marvel-*': all
    marvel_agent:
    cluster: indices:admin/template/get,
    indices:admin/template/put
    indices:
    '.marvel-*': indices:data/write/bulk, create_index

    View Slide

  44. Use-case 4: Ecommerce
    bulk:
    indices:
    'products_*': write, manage, read
    updater:
    indices:
    'products': index, delete, indices:admin/optimize
    webshop:
    indices:
    'products': search, get

    View Slide

  45. Use-case 4: Ecommerce
    monitoring:
    cluster: monitor
    indices:
    '*': monitor
    sales_rep :
    indices:
    'sales_*' : all
    'social_events' : data_access, monitor

    View Slide

  46. Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  47. Next?
    Simplify SSL configuration
    API driven user/role management
    Open up realms API
    Field-level security
    Index Audit Trail into ES

    View Slide

  48. Why?
    How?
    Next?
    What?
    Who?
    Q & A

    View Slide

  49. Q & A
    Thanks for listening!
    Alexander Reelsen
    @spinscale
    [email protected]
    We're hiring
    https://www.elastic.co/about/careers
    We're helping
    https://www.elastic.co/subscriptions

    View Slide

  50. Resources
    Shield documentation
    https://www.elastic.co/guide/en/shield/current/index.html
    Shield: Security in ELK
    https://www.elastic.co/elasticon/2015/sf/security-in-elk
    Shield and Beyond:
    Recommendations for a Secure ELK
    Environment
    https://www.elastic.co/webinars/shield-and-beyond

    View Slide

  51. Resources
    https://discuss.elastic.co/c/shield

    View Slide

  52. Resources

    View Slide

  53. Resources

    View Slide

  54. Q & A
    Thanks for listening!
    Alexander Reelsen
    @spinscale
    [email protected]
    We're hiring
    https://www.elastic.co/about/careers
    We're helping
    https://www.elastic.co/subscriptions

    View Slide