Shield Your cluster - Security with Elasticsearch

This talk provides an overview of the different ways to secure an Elasticsearch cluster and an ELK environment. Some standard use cases are covered, such as putting HTTP proxies in front of Elasticsearch, then illustrating the advantages and obstacles of each option.

This talk explores the how and why Elasticsearch Shield was built, including how it helps you secure both your data and communication. Use case examples for Shield will also be on offer.

Transcript

1. Shield your cluster
   Security with Elasticsearch
   Alexander Reelsen
   @spinscale
   alex@elastic.co
   

   View Slide

2. Agenda
   Why?
   How?
   Next?
   What?
   Who?
   Q & A
   

   View Slide

3. About
   2012
   Elasticsearch got founded
   Series A investment
   Trainings
   Supports subscriptions
   

   View Slide

4. About
   2012
   Series B investment
   Kibana
   Elasticsearch for Apache Hadoop Integration
   Logstash
   Elasticsearch Clients
   2013
   Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
   

   View Slide

5. About
   2012
   Series C investment
   Marvel released
   2013 2014
   

   View Slide

6. About
   2012
   Shield goes GA
   First user conference & rebrand
   Found acquired
   Packetbeat joins
   Watcher in beta
   2013 2014 2015
   

   View Slide

7. About
   2012 2013 2014 2015
   Joined in March 2013
   Working on Elasticsearch & Shield
   Development, Trainings, Conferences, Support, Blog posts
   We're hiring...
   

   View Slide

8. Why?
   How?
   Next?
   What?
   Who?
   Q & A
   

   View Slide

9. Why?
   Elasticsearch: No security OOTB
   No encrypted communication
   No Authorization
   No Authentication
   No Audit Logging
   

   View Slide

10. ES
    nginx
    client
    Filter by HTTP method, URI or IP
    User management via basic auth
    Use aliases & filters
    nginx in front
    

    View Slide

11. ES
    nginx
    client
    How to solve multi index operations?
    nginx in front
    GET /logs-2015.10.10,evil,logs-2015.10.11
    {
    "query" : { "match_all": {} }
    }
    

    View Slide

12. ES
    nginx
    client
    How to solve bulk/multi operations?
    nginx in front
    { "index" : { "_index" : "test1", "_type" : "type1", "_id" : "1" } }
    { "field1" : "value1" }
    { "delete" : { "_index" : "test2", "_type" : "type1", "_id" : "2" } }
    { "create" : { "_index" : "test3", "_type" : "type1", "_id" : "3" } }
    { "field1" : "value3" }
    { "update" : {"_id" : "1", "_type" : "type1", "_index" : "test4"} }
    { "doc" : {"field2" : "value2"} }
    

    View Slide

13. ES
    nginx
    client
    Prevent unwanted accesses
    nginx in front
    HTTP/Transport
    

    View Slide

14. ES
    nginx
    client
    nginx in front
    Firewall
    

    View Slide

15. Data
    ACL
    client
    Configuration scattered across systems
    operational overhead
    IP Filtering
    

    View Slide

16. Data
    ACL
    client
    operational overhead
    IP Filtering Directory
    Configuration scattered across systems
    

    View Slide

17. Why?
    How?
    Next?
    What?
    Who?
    Q & A
    

    View Slide

18. How?
    Elasticsearch modular & pluggable
    Security as a plugin
    HTTP + Transport protocols
    Integration into the ELK stack!
    

    View Slide

19. How?
    Elasticsearch
    Elasticsearch
    auth_token Authentication
    Authorization
    

    View Slide

20. How?
    Elasticsearch
    Elasticsearch
    auth_token
    200 OK
    Authentication
    Authorization
    

    View Slide

21. How?
    Elasticsearch
    Elasticsearch
    auth_token
    401 Unauthorized
    Authentication
    Authorization
    

    View Slide

22. How?
    Getting up and running is easy
    Install elasticsearch 1.6
    bin/plugin install elasticsearch/license/latest
    bin/plugin install elasticsearch/shield/latest
    

    View Slide

23. Why?
    How?
    Next?
    What?
    Who?
    Q & A
    

    View Slide

24. What?
    IP Filtering
    Encrypted communication
    Authentication
    Authorization
    Audit Trail
    

    View Slide

25. Configurable in elasticsearch.yml
    Can be updated dynamically via
    cluster update settings API
    IP Filtering
    shield.transport.filter:
    allow: "192.168.0.1"
    deny: "192.168.0.0/24"
    

    View Slide

26. keystore required
    different config for HTTP and
    transport protocol (+profiles)
    Encrypted communication
    shield.ssl.keystore.path: /path/to/keystore.jks
    shield.ssl.keystore.password: secret
    shield.transport.ssl: true
    shield.http.ssl: true
    

    View Slide

27. Authentication
    "Who are you?"
    Auth mechanisms are called realms
    Available: esusers, ldap, ad, pki
    Realms can be chained
    Support for caching & API for clearing
    

    View Slide

28. Authentication
    shield.authc:
    realms:
    esusers:
    type: esusers
    order: 0
    ldap1:
    type: ldap
    order: 1
    enabled: false
    url: 'url_to_ldap1'
    ...
    ad1:
    type: active_directory
    order: 3
    url: 'url_to_ad'
    

    View Slide

29. ESusers realm
    Local files, can be changed via CLI
    Elasticsearch watches file changes &
    reloads
    config/shield/users
    config/shield/users_roles
    

    View Slide

30. ESusers realm
    bin/shield/esusers useradd alex
    bin/shield/esusers roles alex -a
    admin -r user
    bin/shield/esusers list
    bin/shield/esusers userdel alex
    

    View Slide

31. Fallback to configurable user
    Disabled by default
    Anonymous access
    shield.authc:
    anonymous:
    username: anonymous_user
    roles: role1, role2
    

    View Slide

32. Authorization
    "Are you allowed to do that?"
    File: config/shield/roles.yml
    admin:
    cluster: all
    indices:
    '*': all
    

    View Slide

33. Role Based Access Control
    role
    named set of permissions
    permission
    set of cluster wide privileges
    set of indices/aliases specific privileges
    privilege
    set of one or more action names
    /_search ⬌ indices:data/read/search
    

    View Slide

34. Role Based Access Control
    admin:
    cluster: all
    indices:
    '*': all
    role permission
    

    View Slide

35. Authorization
    user:
    indices:
    '*': read
    events_user:
    indices:
    'events_*': read
    

    View Slide

36. Authorization
    get_user:
    indices:
    'events_index': 'indices:data/read/get'
    logfile_user_readonly:
    indices:
    "logstash-201?-*": read
    

    View Slide

37. Audit Trail
    Writes an own audit log file
    Implemented as logger
    Logs different types of event based
    on log level
    (ip filtering, tampered requests, access denied, auth failed)
    shield.audit.enabled: true
    

    View Slide

38. Integration
    Transport Client
    Logstash
    Kibana 3/4
    Watcher
    Marvel
    

    View Slide

39. Transport Client
    TransportClient client = new TransportClient(builder()
    .put("cluster.name", "myClusterName")
    .put("shield.user", "test_user:changeme")
    .put("shield.ssl.keystore.path", "/path/to/client.jks")
    .put("shield.ssl.keystore.password", "password")
    .put("shield.transport.ssl", "true"))
    .addTransportAddress(new
    InetSocketTransportAddress("localhost", 9300));
    

    View Slide

40. Why?
    How?
    Next?
    What?
    Who?
    Q & A
    

    View Slide

41. Who?
    Use-case 1: Monitoring application
    No write access
    Cluster Health
    Nodes stats/info
    Indices Stats
    

    View Slide

42. Use-case 2: Logstash
    No read access (unless input is used)
    Indices: Indexing
    Cluster: Index templates
    

    View Slide

43. Use-case 3: Marvel
    marvel_user:
    cluster: cluster:monitor/nodes/info,
    cluster:admin/plugin/license/get
    indices:
    '.marvel-*': all
    marvel_agent:
    cluster: indices:admin/template/get,
    indices:admin/template/put
    indices:
    '.marvel-*': indices:data/write/bulk, create_index
    

    View Slide

44. Use-case 4: Ecommerce
    bulk:
    indices:
    'products_*': write, manage, read
    updater:
    indices:
    'products': index, delete, indices:admin/optimize
    webshop:
    indices:
    'products': search, get
    

    View Slide

45. Use-case 4: Ecommerce
    monitoring:
    cluster: monitor
    indices:
    '*': monitor
    sales_rep :
    indices:
    'sales_*' : all
    'social_events' : data_access, monitor
    

    View Slide

46. Why?
    How?
    Next?
    What?
    Who?
    Q & A
    

    View Slide

47. Next?
    Simplify SSL configuration
    API driven user/role management
    Open up realms API
    Field-level security
    Index Audit Trail into ES
    

    View Slide

48. Why?
    How?
    Next?
    What?
    Who?
    Q & A
    

    View Slide

49. Q & A
    Thanks for listening!
    Alexander Reelsen
    @spinscale
    alex@elastic.co
    We're hiring
    https://www.elastic.co/about/careers
    We're helping
    https://www.elastic.co/subscriptions
    

    View Slide

50. Resources
    Shield documentation
    https://www.elastic.co/guide/en/shield/current/index.html
    Shield: Security in ELK
    https://www.elastic.co/elasticon/2015/sf/security-in-elk
    Shield and Beyond:
    Recommendations for a Secure ELK
    Environment
    https://www.elastic.co/webinars/shield-and-beyond
    

    View Slide

51. Resources
    https://discuss.elastic.co/c/shield
    

    View Slide

52. Resources
    

    View Slide

53. Resources
    

    View Slide

54. Q & A
    Thanks for listening!
    Alexander Reelsen
    @spinscale
    alex@elastic.co
    We're hiring
    https://www.elastic.co/about/careers
    We're helping
    https://www.elastic.co/subscriptions
    

    View Slide