Shield

While it has always been possible to secure Elasticsearch clusters by deploying them within well-secured environments, we continuously received requests from customers and users to have a more integrated solution. In response, we created Shield, our first security plugin for Elasticsearch, which went GA in late January.

In this talk, we’ll deep dive into this first release Shield. We went to great lengths preparing Elasticsearch itself for security, not just on its extensibility side, but also carefully rethinking how the data flows in it. We’ve built a foundation that not only delivers immediate tangible value when it comes to securing Elasticsearch clusters, but also enables us to extend its functionality incrementally and rapidly over time.

Among other topics, we’ll cover:
*Authentication
*Authorization
*Encrypted Communication & Node Authentication
*IP Filtering
*Audit Trails

Attendees will leave this talk with a solid understanding of Shield’s functionality, architecture and why it’s the best possible tool to secure Elasticsearch and your ELK cluster.

Transcript

1. shield Uri Boness
2. { } 2 Why Security?
3. { } 3
4. { } Securing Elasticsearch • Nginx as a reverse proxy • IP tables • Use aliases + filters 4
5. { } 5 Shield
6. { } 6 SaaP
7. { } 7 Security as a Plugin
8. { } Shield Lifetime 8 2015 Q4 Q1 Q2 Shield 1.0 GA Shield 1.1
9. { } 9 Shield Authentication Authorization Securing the Wire Cluster Fencing Audit Trail
10. { } 10 Authentication
11. { } 11 (a.k.a) authc
12. { } 12 (a.k.a) who are you? & are you really who you claim to be?
13. { } The User • Represents the actor behind an API call – identified by a principal (username) – has credentials (e.g. password) • Users have roles • relates to authorization… so we’ll discuss it later • fyi, users without roles can’t do much… 13
14. { } 14 Realms
15. { } Realms 15 /_api Realm Shield User auth_token please execute this api… oh.. and here’re my credentials
16. { } Realms 16 please authentication auth_token Realm Shield User auth_token
17. { } Realms 17 Realm Shield User failure
18. { } Realms 18 Realm Shield User 401 failure
19. { } Realms 19 Realm Shield User user success
20. { } Realms 20 user Realm Shield User authorize
21. { } Realms 21 Shield Realm Realm Realm auth_token failure auth_token failure user auth_token
22. { } Realms • esusers – native file based user store • LDAP – integrating in the enterprise • Active Directory – more enterprises 22
23. { } esusers • Simple & Native – file based, htpasswd-like config (bcrypt hashing) – no dependency on 3rd party systems • esusers tool will assist you manage users • Files are auto-loaded when changed • Great for small to medium deployments 23
24. { } LDAP (you know.. for the enterprise) • Authenticate (bind) to LDAP server • Translate LDAP groups of user to Shield roles – using a dedicated role_mapping.yml file • Shield 1.1 – is able to search for the user before the bind – can load-balance between multiple LDAP servers 24
25. { } Active Directory 25 same as LDAP… with simplified settings
26. { } Anonymous User (Shield 1.1) “default” user to take the anonymous user place 26 Realm Shield auth_token failure
27. { } Anonymous User (Shield 1.1) 27 Realm Shield authorize anonymous_user
28. { } 28 Shield Authentication Authorization Securing the Wire Cluster Fencing Audit Trail
29. { } 29 Authorization
30. { } 30 (a.k.a) authz
31. { } 31 (a.k.a) are you allowed to do that?
32. { } Role Based Access Control (RBAC) • role - a named set of permissions • permission – a set of cluster wide privileges – a set of privileges on one or more indices/aliases • privilege - a set of one or more action names 32
33. { } Actions 33 /_search → indices:data/read/search API action name
34. { } Users & Roles • Users have roles • The permission of the user are the combined permission of all its roles • Roles are defined in roles.yml 34
35. { } Users & Roles 35
36. { } 36 Shield Authentication Authorization Securing the Wire Cluster Fencing Audit Trail
37. { } 37 Securing the Wire
38. { } Data Travels • Network Channels – transport: inter-node communication – http: REST interface • MITM (man-in-the-middle) – sniffing, tampering data, imposture 38
39. { } SSL/TLS • This wheel was already invented • Based on native Java support • Integrated in all supported network channel – http/s & transport 39
40. { } Node2 Self-Signed Certs 40 Node1 we generate a self-signed cert for every node
41. { } Node2 Self-Signed Certs 41 Node1 c1 c2 we generate a self-signed cert for every node
42. { } Node2 Self-Signed Certs 42 Node1 c1 c2 for every node, we import its generated cert on all other nodes
43. { } Node2 Self-Signed Certs 43 Node1 c1 c2 for every node, we import its generated cert on all other nodes TLS handshake
44. { } Node2 Self-Signed Certs 44 Node1 c1 c2 the challenge is the elastic nature of… well… elasticsearch TLS handshake
45. { } Node2 Self-Signed Certs 45 Node1 c1 c2 the challenge is the elastic nature of… well… elasticsearch Node 3
46. { } Node2 Self-Signed Certs 46 Node1 c1 c2 the challenge is the elastic nature of… well… elasticsearch Node 3 c2 c1
47. { } Node2 Self-Signed Certs 47 Node1 c1 c2 the challenge is the elastic nature of… well… elasticsearch Node 3 c2 c1
48. { } Node2 Self-Signed Certs 48 Node1 c1 c2 the challenge is the elastic nature of… well… elasticsearch Node 3 c3 c2 c1 c3
49. { } Node2 Self-Signed Certs 49 Node1 c1 c2 the challenge is the elastic nature of… well… elasticsearch c3 c3 Node 3 c2 c1
50. { } Self-Signed Certs 50 the challenge is the elastic nature of… well… elasticsearch handshake Node2 Node1 c1 c2 c3 c3 Node 3 c2 c1
51. { } Self-Signed Certs 51 the challenge is the elastic nature of… well… elasticsearch Node2 Node1 c1 c2 c3 c3 Node 3 c2 c1 requires cluster restart handshake
52. { } Node2 CA-Signed Certs 52 Node1 we find a CA
53. { } Node2 CA-Signed Certs 53 Node1 we find a CA CA
54. { } Node2 CA-Signed Certs 54 Node1 c1 c2 we generate a cert for every node CA
55. { } Node2 CA-Signed Certs 55 Node1 c1 c2 we ask the CA to sign it CA
56. { } Node2 CA-Signed Certs 56 Node1 c1 c2 we ask the CA to sign it CA
57. { } Node2 CA-Signed Certs 57 Node1 c1 c2 we ask the CA to sign it CA
58. { } ca ca Node2 CA-Signed Certs 58 Node1 we also get the CA cert c1 c2 CA
59. { } Node2 CA-Signed Certs 59 Node1 we also get the CA cert ca ca c1 c2 CA
60. { } Node2 CA-Signed Certs 60 Node1 we can now open a secure channel between the nodes ca ca c1 c2 TLS handshake CA
61. { } Node2 CA-Signed Certs 61 Node1 it’s elastic! ca ca c1 c2 CA
62. { } CA-Signed Certs 62 it’s elastic! Node1 ca c1 Node2 ca c2 Node 3 CA
63. { } CA-Signed Certs 63 it’s elastic! Node1 ca c1 Node2 ca c2 Node 3 c3 ca CA
64. { } CA-Signed Certs 64 it’s elastic! Node1 ca c1 Node2 ca c2 Node 3 c3 ca handshake CA
65. { } CA-Signed Certs 65 it’s elastic! Node1 ca c1 Node2 ca c2 Node 3 c3 ca no cluster restart required! handshake CA
66. { } 66 Shield Authentication Authorization Securing the Wire Cluster Fencing Audit Trail
67. { } 67 Cluster Fencing
68. { } Node to Node Authentication • Allow/Deny nodes trying to connect to the cluster – unknown nodes may be malicious • Relying on SSL/TLS – if a node has a valid certs, access granted 68
69. { } Client to Node Authentication • Allow/deny connections from clients • usually SSL/TLS is a one way protection – the client trusts the server • with client certs, it is a two way protection – the client trusts the server – the server trusts the client 69
70. { } Transport Profiles • Added to elasticsearch 1.4 • Enables binding to multiple host:port pairs • Enables clear separation between transport client and inter node communication • client certs within the cluster • transport client doesn’t need client certs 70
71. { } IP Filtering • A simple alternative to IP tables • Control who can connect to the cluster based on hosts, IPs and CIDRs allow/deny rules. • Built into the internal networking layers in elasticsearch – transport & http • Shield 1.1 lets you update the filters dynamically 71
72. { } 72 Shield Authentication Authorization Wire Encryption Cluster Fencing Audit Trail
73. { } 73 Audit Trail
74. { } • Records the activity on each node in the cluster – anonymous_access_denied – authentication_failed – access_denied /access_granted – connection_denied / connection_granted 74 Audit Trail
75. { } Access Logs • An audit trail that logs activity to log files • Utilizes the existing logging infrastructure • separate access.log files 75
76. { } 76 What’s Next?
77. { } Shield 1.1 • Anonymous access • Enhanced LDAP support • Dynamically update IP filters 77
78. { } Shield X.X • Additional Realms – new native realm managed via API – kerberos, other SSO mechanisms • Additional authentication methods • API token/key authentication • Enhanced support for kibana 78
79. { } thank you! @uboness
80. { } 80