KCD Czech & Slovak 2024 - Highly available identity and access management with Keycloak in the cloud

Transcript

1. Highly available Identity and Access Management with Keycloak in the cloud Michal Hajas | Principal Software Engineer | Red Hat KCD Czech & Slovak | 2024-06-07 Slides are based on a previous talk of Alexander Schwartz and Ryan Emerson
2. What is Identity and Access Management (IAM), and do I need one?
3. Authenticate and authorize users for services Login Request Verify token < Token > API Cloud Services
4. Benefits of using Keycloak • Authenticate only once per day • Integrate LDAP and Kerberos • Brokering to existing OIDC/SAML services • Integrate existing custom stores
5. Keycloak provides the login screen for your apps
6. Use various user sources
7. Eliminate daily churn • User password recovery (even when using LDAP) • Configure second factor authentication • Self-registration for users • User data self-management
8. Password recovery and self-registration
9. Keycloak is now a critical component You want it to be available 24/7.
10. N Keycloak Pods Clustered session state Database Persistent State Single Availability Zone • Deploy with Kubernetes • Sessions replicated between Keycloak processes • Can tolerate K8s Node/Pod failures
11. Tolerating Availability Zone Failures
12. Multi Site Availability Zone 1 Availability Zone 2 Persistent State ? ? • Deploy Keycloak to multiple availability zones • How to maintain session and persistent state?
13. Multi Site - Active/Passive AZ-1 Active AZ-2 Passive • Keycloak site “Active” or “Passive” • All user requests forwarded to Active site • Greatly simplifies write semantics
14. Multi Site - Active/Passive AZ-1 Active AZ-2 Active • Passive becomes Active after failover • Users connect to backup only after failover
15. Managing User Connections
16. User Connections Health Check Health Check • All users requests via Route53 hostname • Route53 determines which site is “Active” • Requests always routed to Active site • Periodic Health checks determine site health AZ-1 AZ-2
17. User Connections AZ-1 AZ-2 Health Check Health Check • On failover DNS routing updated and new Active site established • DNS caching can lead to longer failover times for some users
18. Session Failover
19. Infinispan • In-memory Key/Value Cache • Advanced clustering capabilities • Kubernetes Operator • Apache 2.0 License App App • Embedded Mode • Client/Server
20. Active/Passive Session Replication AZ-1 Active AZ-2 Passive Session writes replicated across availability zones • Utilise Infinispan Server Read/Writes Writes • Infinispan Cross-Site replication used to sync session state • Infinispan server supports advanced admin operations for Cross-Site failover management • Session data now survives Keycloak pod restarts
21. Database Failover
22. AWS Region Aurora DB Aurora Availability Zone 1 Writer Availability Zone 2 Reader Reads Writes Reads • All read/writes handled by a single “Writer” instance • Writer instance hosted in same AZ as Active site • Data written to both Availability Zones to allow failover
23. AWS Region Aurora DB Aurora Availability Zone 1 Writer Availability Zone 2 Writer Reads Writes Reads • Election of new writer instance managed by Aurora • Failover takes ~ 1m • AWS JDBC wrapper ensures Keycloak pods drop old connections on failover • No additional Keycloak semantics required Writes
24. Architecture Overview
25. AWS Region Availability Zone 1 Availability Zone 2 HA Architecture Connect to Route53 DNS Forward req to Active Keycloak Sessions replicated across AZs Aurora multi-az replication Failover to Passive
26. Surprising system behaviors under load
27. Overload situations requiring load shedding • Overload: More requests incoming than can be handled • Observation: Requests queue up, memory usage increases, client requests time out. • Remedy: Drop requests by replying with 503 Queue length > max? Eneue and process request Respond with result 503 immediately no yes
28. Cache stampede protection when restarting pods • Cache stampede: When restarting under high load, parallel requests access the database while the cache is empty • Observation: Timeouts, exhaustion of DB connections. • Remedy: JVM locking if the same resource is about to be fetched from the database. Pending request? Fetch information from database Block for pending request to return no yes
29. Tackling blocking probes and metrics • Overload: Too many requests make responses slow or lead to load shedding. • Observation: Blocking probes stop working and Pod restart. Metrics become unavailable. • Remedy: Use non-blocking probes that don’t enqueue in an overload situation. Disable load-shedding for metrics. Liveness probe failed / Timeout / n times in the last x seconds Container xxx failed liveness probe, will be restarted Symptoms: Kubernetes events similar to:
30. Good habits
31. Up-to-date documentation • Deploying Keycloak • Creating datasets • Load running
32. Ephemeral environments You can have as many environments as you want, but they will be deleted automatically at the end of the day.
33. Measure, record and repeat • Add Metrics collection and searchable logs as early as possible. • Capture all insights after a run automatically for an ephemeral environment is hard but safes time in the long run. • Nightly runs for performance and functional tests show if there are regressions.
34. Tools that were most helpful
36. The Future
37. Persistent sessions Database Persistent State Keycloak Users Infinispan does not store sessions data https://github.com/keycloak/keycloak/discussions/28271
38. Future roadmap for Keycloak HA and the cloud Ideas for enhancements (but no planned releases): • Support active-active deployments and evaluate possible benefits • Simplify deployment and upgrades • Support more cloud platform and databases • Provide security-hardened blueprints
39. • Keycloak https://www.keycloak.org/ • Keycloak Benchmark Project https://www.keycloak.org/keycloak-benchmark/ • Keycloak High Availability Guide https://www.keycloak.org/high-availability/introduction • Keycloak Book 2nd Edition https://www.packtpub.com/product/kc/9781804616444 • Infinispan https://infinispan.org/ Links
40. Contact Michal Hajas Principal Software Engineer mhajas@redhat.com github.com/mhajas
41. Questions and answers