Joining Cloud VMs to FreeIPA Partly Cloudy IPA Fraser Tweedale Principal Software Engineer 1

What we’ll discuss today Agenda 2 ▸ The problem: pain-free identity management in hybrid cloud envs ▸ Solution overview: the Podengo project ▸ Demo time! ▸ Gory technical details ▸ Gaps, future directions, opportunities

Preliminaries 3 ▸ I work in the Identity Management team at Red Hat ▸ The Podengo project is the hard work of a small sub-team, assisted by many collaborators (service delivery, UX, docs, …) ▸ This presentation is also a collaboration Introductions

Preliminaries 4 ▸ A basic understanding of cloud computing: cloud providers and VMs ▸ Basic identity management concepts: hosts and users, SSH, HBAC Assumed knowledge

What problem are we trying to solve? 5

Problem description 6 So you launched a VM… ▸ How do you authenticate to it? (most often: SSH keys) ▸ How does it authenticate to other machines / services? ▸ What if many users need to access the machine / workload? ▸ What if someone leaves the company or you have to revoke access? ▸ How do you enforce access policies? Cloud VMs

Problem description 7 ▸ Just use SSH keys - doesn't scale well ▸ SSH certificates - scales well, but requires special-purpose PKI ▸ Privileged Account Management - 3rd party [commercial] solutions ▸ Corporate IdM (FreeIPA, AD) - need to enrol clients somehow ▸ Corporate cloud-based IdM (Entra ID) - host authentication techniques not mature Identity management approaches for cloud VMs

Problem description 8 ▸ Just use SSH keys - doesn't scale well ▸ SSH certificates - scales well, but requires special-purpose PKI ▸ Privileged Account Management - 3rd party [commercial] solutions ▸ Corporate IdM (FreeIPA, AD) - need to enrol clients somehow ▸ Corporate cloud-based IdM (Entra ID) - host authentication techniques not mature Identity management approaches for cloud VMs

Problem description 9 ▸ Manage users, hosts, services, access policies ▸ 2FA, passkeys, smart card, SSH keys, external IdP, password ▸ Active Directory trust ▸ Linux-specific policies: Sudo rules, SELinux user maps FreeIPA / Red Hat Identity Management

Domain using Red Hat Identity Management (IPA) Domain using Red Hat Identity Management (IPA) Joining cloud VMs - today New VMs are not in the IPA domain - no user access except via SSH keys and no policy enforcement Newly provisioned host running on the cloud Existing hosts in the org Existing hosts in the organisation Hosts joined to the domain recognise org users and enforce security policies Newly provisioned host running on the cloud Existing hosts in the org Existing hosts in the org Newly provisioned host running on the cloud Existing hosts in the org Newly provisioned hosts on cloud Newly provisioned host running on the cloud Existing hosts in the org Newly provisioned hosts on cloud ● Less secure during time delay to join ● Manual intervention or use 3rd party config management solution ● Sensitive (admin) credentials req’d This is the problem Problem description

Problem description 11 ▸ Reduce complexity and cost of robust identity management in cloud environments ▸ Let companies use their existing IdM to enable easy and safe transition to hybrid cloud environment ▸ Don't sacrifice security in the name of convenience The bottom line

Podengo and Red Hat Hybrid Cloud Console 12 Solution overview

Solution overview 13 ▸ Hosted services to manage Red Hat environments ▸ For RHEL: Red Hat Insights, subscription management, inventory, CVEs, remediations, images ▸ Management features for other products: Ansible, OpenShift, … ▸ Supported cloud providers: AWS, Azure, Google Cloud, Oracle Cloud Red Hat Hybrid Cloud Console console.redhat.com

Solution overview 14 1. Register your [Free]IPA deployment with Hybrid Cloud Console 2. Build images containing the client RPMs 3. Launched VMs get introduced to IPA, and securely enrol A solution in three acts

Solution overview 15 ▸ Directory & Domain Services: name of registration service in HCC ▸ Domain Join [on launch]: common name for our HCC feature ▸ idmsvc: internal name of the hosted service component(s) ▸ Podengo: open source project used to implement the above Terminology

Newly provisioned hosts in their cloud immediately* join their domain without any further user intervention. No credentials seen by HCC Automatic and immediate Domain Join - benefits Launched VMs communicate securely with HCC and the IPA server. Join cloud VMs to the organisation's existing identity management system Leverage existing IAM Solution overview

Demo 17

How does it work? 18

Implementation 19 ▸ idmsvc-backend: service component running on HCC (Golang) ▸ idmsvc-frontend: service UI (React / PatternFly / TypeScript) ▸ ipa-hcc-server: enrollment agent plugin for IPA server ▸ ipa-hcc-client: client package with auto-join behaviour Components

Implementation 20 ▸ Portuguese podengo - a dog with three sub-breeds (a la Kerberos) ▸ Pod (containers) + Go (language) ▸ Every project should have a cute mascot! ▸ https://github.com/podengo-project Podengo project https://commons.wikimedia.org/wiki/File:Podengo _podengo_portobello_sitting.jpg Public domain

21 Architecture overview idmsvc backend service IPA server Control Plane (HCC) Data Plane (customer site / cloud) REST service DB ipa-hcc-server IPA API Client VM ipa-hcc-client /host-conf /domains /token idmsvc frontend Registration wizard Registration workflow (1.x) Join workflow (2.x) 1.1. request registration token 1.2. user executes ipa-hcc register 1.3. register IPA server API call) 1.4. store registration 2.1. get enrollment token (req) IPA client 2.2. create enrollment token 2.3. (resp) 2.4. request join (authz by token) 2.5. ipa host-add 2.6. invoke ipa-client-install 2.7. join domain (get keytab, etc)

Implementation 22 ▸ HCC UI: registration "wizard" requests token bound to org account ▸ IPA server: user runs ipa-hcc register ･ POST authenticated by RHSM client certificate ▸ HCC UI: user returns to complete registration and enable auto-join Registration workflow

Implementation 23 Registration workflow - token ▸ Token binds org id, domain type, and expiration time ▸ Validation: exp. time from token, domain type from request, org id from session ▸ Ephemeral tokens ; store nothing until registration POST ▸ Deterministic domain UUID ; UI can poll backend for registration status

Implementation 24 ▸ Image must include ipa-client and ipa-hcc-client RPMs ▸ After subscription-manager registration, auto-join script fires ▸ Query backend /host-conf resource; receive enrollment token ･ Authenticated by subscription-manager client certificate ▸ Present token to IPA server; IPA server creates host entry ▸ Client joins the domain (PKINIT-authenticated keytab retrieval) Join workflow

Implementation 25 ▸ JSON Web Token (JWT) signed by backend service ▸ Additional claims: org id, domain uuid, host inventory uuid, FQDN ▸ ECDSA signing, periodic key rotation ▸ IPA server verifies signature, expiry, domain uuid and org match configuration and client cert ▸ New host entry DN based on FQDN claim Join workflow - enrollment token

Implementation 26 Backend API GET /api/idmsvc/v1/domains List domains in the organisation POST /api/idmsvc/v1/domains Register a domain POST /api/idmsvc/v1/domains/tokens Get a domain registration token GET /api/idmsvc/v1/domains/{uuid} Get a registered domain DELETE /api/idmsvc/v1/domains/{uuid} Delete a registered domain PATCH /api/idmsvc/v1/domains/{uuid} Update a registered domain (user-editable fields) PUT /api/idmsvc/v1/domains/{uuid} Refresh domain operational data GET /api/idmsvc/v1/signing_keys Retrieve signing keys POST /api/idmsvc/v1/host-conf/{inventory_id}/{fqdn} Acquire domain join authority for VM OpenAPI spec: https://github.com/podengo-project/idmsvc-api/blob/main/public.openapi.yaml

Implementation 27 ▸ Several things have to be "just right" for this to work ▸ HCC and IPA server must be reachable from the cloud environment ▸ DNS, routes and firewalls can all cause problems ▸ IPA uses lots of ports for lots of protocols: https, ldap, ldaps, kerberos, kpasswd, dns, … ▸ Clocks have to be in sync ▸ tl;dr it's always DNS Troubleshooting

Status, gaps, and possible futures 28

Status and future 29 ▸ Feature is in production on Hybrid Cloud Console - preview mode ▸ ipa-hcc RPMs are in Fedora and EPEL (RHEL later) ▸ Documentation is published but needs expansion ･ Cloud provider-specific onboarding guides to come ▸ Collecting metrics and user / customer feedback to inform next steps ▸ Only one active domain per org Current status

Status and future 30 ▸ Add Active Directory support ･ Expand solution to more organisations ▸ Manage domain enrolment from Insights Inventory ･ Expand solution to more use cases ▸ Verify / assist users with cloud environment set up ･ Improve user success without expanding scope Big ideas What comes next?

31 Architecture overview (IPA) idmsvc backend service IPA server Control Plane (HCC) Data Plane (customer site / cloud) REST service DB ipa-hcc-server IPA API Client VM ipa-hcc-client /host-conf /domains /token idmsvc frontend Registration wizard Registration workflow (1.x) Join workflow (2.x) 1.1. request registration token 1.2. user executes ipa-hcc register 1.3. register IPA server (API call) 1.4. store registration 2.1. get enrollment token (req) IPA client 2.2. create enrollment token 2.3. (resp) 2.4. request join (authz by token) 2.5. ipa host-add 2.6. invoke ipa-client-install 2.7. join domain (get keytab, etc)

Update confidential designator here 32 Architecture sketch (AD support) RHEL machine idmsvc backend service AD domain controller Control Plane (HCC) Data Plane (customer site / cloud) REST service DB ad-hcc-server AD Client VM ad-hcc-client /host-conf /domains /token idmsvc frontend Registration wizard Registration workflow (1.x) Join workflow (2.x) 1.1. request registration token 1.2. user executes ad-hcc register 1.3. register AD deployment (API call) 1.4. store registration 2.1. get enrollment token (req) AD machinery 2.2. create enrollment token 2.3. (resp) 2.4. request join (authz by token) 2.5. Add machine account (prepare for join) (offline-join?) 2.6. perform AD join 2.7. join domain (get keytab, etc)

Status and future 33 ▸ Image Builder integration ･ Make it easier to prepare image ･ Increase visibility of the Domain Join feature ▸ Support for multiple domains ･ Currently fails if >1 (active) registered domain ･ Start with basic heuristics (e.g. match VM DNS domain to registered deployments) Smaller ideas What comes next?

Status and future 34 Image Builder integration (sketch) idmsvc-backend REST service DB /host-conf /domains /token Image Builder service Has registered IdP(s)? Show checkbox to include ipa-hcc-client Don't show checkbox for ipa-hcc-client Yes No

Status and future 35 ▸ Hosts consume console.redhat.com user identities ▸ Single unified identity domain ▸ Option 1: IPA with External IdP (requires IPA) ▸ Option 2: localkdc (no IPA, hosted IdP -> reduced effort and cost) ･ Enable POSIX system login from cloud / web SSO ･ Upcoming FOSDEM presentation (2025-02-02): https://fosdem.org/2025/schedule/event/fosdem-2025-561 8-localkdc-a-general-local-authentication-hub/ A grand vision

Status and future 36 ▸ Our architecture** is not tightly bound to HCC ･ **shameful truth: the code kinda is… ･ HCC: hosts the idmsvc, authenticates clients ▸ What is required to use Podengo in other contexts? ･ X.509 certs for backend/IPA/PKINIT authentication ･ OR some other way to authenticate VMs + extend VM->IPA protocol to enable OTP join ▸ Got a use case? Please tell us about it! (GitHub issue, mailing list) Non-Insights/HCC applications

37 Architecture overview (authentication) idmsvc backend service IPA server Control Plane (HCC) Data Plane (customer site / cloud) REST service DB ipa-hcc-server IPA API Client VM ipa-hcc-client /host-conf /domains /token idmsvc frontend Registration wizard 1.4. store registration IPA client 2.2. create enrollment token 2.5. ipa host-add 2.6. invoke ipa-client-install TLS cert authn 3Scale proxy: user authnz, TLS cert authn (VMs) User

Conclusion 38

Conclusion 39 ▸ Official docs: Deploying and managing RHEL systems in hybrid clouds | Red Hat Product Documentation ▸ github.com/podengo-project ▸ EO2024 talk: Passwordless Linux FreeIPA - Passkey and External IdP login with FreeIPA ▸ EO2023 talk: Kerberos PKINIT (video ; slides) ▸ Mailing list: [email protected] ▸ This slide deck: is.gd/partly_cloudy_ipa Resources

Questions? 40 https://commons.wikimedia.org/wiki/File:Three_Weavers_Cloud_City_Hazy_IPA.jpg CC-BY-4.0 (no changes)