Secure AWS Access from On-Prem Kubernetes Using EKS Pod Identity and IAM Roles Anywhere

https://event.cloudnativedays.jp/cndw2025/talks/2720

Transcript

1. Secure AWS Access from On-Prem Kubernetes Using EKS Pod Identity and IAM Roles Anywhere CloudNative Days Winter 2025 Money Forward, Inc. Platform and Reliability Engineering Department Kota Yagi
2. Agenda • Company & Tech Stack Overview • Challenge & Solution Overview • Solution Details ◦ IAM Roles Anywhere Mechanics ◦ EKS Pod Identity Mechanics ◦ Custom Pod Identity Mechanics • Summary
3. Money Forward, Inc. 3 Company and Tech Stack Overview 01
5. Main Services and Divisions ・Exceeded 17 million users ・No. 1 recognition rate, and utilization rate among household accounting applications and asset management applications1 A division that creates services in cooperation with clients, mainly financial institutions. Developed "Mikatano" (a business DX service for small to medium local enterprises delivered through local financial institutions), "BANK APP" (a banking application), etc. A service that visualizes money that anyone can continue with ease. Introduces centralized management to household budgets and assets to let the user grasp money flow and the current state of the assets, thereby supporting the initial step in eliminating financial anxiety ・The number of paying customers exceeded 410,000 ・No. 1 satisfaction among cloud accounting softwares1 ・No. 1 share within SaaS payroll management2 A SaaS style platform for businesses that connects back office data of various kinds and thereby optimizes tedious tasks related to accounting, human resources and labour affairs, and legal affairs 1. Delegated research by Macromill, Inc. / Survey target and number of respondents: 1,034 household accounting app users and 1,034 asset management app users in their 20s to 60s / Survey period: Household accounting app - August 13, 2024 to August 14, 2024; Asset management app - August 13, 2024 to August 16, 2024 / Survey method: Internet research 1. July 2024, JustSystems Co., Ltd. Fastask survey (responses fewer than 30 excluded). 2. Source: ITR "ITR Market View：人事・給与・就業管理市場2025" Payroll Management Market - Less than 100 Employees: Sales Revenue and Share by Vendor (FY2022-FY2024 Forecast) Business Individuals Financial Institutions
6. Product Lineup POINT Corporates Individuals Financial Institutions Our products and services span across the B2B, B2C and other domains. POINT 出典元 : 株式会社マネーフォワード 会社紹介資料 Business Finance Home X Money Forward for Financial Institutions & Specific Services Digital Passbook & Easy Passbook
7. Technology Stack Developm ent Language Frontend JavaScript/TypeScript Backend Ruby/Go/Kotlin/Java/Python/Rust Mobile Android: Kotlin/Java iOS: Swift/Objective-C Cross Platform: Dart (Flutter) Infrastructure AWS/GCP/EKS/ECS/Kubernetes (on-premises) Database MySQL/MongoDB/BigQuery Security WAF/EDR/SIEM/CSPM/SAST/DAST Quality Assurance TestRail/Autify/MagicPod/mabl/Gauge/Selenium/ Postman/DataDog Surveillance, Monitoring DataDog/PagerDuty Environment Terraform/ansible Continuous Integration, Continuous Delivery CircleCI/GitHub Actions/Bitrise/ArgoCD GitHub Actions
8. Money Forward, Inc. 8 Challenge and Solution Overview 02
9. About On-Prem Kubernetes • On-Prem Kubernetes Cluster (Production) ◦ Nodes: 23 (control plane: 3, etcd: 5, worker node: 15) ◦ Pods: +350 ◦ Namespaces: +35 ◦ Workloads: Microservices + Batch ◦ Add-ons: Ingress NGINX Controller, ExternalDNS, MetalLB, etc…
10. Problem in On-Prem Kubernetes On-Prem K8s workloads used Long-Term IAM User Credentials for AWS access Problem: • High Security Risk: Severe impact if leaked • Operational Toil: Requires tedious manual, periodic rotation
11. AWS IAM Best Practices • Use temporary security credentials (IAM roles) instead of creating long-term credentials (IAM User) • Follow Security best practices in IAM
12. Challenge: Using IAM Roles from K8s Workload Outside of AWS Outside AWS(On-Prem K8s): How can we assume an IAM Role securely? • Outside AWS: Solved by IAM Roles Anywhere • Inside AWS(EKS): Solved by EKS Pod Identity 👉 Outside AWS(On-Prem K8s): Solved by a combination of IAM Roles Anywhere + EKS Pod Identity ?
13. Idea: EKS Pod Identity × IAM Roles Anywhere Custom Pod Identity developed by combinating IAM Roles Anywhere and EKS Pod Identity • IAM Roles Anywhere ◦ Provides a way to get temporary credentials for a workload or process that runs outside of AWS, which means we can get around the limitation of EKS Pod Identity(work on EKS). • EKS Pod Identity ◦ Allows Kubernetes pods running on EKS to securely access AWS services by assigning IAM roles to pods.
14. Custom Pod Identity Overview
15. Money Forward, Inc. 15 Kota Yagi Profile: • SRE at Money Forward, Inc. (2024 Apr ~) • sig-etcd member (since 2025 Oct ~) Other Notes: • Hobby: Anime • Current obsession: LE SSERAFIM 🍝 X ID: @88888888_kota
16. Money Forward, Inc. 16 IAM Roles Anywhere Mechanics 04
17. IAM Roles Anywhere relies on PKI Public key infrastructure (PKI) • PKI is a system for issuing, managing, and validating digital certificates. • See RFC 5280
18. IAM Roles Anywhere Overview
19. Money Forward, Inc. 19 EKS Pod Identity Mechanics 03
20. EKS Pod Identity Overview A mechanism for assigning temporary IAM Role credentials to Pods running on EKS. • Simplifies IAM Role assignment compared to older methods (IRSA/OIDC). • Components: ◦ EKS Pod Identity Webhook ◦ EKS Pod Identity Agent
21. EKS Pod Identity Overview(contd.)
22. EKS Pod Identity Overview(contd.)
23. IAM Role
24. IAM Role(contd.) The principal in IAM Role should be pods.eks.amazonaws.com
25. EKS Pod Identity Webhook
26. EKS Pod Identity Webhook(contd.) Injects the necessary configs(envs) into the pods that require AWS IAM access Two important envs https://kubernetes.io/blog/2019/03/ 21/a-guide-to-kubernetes-admissio n-controllers/
27. EKS Pod Identity Webhook(contd.) • AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE ◦ Points to a file containing the JWT token for pod authentication • AWS_CONTAINER_CREDENTIALS_FULL_URI ◦ Specifies the endpoint (http://169.254.170.23/v1/credentials) of the EKS Pod Identity Agent ▪ 169.254.170.23: link local address for the agent
28. EKS Pod Identity Webhook(contd.) JWT token used by EKS Pot Identity Agent for requesting EKS Auth API
29. AWS SDK
30. AWS SDK(contd.) • AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE ◦ Points to a file containing the JWT token for pod authentication • AWS_CONTAINER_CREDENTIALS_FULL_URI ◦ Specifies the endpoint (http://169.254.170.23/v1/credentials) of the EKS Pod Identity Agent AWS SDK calls (2) with the JWT Token in (1) for retrieving the credentials from EKS Pod Identity Agent
31. AWS SDK for Go v2 Client https://docs.aws.amazon.com/eks/late st/userguide/pod-id-minimum-sdk.ht ml https://github.com/aws/aws-sdk-go-v2
32. AWS SDK for Go v2 Client(contd.) https://docs.aws.amazon.com/sdkref/latest/guide/standardized-cred entials.html
33. AWS SDK for Go v2 Client(contd.) credential provider chain Implementation github.com/aws/aws-sdk-go-v2/config/resolve_credentials.go • resolveCredentialChain ◦ Top-level credential resolution that prioritizes environment variables over profile configurations. • resolveCredsFromProfile ◦ Detailed credential evaluation within profile configuration, combining both profile settings and environment variables.
34. AWS SDK for Go v2 Client(contd.)
35. AWS SDK for Go v2 Client(contd.)
36. AWS SDK for Go v2 Client(contd.) Configure Client Send request to EKS Pod Identity Agent Get response
37. AWS SDK for Go v2 Client(contd.) Set JTW Token in Authorization header content of AWS_CONTAINER_AUTH ORIZATION_TOKEN_FILE
38. AWS SDK for Go v2 Client(contd.) Json Response
39. EKS Pod Identity Agent
40. EKS Pod Identity Agent(contd.) Agent that is responsible for retrieving the credentials by AssumeRoleForPodIdentity API • Deamonset • Endpoint: GET http://169.254.170.23/v1/credentials • Header: Authorization: [JTW Token] • Main logic: GetIamCredentials
41. EKS Pod Identity Agent(contd.) Call AssumeRoleForPodIdentity API using Cluster name and JWT Token Response to AWS SDK
42. Money Forward, Inc. 42 Implementing Custom Pod Identity 05
43. EKS Pod Identity Overview
44. EKS Pod Identity for On-Prem? ❌ This approach won't work
45. EKS Pod Identity for On-Prem? 👍 Replace this with IAM Roles Anywhere
46. Custom Pod Identity Overview
47. Required AWS Configuration(IAM) Trust relationships Permissions
48. Required Kubernetes Configuration Custom controller watch this label Custom agent read this annotation
49. Custom Pod Identity Overview
50. Custom Controller
51. Custom Controller(contd.) Custom Controller developed by kubebuilder • Watches namespaces with specific label custom-pod-identity.example.com/enabled: “true” • Generates certificate signed by internal CA, which is the trust anchor in IAM Roles Anywhere • Creates secret in target namespaces
52. Custom Controller(contd.)
53. IAM Role
54. IAM Role(contd.) • The principal in IAM Role should be Arn of IAMRA • Restrict which namespaces can use this IAM role using aws:PrincipalTag/x5 09Subject/CN
55. Custom Agent
56. Custom Agent(contd.) (1)Certificate-Based Authentication with IAM Roles Anywhere (2)Assume Role with STS
57. Money Forward, Inc. 57 Summary (Screenshot time) 06
58. IAM Roles Anywhere Overview
59. EKS Pod Identity Overview
60. Custom Pod Identity Overview
61. Summary Custom Pod Identity • Developed by combinating EKS Pod Identity and IAM Roles Anywhere • Allow Kubernetes pods securely assume IAM role • Increase security and reduce operational toil
62. Thank you!
63. Secure AWS Access from On-Prem Kubernetes Using EKS Pod Identity and IAM Roles Anywhere CloudNative Days Winter 2025 Money Forward, Inc. Platform and Reliability Engineering Department Kota Yagi