Secure AWS Access from On-Prem Kubernetes Using EKS Pod Identity and IAM Roles Anywhere CloudNative Days Winter 2025 Money Forward, Inc. Platform and Reliability Engineering Department Kota Yagi 

Agenda ● Company & Tech Stack Overview ● Challenge & Solution Overview ● Solution Details ○ IAM Roles Anywhere Mechanics ○ EKS Pod Identity Mechanics ○ Custom Pod Identity Mechanics ● Summary 

Money Forward, Inc. 3 Company and Tech Stack Overview 01 

No content

Main Services and Divisions ・Exceeded 17 million users ・No. 1 recognition rate, and utilization rate among household accounting applications and asset management applications1 A division that creates services in cooperation with clients, mainly financial institutions. Developed "Mikatano" (a business DX service for small to medium local enterprises delivered through local financial institutions), "BANK APP" (a banking application), etc. A service that visualizes money that anyone can continue with ease. Introduces centralized management to household budgets and assets to let the user grasp money flow and the current state of the assets, thereby supporting the initial step in eliminating financial anxiety ・The number of paying customers exceeded 410,000 ・No. 1 satisfaction among cloud accounting softwares1 ・No. 1 share within SaaS payroll management2 A SaaS style platform for businesses that connects back office data of various kinds and thereby optimizes tedious tasks related to accounting, human resources and labour affairs, and legal affairs 1. Delegated research by Macromill, Inc. / Survey target and number of respondents: 1,034 household accounting app users and 1,034 asset management app users in their 20s to 60s / Survey period: Household accounting app - August 13, 2024 to August 14, 2024; Asset management app - August 13, 2024 to August 16, 2024 / Survey method: Internet research 1. July 2024, JustSystems Co., Ltd. Fastask survey (responses fewer than 30 excluded). 2. Source: ITR "ITR Market View：人事・給与・就業管理市場2025" Payroll Management Market - Less than 100 Employees: Sales Revenue and Share by Vendor (FY2022-FY2024 Forecast) Business Individuals Financial Institutions 

Product Lineup POINT Corporates Individuals Financial Institutions Our products and services span across the B2B, B2C and other domains. POINT 出典元 : 株式会社マネーフォワード 会社紹介資料 Business Finance Home X Money Forward for Financial Institutions & Specific Services Digital Passbook & Easy Passbook 

Technology Stack Developm ent Language Frontend JavaScript/TypeScript Backend Ruby/Go/Kotlin/Java/Python/Rust Mobile Android: Kotlin/Java iOS: Swift/Objective-C Cross Platform: Dart (Flutter) Infrastructure AWS/GCP/EKS/ECS/Kubernetes (on-premises) Database MySQL/MongoDB/BigQuery Security WAF/EDR/SIEM/CSPM/SAST/DAST Quality Assurance TestRail/Autify/MagicPod/mabl/Gauge/Selenium/ Postman/DataDog Surveillance, Monitoring DataDog/PagerDuty Environment Terraform/ansible Continuous Integration, Continuous Delivery CircleCI/GitHub Actions/Bitrise/ArgoCD GitHub Actions 

Money Forward, Inc. 8 Challenge and Solution Overview 02 

About On-Prem Kubernetes ● On-Prem Kubernetes Cluster (Production) ○ Nodes: 23 (control plane: 3, etcd: 5, worker node: 15) ○ Pods: +350 ○ Namespaces: +35 ○ Workloads: Microservices + Batch ○ Add-ons: Ingress NGINX Controller, ExternalDNS, MetalLB, etc… 

Problem in On-Prem Kubernetes On-Prem K8s workloads used Long-Term IAM User Credentials for AWS access Problem: ● High Security Risk: Severe impact if leaked ● Operational Toil: Requires tedious manual, periodic rotation 

AWS IAM Best Practices ● Use temporary security credentials (IAM roles) instead of creating long-term credentials (IAM User) ● Follow Security best practices in IAM 

Challenge: Using IAM Roles from K8s Workload Outside of AWS Outside AWS(On-Prem K8s): How can we assume an IAM Role securely? ● Outside AWS: Solved by IAM Roles Anywhere ● Inside AWS(EKS): Solved by EKS Pod Identity 👉 Outside AWS(On-Prem K8s): Solved by a combination of IAM Roles Anywhere + EKS Pod Identity ? 

Idea: EKS Pod Identity × IAM Roles Anywhere Custom Pod Identity developed by combinating IAM Roles Anywhere and EKS Pod Identity ● IAM Roles Anywhere ○ Provides a way to get temporary credentials for a workload or process that runs outside of AWS, which means we can get around the limitation of EKS Pod Identity(work on EKS). ● EKS Pod Identity ○ Allows Kubernetes pods running on EKS to securely access AWS services by assigning IAM roles to pods. 

Custom Pod Identity Overview 

Money Forward, Inc. 15 Kota Yagi Profile: ● SRE at Money Forward, Inc. (2024 Apr ~) ● sig-etcd member (since 2025 Oct ~) Other Notes: ● Hobby: Anime ● Current obsession: LE SSERAFIM 🍝 X ID: @88888888_kota 

Money Forward, Inc. 16 IAM Roles Anywhere Mechanics 04 

IAM Roles Anywhere relies on PKI Public key infrastructure (PKI) ● PKI is a system for issuing, managing, and validating digital certificates. ● See RFC 5280 

IAM Roles Anywhere Overview 

Money Forward, Inc. 19 EKS Pod Identity Mechanics 03 

EKS Pod Identity Overview A mechanism for assigning temporary IAM Role credentials to Pods running on EKS. ● Simplifies IAM Role assignment compared to older methods (IRSA/OIDC). ● Components: ○ EKS Pod Identity Webhook ○ EKS Pod Identity Agent 

EKS Pod Identity Overview(contd.) 

EKS Pod Identity Overview(contd.) 

IAM Role 

IAM Role(contd.) The principal in IAM Role should be pods.eks.amazonaws.com 

EKS Pod Identity Webhook 

EKS Pod Identity Webhook(contd.) Injects the necessary configs(envs) into the pods that require AWS IAM access Two important envs https://kubernetes.io/blog/2019/03/ 21/a-guide-to-kubernetes-admissio n-controllers/ 

EKS Pod Identity Webhook(contd.) ● AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE ○ Points to a file containing the JWT token for pod authentication ● AWS_CONTAINER_CREDENTIALS_FULL_URI ○ Specifies the endpoint (http://169.254.170.23/v1/credentials) of the EKS Pod Identity Agent ■ 169.254.170.23: link local address for the agent 

EKS Pod Identity Webhook(contd.) JWT token used by EKS Pot Identity Agent for requesting EKS Auth API 

AWS SDK 

AWS SDK(contd.) ● AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE ○ Points to a file containing the JWT token for pod authentication ● AWS_CONTAINER_CREDENTIALS_FULL_URI ○ Specifies the endpoint (http://169.254.170.23/v1/credentials) of the EKS Pod Identity Agent AWS SDK calls (2) with the JWT Token in (1) for retrieving the credentials from EKS Pod Identity Agent 

AWS SDK for Go v2 Client https://docs.aws.amazon.com/eks/late st/userguide/pod-id-minimum-sdk.ht ml https://github.com/aws/aws-sdk-go-v2 

AWS SDK for Go v2 Client(contd.) https://docs.aws.amazon.com/sdkref/latest/guide/standardized-cred entials.html 

AWS SDK for Go v2 Client(contd.) credential provider chain Implementation github.com/aws/aws-sdk-go-v2/config/resolve_credentials.go ● resolveCredentialChain ○ Top-level credential resolution that prioritizes environment variables over profile configurations. ● resolveCredsFromProfile ○ Detailed credential evaluation within profile configuration, combining both profile settings and environment variables. 

AWS SDK for Go v2 Client(contd.) 

AWS SDK for Go v2 Client(contd.) 

AWS SDK for Go v2 Client(contd.) Configure Client Send request to EKS Pod Identity Agent Get response 

AWS SDK for Go v2 Client(contd.) Set JTW Token in Authorization header content of AWS_CONTAINER_AUTH ORIZATION_TOKEN_FILE 

AWS SDK for Go v2 Client(contd.) Json Response 

EKS Pod Identity Agent 

EKS Pod Identity Agent(contd.) Agent that is responsible for retrieving the credentials by AssumeRoleForPodIdentity API ● Deamonset ● Endpoint: GET http://169.254.170.23/v1/credentials ● Header: Authorization: [JTW Token] ● Main logic: GetIamCredentials 

EKS Pod Identity Agent(contd.) Call AssumeRoleForPodIdentity API using Cluster name and JWT Token Response to AWS SDK 

Money Forward, Inc. 42 Implementing Custom Pod Identity 05 

EKS Pod Identity Overview 

EKS Pod Identity for On-Prem? ❌ This approach won't work 

EKS Pod Identity for On-Prem? 👍 Replace this with IAM Roles Anywhere 

Custom Pod Identity Overview 

Required AWS Configuration(IAM) Trust relationships Permissions 

Required Kubernetes Configuration Custom controller watch this label Custom agent read this annotation 

Custom Pod Identity Overview 

Custom Controller 

Custom Controller(contd.) Custom Controller developed by kubebuilder ● Watches namespaces with specific label custom-pod-identity.example.com/enabled: “true” ● Generates certificate signed by internal CA, which is the trust anchor in IAM Roles Anywhere ● Creates secret in target namespaces 

Custom Controller(contd.) 

IAM Role 

IAM Role(contd.) ● The principal in IAM Role should be Arn of IAMRA ● Restrict which namespaces can use this IAM role using aws:PrincipalTag/x5 09Subject/CN 

Custom Agent 

Custom Agent(contd.) (1)Certificate-Based Authentication with IAM Roles Anywhere (2)Assume Role with STS 

Money Forward, Inc. 57 Summary (Screenshot time) 06 

IAM Roles Anywhere Overview 

EKS Pod Identity Overview 

Custom Pod Identity Overview 

Summary Custom Pod Identity ● Developed by combinating EKS Pod Identity and IAM Roles Anywhere ● Allow Kubernetes pods securely assume IAM role ● Increase security and reduce operational toil 

Thank you! 

Secure AWS Access from On-Prem Kubernetes Using EKS Pod Identity and IAM Roles Anywhere CloudNative Days Winter 2025 Money Forward, Inc. Platform and Reliability Engineering Department Kota Yagi