#ossummit @ Architecting Kubernetes-Based Internal Developer Platforms: Essential Patterns and Practices Hiroshi Hayakawa, LY Corporation @hhiroshell

About Me • Working for LY Corporation • An internet company that offers various services, including communication, internet portals, media, and commerce …etc, primarily in Japan. • Contributing to CNCF TAG App Delivery • Author of books on Kubernetes • DIY keyboard enthusiast Hiroshi Hayakawa | @hhiroshell

Agenda • Background: raise of Kubernetes based internal developer platforms! • Let’s build our Kubernetes based IDP!! • High-level overview of technical and organizational structure • Detailed technical considerations • Conclusion

Agenda • Background: raise of Kubernetes based internal developer platforms! • Let’s build our Kubernetes based IDP!! • High-level overview of technical and organizational structure • Detailed technical considerations • Conclusion

What is Platform Engineering? • Recent IT paradigms have shortened release cycles but also burdened developers with mastering many tools (cf. extraneous cognitive load) Cloud Infrastructure & IaC Microservices Continuous Integration & Delivery (CI/CD) DevOps

What is Platform Engineering? • An initiative to provide foundations called IDP(Internal Developer Platform) • IDP allows internal developers to focus on creating essential values for their business versatile but burdening tools and infrastructures IDP Platform Team Developers use provide use

Kubernetes is an IDP, right? • It streamlines application developments and day-by-day operations by providing: • abstraction of low-level computing resources • declarative workload management • auto-healing and scaling • safe application updates • It has extensibility to optimize for organizational requirements

Then… should we do like this? Developers request Cloud provider’s Kubernetes as a service capability Kubernetes cluster use create cluster

But… things are not so easy Developers request Cloud provider’s Kubernetes as a Service capability Kubernetes cluster ✓ RBAC ✓ Network Policy ✓ Admission policy ✓ Monitoring Agent ✓ Ingress / storage driver ✓ Deploy applications ✓ Upgrade the cluster create vanilla cluster use

What exactly are the developers’ demand? 1. Out-of-the-box Kubernetes as a Service • for running general workload such as business applications • Tailored to organizational policies • Commonly required addons are pre-configured 2. X as a Service • for providing typical capabilities that functions as building blocks of the system • e.g., ML, Database, Messaging • No need for Kubernetes management • Optimized Interfaces for each capability

So, let’s build our Kubernetes based IDP! • From the next section, we’ll explore: • High-level overview of technical and organizational structure • Detailed technical considerations

Agenda • Background: raise of Kubernetes based internal developer platforms! • Let’s build our Kubernetes based IDP!! • High-level overview of technical and organizational structure • Detailed technical considerations • Conclusion

A long time ago in a galaxy far, far away….

No content

Ancient, primitive clusters and it’s administrator Chat System Developers request Cloud provider’s Kubernetes as a service capability use Cluster Admin Provisioning script provision Kubernetes cluster tailored to organizational requirements

The basic architecture of KaaS IDP interface Developers request Kubernetes tenant tailored to organizational requirements use provision provisioner

The basic architecture of KaaS IDP interface Developers request use provision Platform Team Kubernetes tenant tailored to organizational requirements provisioner

Example: Scalable, self-service Kubernetes IDP CLI / GUI Developers request use EKS Provider Helm Provider Platform Cluster ✓ RBAC ✓ ingress controller ✓ monitoring agent ✓ … create cluster apply policies and addons Crossplane composition

Provisioner options Scripts (Bash, Python…) IaC Templates & Modules IaC + Automatic trigger (e.g., CD Pipelines) Kubernetes Operator * Cluster API initial cost scalability & cluster manageability Eliminate platform team’s manual tasks

Eliminate platform team’s manual tasks Interface options Ticketing System Expose the provisioner’s API to developers (e.g., K8s custom resource, CD pipeline) Developer friendly utilities (REST API, CLI, GUI) initial cost useability

Let’s summarize up to this point • KaaS IDP enables developers to access ready-to-use Kubernetes clusters out-of- the-box • Kubernetes as a Service IDP can be broken down to 3 components: Interface, Provisioner and Tenant • Making technology choices with balancing: • scalability • usability (for developers) • cluster manageability (for platform teams) • initial development cost • platform team’s skill set

A long time ago in a galaxy far, far away….

No content

Another requirement… interface Developers request use provision Platform Team I want a scalable RDB. But managed RDB is expensive for us. Kubernetes tenant tailored to organizational requirements provisioner

provisioner Platform Team Another requirement… interface Developers request use provision I have a bad feeling about this. Kubernetes tenant tailored to organizational requirements

Another requirement… interface Developers request use provision Vitess Operator extends Kubernetes’ API and add a scalable RDB capabiliy Kubernetes tenant tailored to organizational requirements Platform Team provisioner

What’s the problem? • Day 1 experience is easy and smooth in the most of major Kubernetes operators • Handling day 2 operations is a challenge for both developers and platform teams, while doing their original roles • For example, using RDB operator needs to: • manage, tune, and trouble shoot for databases running as containers on top of Kubernetes • update the operator considering compatibility between Kubernetes and the operator

provisioner Kubernetes based XaaS interface Developers provision XaaS interface XaaS Capability team Platform Team request Kubernetes tenant tailored to organizational requirements use

XaaS Interface options Expose the operator’s API to developers Developer friendly utilities (REST API, CLI, GUI) Single pane platform interface useability initial cost Offers consistent developer experience between multiple IDPs

Agenda • Background: raise of Kubernetes based internal developer platforms! • Let’s build our Kubernetes based IDP!! • High-level overview of technical and organizational structure • Detailed technical considerations • Conclusion

We’ll pick up two considerations interface Developers request Kubernetes tenant use provision provisioner #1 Isolation and multi-tenancy options #2 Single sign-on for multiple clusters

We’ll pick up two considerations interface Developers request Kubernetes tenant use provision provisioner #1 Isolation and multi-tenancy options #2 Single sign-on for multiple clusters

Isolation and multi-tenancy options • Namespace per tenant • Developers have access to Kubernetes capabilities within a namespace and share a single cluster resource • Logical cluster per tenant • Developers own a logically created cluster that runs on top of a real cluster and share the resources of the actual cluster • This is not native Kubernetes functionality. and requires additional layer like vCluster • Cluster per tenant • Developers own a whole single cluster

Isolation and multi-tenancy options Namespace Logical cluster (vCluster) Cluster Isolation very weak strong very strong Access for Tenants very restricted vCluster admin cluster admin Resource Sharing easy easy very hard Overhead very low very low very high Cost very cheap cheap expensive * https://www.vcluster.com/docs/#benefits

Isolation and multi-tenancy options Namespace Logical cluster (vCluster) Cluster Isolation very weak strong very strong Access for Tenants very restricted vCluster admin cluster admin Resource Sharing easy easy very hard Overhead very low very low very high Cost very cheap cheap expensive * https://www.vcluster.com/docs/#benefits Complexity low • needs additional control plane layer low

Isolation and multi-tenancy options Namespace Logical cluster (vCluster) Cluster Isolation very weak strong very strong Access for Tenants very restricted vCluster admin cluster admin Resource Sharing easy easy very hard Overhead very low very low very high Cost very cheap cheap expensive * https://www.vcluster.com/docs/#benefits Complexity low • needs additional control plane layer low Manageability • cluster wide resource • noisy neighbor • noisy neighbor • multi cluster management

We’ll pick up two considerations interface Developers request Kubernetes tenant use provision provisioner #1 Isolation and multi-tenancy options #2 Single sign-on for multiple clusters

Single sign-on for Kubernetes… why? • As the number of KaaS and XaaS clusters increases, logging into each cluster becomes burdensome for developers. Kubernetes API Server Kubernetes API Server Kubernetes API Server XaaS-A Cluster XaaS-B Cluster Developers browser kubectl XaaS-B CLI KaaS Cluster login login login

Centralized authentication and authorization solution Kubernetes API Server Kubernetes API Server Kubernetes API Server XaaS-A Cluster XaaS-B Cluster Developers browser kubectl XaaS-B CLI Athenz Policy OIDC Provider e.g., Keycloak, Dex with Github, Okta … KaaS Cluster request w/ OIDC token authenticate & get OIDC token verify the token verify user’s access rights

Agenda • Background: raise of Kubernetes based internal developer platforms! • Let’s build our Kubernetes based IDP!! • High-level overview of technical and organizational structure • Detailed technical considerations • Conclusion

Key lessons & takeaways • IDP enables developers to focus on application development and operations • Kubernetes is good foundation for implementing an IDP: • A tailored Kubernetes cluster • A Kubernetes-based XaaS • The right technical and organizational architecture are key to success • While there are some technical considerations, they could be addressed by integrating Cloud Native technologies.

Kubernetes-related IDP w/ Cloud Native technologies • Cluster itself: • Kubernetes • AuthN/Z: • Dex • Keycloak • Athenz • Monitoring, Logging: • Prometheus • fluentd / fluent bit • Open Telemetry • Cluster Provisioning: • Crossplane • Cluster API

Cloud Native Platform Engineering • Cloud Native technologies serve as crucial building blocks in creating IDPs • A variety of middleware and a robust OSS ecosystem centered around Kubernetes * CNCF graduated projects

Any Questions?