Developer Platform-Part I: Infrastructure Provisioning

In this series of presentations, we aim to describe the main challenges that platform organizations face in trying to enable their dev. teams to build applications on Kubernetes, esp. in public cloud environments. We then explore the most relevant CNCF projects that can help address those challenges.

In Part I (this deck), we explore the problem of infrastructure/resource provisioning. Today, an application packaged for and deployed in a Kubernetes cluster specifies some of its dependencies in KRM format. Examples of these are resources needed by containerized workloads (cpu, memory, storage), basic networking and load balancing, TLS configuration, etc. However, many other dependencies such as databases, message queues, topics, caches, etc. are left to the application teams to determine. We look into several alternatives and evaluate their pros and cons and describe what it takes to create an all-encompassing platform that standardizes around Kubernetes APIs. The main goal with that is to make it possible for application developers to specify all dependencies using Kubernetes APIs and for platform teams and application Ops teams to fulfill those dependencies by leveraging Kubernetes APIs and control loops.

(The series is organized to be digestible in smaller chunks by focusing on a single problem in each section. )

Transcript

1. Development Platform for Cloud Native Applications Part I: Infrastructure/Resource Provisioning Hossein Kassaei hossein.kassaei@ericsson.com
2. Kubernetes - Pod Pod App I need cpu and memory! (PowerPoint’s stock image for memory)!
3. Kubernetes - Volumes I need storage! PV App file/block
4. Kubernetes - Service App I need a stable endpoint to let clients discover me + network load balancer Service
5. Kubernetes – Ingress App I need application load balancer and TLS encryption Ingress
6. So far so good! Ok, I like this whole dependency abstraction and resource brokering that Kubernetes gives me! Let’s build on Kubernetes in the cloud …
7. I need an object store App I need somewhere to store objects! bucket Bucket? Sorry, not sure what you mean! But maybe come back later!? Container Object Storage Interface (COSI)
8. I need PostgreSQL App I need a Relational Database (Postgres) Sorry, no clue what you mean!
9. I need a cache App I need a key-value Database (Redis) Sorry, no clue what you mean!
10. I need a secret store App I need to store my secrets in a secret store (not in k8s secrets/etcd) Sorry, no clue what you mean! h/t External Secrets Operator
11. And many other things … App You get the idea … (I have many needs!) IAM roles/policies certificates message queues topics …
12. Dev. Persona: how do I specify all these other dependencies? SRE Persona: how do I provision and monitor all these other dependencies? “Terraform has been around since 2014 and widely used by anyone wanting to do IaC. Should we continue using that also for applications deployed in Kubernetes?” 🤔
13. Terraform App I need a Relational Database (Postgres) AWS RDS for PostgreSQL Full example on GitHub
14. Terraform App I need a Relational Database (Postgres) Google CloudSQL for PostgreSQL Full example on GitHub
15. Terraform App I need a Relational Database (Postgres) Azure Database for PostgreSQL Full example on GitHub
16. Terraform App I need a Relational Database (Postgres) Azure Database for PostgreSQL Google CloudSQL for PostgreSQL AWS RDS for PostgreSQL
17. Summary • Some application dependencies are specified in k8s yaml manifest files • Many other dependencies are specified using terraform modules and HCL • The most comprehensive repository of reusable modules! • Declarative vs. Imperative? • Drift detection? • Continuous reconciliation? • Cloud-specific provisioning logic exposed to application dev. could be bad or ok depending on: - how many cloud environments we need to support - if this is done by a dedicated platform team or by all app teams
18. Use Kubernetes for all dependencies App I need a Relational Database (Postgres) AWS Controllers for Kubernetes (ACK) AWS RDS for PostgreSQL Full example
19. Use Kubernetes for all dependencies App I need a Relational Database (Postgres) Google Config Connector Google CloudSQL for PostgreSQL More info
20. Use Kubernetes for all dependencies App I need a Relational Database (Postgres) Azure Service Operator (v2) Azure Database for PostgreSQL more info
21. Use Kubernetes for all dependencies App I need a Relational Database (Postgres) Cloud provider proprietary APIs AWS RDS for PostgreSQL Google CloudSQL for PostgreSQL Azure Database for PostgreSQL
22. Summary • All application dependencies are specified in k8s yaml manifest files • All declarative (it’s Kubernetes!) • Drift detection (it’s Kubernetes!) • Continuous reconciliation (it’s Kubernetes!) • Cloud-specific provisioning logic exposed to application dev. Could be bad or ok depending on: - how many cloud environments we need to support - if this is done by a dedicated platform team or by all app teams
23. Starting to look good … but can we do even better?
24. Enter Crossplane Key concepts • Managed Resource (MR) • Composite Resource (XR) • Composition • Composite Resource Claim (claim)
25. Composite Resource & Composition • Enable two important capabilities 1) Abstraction 2) Composition
26. Abstraction App Platform engineer needs to tell Crossplane how to translate this abstract definition to a cloud-specific resource definition composition … Google CloudSQL for PostgreSQL MR provided by Crossplane gcp-provider XR defined by the platform team
27. Abstraction App AWS RDS for PostgreSQL Google CloudSQL for PostgreSQL Azure Database for PostgreSQL translate to cloud specific service Makes it possible to create our own declarative APIs! (without writing k8s controllers)
28. Composition Network Makes it possible to hide a lot of unnecessary details from the application teams! composition Subnetwork Cluster NodePool
29. Bringing it all together my-not-so-simple-app infra KubernetesCluster.yaml deployment-1.yaml deployment-2.yaml ingress.yaml service.yaml postgresql.yaml redis.yaml service-account.yaml …
30. Summary • All application dependencies are specified in k8s yaml manifest files • All declarative (it’s Kubernetes!) • Drift detection (it’s Kubernetes!) • Continuous reconciliation (it’s Kubernetes) • Uniform provisioning logic exposed to application dev • Platform team needs to do the heavy lifting (but still much work if using Crossplane)
31. For further exploration • Harmonize on-prem and public cloud deployments when it comes to consuming services by the application App Google CloudSQL for PostgreSQL On-prem PostgreSQL Instance We also need to harmonize how database connection details are exposed to the App e.g., managed by Zalando Postgres operator
32. Hybrid and Multi-cloud App Kubernetes App manifests + “claims” for resources reconciliation on-prem
33. References • Terraform: https://registry.terraform.io/ • AWS Controllers for Kubernetes: https://aws-controllers-k8s.github.io/community/docs/community/overview/ • Azure Service Operator v2: https://azure.github.io/azure-service-operator/reference/ • Google Config Connector resources: https://cloud.google.com/config-connector/docs/reference/overview • Crossplane: • https://crossplane.io/ • https://crossplane.io/docs/ • Upbound: • https://www.upbound.io/ • https://marketplace.upbound.io/