OAM as Kubernetes Application Definition

This slides is about how to use OAM as Kubernetes application definition and build standard but extensible application platform with it. Real world use cases in Alibaba are included as well.

Transcript

1. https://oam.dev Open Application Model Open Application Model as Kubernetes Application Definition by Lei Zhang (Harry)
2. What’s an application in Kubernetes?
3. Deployment?
4. More like this ...
5. And could be more ... Application
6. https://thenewstack.io/why-apps-are-fundamentally-different-in-the-kubernetes-era/
7. Application Component 1 Component 2 scaling auto route /index security group public scaling manual rollout strategy canary security group private The “Modern Application”
8. Application Component 1 Operational Configurations Component 2 Operational Strategy 1 Operational Strategy 3 Operational Strategy 2 Operational Strategy 4 How to group? How to group? What to run? What to run? The Pattern Behind “Modern Application”
9. Application Component ApplicationConfiguration Workload Component Workload Trait Trait Trait Trait Scope Scope What to run? How to operate? trait scope Open Application Model
10. Application Component ApplicationConfiguration Component scaling: auto scaling: manual route: /index rollout strategy: canary security group: public security group: private What to run? How to operate? trait scope Open Application Model as Kubernetes Application Definition They are all K8s API resources
11. Component Component B Component C Component D Application is composed by Components which defines ”what to run” apiVersion: core.oam.dev/v1alpha2 kind: Component metadata: name: frontend annotations: description: Container workload spec: workload: # a k8s workload CR apiVersion: apps/v1 kind: Deployment spec: containers: - name: web image: php:latest env: - name: OAM_TEXTURE value: texture.jpg ports: - containerPort: 8001 name: http protocol: TCP Component A $ kubectl get components NAME WORKLOAD frontend deployment.apps.k8s.io
12. Component Component B Component C Component D Workload is pluggable in Component apiVersion: core.oam.dev/v1alpha2 kind: Component metadata: name: redis annotations description: Azure RedisCache Instance spec: workload: # a redis instance provided by azure operator apiVersion: azure.microsoft.com/v1alpha1 kind: RedisCache metadata: labels: tag1: value1 tag2: value2 spec: location: westus resourceGroup: resgrp-azure-operators properties: sku: name: Basic family: C capacity: 1 enableNonSslPort: true Component A $ kubectl get components NAME WORKLOAD frontend deployment.apps.k8s.io redis azure.microsoft.com
13. Trait and Application Configuration Trait: declarative operational combabilities apiVersion: core.oam.dev/v1alpha2 kind: ApplicationConfiguration metadata: name: helloworld spec: components: # 1st component - componentName: frontend traits: - trait: apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler spec: minReplicas: 1 maxReplicas: 10 - trait: apiVersion: networking.alibaba-inc.com/v1 kind: APIGateway spec: hostname: app.alibaba.com path: / service_port: 8001 # 2nd component - componentName: redis Application Application Configuration Component B Component C Component D Traits Traits Traits Component A Traits ApplicationConfiguration: assigning traits to components
14. Definition Object Register and discover workloads and traits API resources apiVersion: core.oam.dev/v1alpha2 kind: TraitDefinition metadata: name: manualscaler.core.oam.dev annotations: alias: scale spec: # can only be bound to api resources under apps.k8s.io group appliesTo: - apps.k8s.io # CRD of ManualScaler definition: manualscaler.core.oam.dev --- apiVersion: core.oam.dev/v1alpha2 kind: WorkloadDefinition metadata: name: service.serving.knative.dev annotations: alias: ksvc spec: definitionRef: name: service.serving.knative.dev $ kubectl get traits NAME DEFINITION APPLIES TO scale manualscaler.core.oam.dev apps.k8s.io route route.core.oam.dev apps.k8s.io tls tls.core.oam.dev apps.k8s.io $ kubectl get workloads NAME DEFINITION deployment apps.k8s.io ksvc service.serving.knative.dev
15. apiVersion: core.oam.dev/v1alpha12 kind: ApplicationConfiguration metadata: name: helloworld spec: components: - componentName: frontend traits: - trait: apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: scale-hello spec: minReplicas: 1 maxReplicas: 10 - trait: apiVersion: networking.Alibaba-inc.com/v1 kind: APIGateway metadata: name: api spec: hostname: app.alibaba.com path: / service_port: 8001 applicationScopes: - my-vpc-network-public - componentName: redis applicationScopes: - my-vpc-network-private Application Network Scope Y Network Scope X Application Configuration Component B Component C Component D Trait Trait Trait Component A Trait A way to loosely couple components into groups with common characteristics. Application Scope
16. And More ... • Out-of-box support • Dependency guarantee • e.g. PHP component waits for MySQL component to be ready (by status field, not just container started) • Declare data input/output • e.g. auto inject credential from MySQL component to PHP component’s ENV • Revision system • every change of Component object will generate a revision and can be tracked and referenced by ApplicationConfiguration/Trait
17. Seamlessly Integrated with the Ecosystem OAM YAML files OAM Platform Kubernetes Cloud Resources GitOps Helm, CNAB, OCI packaging deploy OAM Spec What to run: Components Operational capability：Traits Container Function VM Cloud Resource scaling routing ingress manually create Prometheus Istio ... Helm, Kustomize, DSL (e.g. cdk8s) trigger
18. A Typical OAM based App Platform Kubernetes (+ Crossplane) Components Traits/Scope ApplicationConfiguration Container Function Cloud Resources Traffic Management Canary Blue-Green A/B testing Scaling Dev Ops A Micro-App Engine built by OAM Traits System + Registry CLI/UI/Dashboard Capability System (Automatically expose capabilities from installed plugins)
19. Summary of OAM • Right level of abstraction: developer level abstraction to consolidate the application management experience, • Team-centric: separate concerns for developers, operators, and infra engineers. • Platform agnostic: deploy with same YAML to serverless platform, k8s on prems, or any cloud. • Manageability and discoverability: detect conflicted operational strategies at deploy
20. Call for collaboration • Open Application Model community (https://oam.dev) • The Model • https://github.com/oam-dev/spec#community • The OAM Kubernetes Plugin (join effort with Crossplane ) • https://github.com/crossplane/oam-kubernetes-runtime • What’s next? • A OAM based Micro-App Engine!
21. How Alibaba is using OAM?
22. Scale: - 10,000 nodes/cluster - 100,000 apps/cluster - 1,000,000 containers/cluster Velocity: - 100,000 deploys/day - 500~1000 replicas/app Gateway Route Traffic Monitor Alert Deployment App Instance HPA Function Challenges • Fragmentation in app layer, ~11 internal PaaS/Serverless • Siloes, lack reusability and interoperability • Infra centric, low level primitives leaked to end user Gateway Route Traffic Rollout Job Infra Ops Developers Operators Deployment Route Service Job PaaS A PaaS B Serverless C Application Management At Alibaba
23. OAM in Alibaba at highest level App PaaS Operator platform Serverless Hybrid Cloud ISV IoT cloud resource definition service binding DevOps model workload definition trait definition OAM Standard Scenarios on Alibaba Cloud App Marketplace CI/CD Pipeline
24. App/Serverless platforms in Alibaba • Alibaba Application Platform (EDAS) • distributed Java application platform • Public product • Alibaba Operator Platform • Stateful application platform • Internal platform • Alibaba Serverless Platform • Serverless (event driven) application platform • Both public product and internal platform
25. OAM + Alibaba Application PaaS (EDAS) Feature OAM Category Description • ContainerizedWorkload • component • Stateless app • Virtual Machine • component • Legacy app • traffic rollout • trait • Traffic shifting • monitoring • trait • Prometheus configuration • logging • trait • SLS log collecting policy • auto-scaler • trait • HPA • manual-scaler • trait • Manual scaling • rollout • trait • Blue-green deployment • ingress • trait • Routing policy Balance extensibility of k8s with best end-user experience by OAM based abstractions Reusable components and traits to save engineering time Discoverability for capabilities in the system, detect strategy conflict at deploy • Before: traditional PaaS with close ecosystem • After: OAM as core application model, provide simplified end user experience but still keep the extensibility of k8s, make EDAS become into an open ecosystem Share with other platforms
26. OAM + Alibaba Operator Platform feature OAM Category Description • OpenKruise • component • Alternative to StatefulSet • Operators • component • Workloads based on operator • In-place rollout • trait • In-place rollout • canary • trait • Canary deployment • khpa • Trait • HPA to scale Operators • pvc • trait • Claim persistent storage • backup • trait • Data backup • restore • trait • Data restore • ingress • trait • Routing policy • monitoring • trait • Integrated internal monitoring policy All speak OAM API, easy to integrate with other OAM systems OAM to decouple application from runtime, easy to upgrade infrastructure Reusable components and traits to save engineering time • Before: in-house app definition, lack of portability and interoperability across clusters • After: OAM as core application model to manage stateful applications, e.g. Kafka, Zookeeper and Database on any cluster Share with other platforms
27. OAM + Alibaba Serverless Platform • Before: in-house app definition, reinvent every wheel for serverless • After: OAM as unified definition of various workload types, and reuse traits from may other systems like EDAS. feature OAM Category Description • ContainerizedWorkload • component • Serverless container • Function • component • FaaS workload • Knative service (WIP) • component • Knative serving workload • traffic-mgmt • trait • Istio based Traffic shifting • cron-scaler • trait • Cron HPA • manual-scaler • trait • Manual scale • auto-scaler • trait • HPA • sls-project, sls-logstore, sls- index • component • Alibaba Cloud Resources as BaaS Separate concerns make developers accept serverless model way easier Migrate from K8s/PaaS to Serverless only need to update its trait Reusable components and traits to save engineering time Connect and consume cloud resources thru OAM based BaaS Share with other platforms
28. Traits/Scopes Scale: - 10,000 nodes/cluster - 100,000 apps/cluster - 1,000,000 containers/cluster Applications: - 100,000 deploys/day - 500~1000 replicas/app Function Job Container Virtual Machine Gateway Route Traffic Alert Monitor Service Binding Rollout Ingress interpretability Infra Ops Application Configuration Application Configuration Application Configuration PaaS A PaaS B Serverless C Components Traits Components Traits Components Traits Components Application Management At Alibaba Today Developers Operators
29. Kubernetes in Alibaba with OAM Modular, reusable, standardized, and manageable platform capabilities, best engineering efficiency Clear separate of concerns in the workflow of app management, best communication and collaboration efficiency For Platform • Simple, team-centric and unified experience • Enjoy “unlimited” capabilities provided by k8s ecosystem - the “open” PaaS For end users (Dev and Ops) Minimal migration effort from PaaS/K8s to OAM platform The simplest and unified approach to integrate and consume Alibaba Cloud Resources
30. Thank You Application is the next hype!