Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OAM as Kubernetes Application Definition

OAM as Kubernetes Application Definition

This slides is about how to use OAM as Kubernetes application definition and build standard but extensible application platform with it. Real world use cases in Alibaba are included as well.

Lei (Harry) Zhang

June 25, 2020
Tweet

More Decks by Lei (Harry) Zhang

Other Decks in Technology

Transcript

  1. https://oam.dev
    Open
    Application
    Model
    Open Application Model as
    Kubernetes Application
    Definition
    by Lei Zhang (Harry)

    View Slide

  2. What’s an application in
    Kubernetes?

    View Slide

  3. Deployment?

    View Slide

  4. More like this ...

    View Slide

  5. And could be more ...
    Application

    View Slide

  6. https://thenewstack.io/why-apps-are-fundamentally-different-in-the-kubernetes-era/

    View Slide

  7. Application
    Component 1 Component 2
    scaling auto
    route /index
    security group public
    scaling manual
    rollout strategy canary
    security group private
    The “Modern Application”

    View Slide

  8. Application
    Component 1
    Operational Configurations
    Component 2
    Operational Strategy 1
    Operational Strategy 3
    Operational Strategy 2
    Operational Strategy 4
    How to group?
    How to group?
    What to run?
    What to run?
    The Pattern Behind “Modern Application”

    View Slide

  9. Application
    Component
    ApplicationConfiguration
    Workload
    Component
    Workload
    Trait
    Trait
    Trait
    Trait
    Scope
    Scope
    What to run? How to operate?
    trait
    scope
    Open Application Model

    View Slide

  10. Application
    Component
    ApplicationConfiguration
    Component
    scaling: auto
    scaling: manual
    route: /index
    rollout strategy: canary
    security group: public
    security group: private
    What to run? How to operate?
    trait
    scope
    Open Application Model as Kubernetes Application Definition
    They are all K8s API
    resources

    View Slide

  11. Component
    Component
    B
    Component
    C
    Component
    D
    Application is composed by Components
    which defines ”what to run”
    apiVersion: core.oam.dev/v1alpha2
    kind: Component
    metadata:
    name: frontend
    annotations:
    description: Container workload
    spec:
    workload:
    # a k8s workload CR
    apiVersion: apps/v1
    kind: Deployment
    spec:
    containers:
    - name: web
    image: php:latest
    env:
    - name: OAM_TEXTURE
    value: texture.jpg
    ports:
    - containerPort: 8001
    name: http
    protocol: TCP
    Component
    A
    $ kubectl get components
    NAME WORKLOAD
    frontend deployment.apps.k8s.io

    View Slide

  12. Component
    Component
    B
    Component
    C
    Component
    D
    Workload is pluggable in Component
    apiVersion: core.oam.dev/v1alpha2
    kind: Component
    metadata:
    name: redis
    annotations
    description: Azure RedisCache Instance
    spec:
    workload:
    # a redis instance provided by azure operator
    apiVersion: azure.microsoft.com/v1alpha1
    kind: RedisCache
    metadata:
    labels:
    tag1: value1
    tag2: value2
    spec:
    location: westus
    resourceGroup: resgrp-azure-operators
    properties:
    sku:
    name: Basic
    family: C
    capacity: 1
    enableNonSslPort: true
    Component
    A
    $ kubectl get components
    NAME WORKLOAD
    frontend deployment.apps.k8s.io
    redis azure.microsoft.com

    View Slide

  13. Trait and Application Configuration
    Trait: declarative operational combabilities
    apiVersion: core.oam.dev/v1alpha2
    kind: ApplicationConfiguration
    metadata:
    name: helloworld
    spec:
    components:
    # 1st component
    - componentName: frontend
    traits:
    - trait:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    spec:
    minReplicas: 1
    maxReplicas: 10
    - trait:
    apiVersion: networking.alibaba-inc.com/v1
    kind: APIGateway
    spec:
    hostname: app.alibaba.com
    path: /
    service_port: 8001
    # 2nd component
    - componentName: redis
    Application
    Application Configuration
    Component
    B
    Component
    C
    Component
    D
    Traits Traits Traits
    Component
    A
    Traits
    ApplicationConfiguration: assigning traits to
    components

    View Slide

  14. Definition Object
    Register and discover workloads and
    traits API resources
    apiVersion: core.oam.dev/v1alpha2
    kind: TraitDefinition
    metadata:
    name: manualscaler.core.oam.dev
    annotations:
    alias: scale
    spec:
    # can only be bound to api resources under
    apps.k8s.io group
    appliesTo:
    - apps.k8s.io
    # CRD of ManualScaler
    definition: manualscaler.core.oam.dev
    ---
    apiVersion: core.oam.dev/v1alpha2
    kind: WorkloadDefinition
    metadata:
    name: service.serving.knative.dev
    annotations:
    alias: ksvc
    spec:
    definitionRef:
    name: service.serving.knative.dev
    $ kubectl get traits
    NAME DEFINITION APPLIES TO
    scale manualscaler.core.oam.dev apps.k8s.io
    route route.core.oam.dev apps.k8s.io
    tls tls.core.oam.dev apps.k8s.io
    $ kubectl get workloads
    NAME DEFINITION
    deployment apps.k8s.io
    ksvc service.serving.knative.dev

    View Slide

  15. apiVersion: core.oam.dev/v1alpha12
    kind: ApplicationConfiguration
    metadata:
    name: helloworld
    spec:
    components:
    - componentName: frontend
    traits:
    - trait:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    metadata:
    name: scale-hello
    spec:
    minReplicas: 1
    maxReplicas: 10
    - trait:
    apiVersion: networking.Alibaba-inc.com/v1
    kind: APIGateway
    metadata:
    name: api
    spec:
    hostname: app.alibaba.com
    path: /
    service_port: 8001
    applicationScopes:
    - my-vpc-network-public
    - componentName: redis
    applicationScopes:
    - my-vpc-network-private
    Application
    Network Scope Y
    Network Scope X
    Application Configuration
    Component
    B
    Component
    C
    Component
    D
    Trait Trait Trait
    Component
    A
    Trait
    A way to loosely couple components
    into groups with common
    characteristics.
    Application Scope

    View Slide

  16. And More ...
    • Out-of-box support
    • Dependency guarantee
    • e.g. PHP component waits for MySQL component to be ready (by status field, not just
    container started)
    • Declare data input/output
    • e.g. auto inject credential from MySQL component to PHP component’s ENV
    • Revision system
    • every change of Component object will generate a revision and can be tracked and
    referenced by ApplicationConfiguration/Trait

    View Slide

  17. Seamlessly
    Integrated with the
    Ecosystem
    OAM YAML files
    OAM Platform
    Kubernetes Cloud Resources
    GitOps
    Helm, CNAB, OCI
    packaging
    deploy
    OAM Spec
    What to run: Components
    Operational capability:Traits
    Container
    Function
    VM
    Cloud Resource
    scaling
    routing
    ingress
    manually create
    Prometheus
    Istio
    ...
    Helm, Kustomize, DSL (e.g. cdk8s)
    trigger

    View Slide

  18. A Typical OAM based App Platform
    Kubernetes (+ Crossplane)
    Components Traits/Scope
    ApplicationConfiguration
    Container Function Cloud Resources
    Traffic
    Management
    Canary
    Blue-Green
    A/B testing
    Scaling
    Dev Ops
    A Micro-App Engine built by OAM
    Traits System + Registry
    CLI/UI/Dashboard
    Capability System (Automatically expose capabilities from installed plugins)

    View Slide

  19. Summary of OAM
    • Right level of abstraction: developer level abstraction to consolidate the application management
    experience,
    • Team-centric: separate concerns for developers, operators, and infra engineers.
    • Platform agnostic: deploy with same YAML to serverless platform, k8s on prems, or any cloud.
    • Manageability and discoverability: detect conflicted operational strategies at deploy

    View Slide

  20. Call for collaboration
    • Open Application Model community (https://oam.dev)
    • The Model
    • https://github.com/oam-dev/spec#community
    • The OAM Kubernetes Plugin (join effort with Crossplane )
    • https://github.com/crossplane/oam-kubernetes-runtime
    • What’s next?
    • A OAM based Micro-App Engine!

    View Slide

  21. How Alibaba is using OAM?

    View Slide

  22. Scale:
    - 10,000 nodes/cluster
    - 100,000 apps/cluster
    - 1,000,000 containers/cluster
    Velocity:
    - 100,000 deploys/day
    - 500~1000 replicas/app
    Gateway
    Route Traffic
    Monitor
    Alert
    Deployment
    App Instance
    HPA
    Function
    Challenges
    • Fragmentation in app layer, ~11
    internal PaaS/Serverless
    • Siloes, lack reusability and
    interoperability
    • Infra centric, low level primitives
    leaked to end user
    Gateway
    Route
    Traffic
    Rollout
    Job
    Infra Ops
    Developers Operators
    Deployment
    Route
    Service
    Job
    PaaS A
    PaaS B
    Serverless C
    Application Management At Alibaba

    View Slide

  23. OAM in Alibaba at highest level
    App PaaS
    Operator
    platform
    Serverless
    Hybrid Cloud
    ISV
    IoT
    cloud resource definition
    service binding
    DevOps model
    workload definition
    trait definition
    OAM Standard
    Scenarios on
    Alibaba Cloud
    App Marketplace
    CI/CD Pipeline

    View Slide

  24. App/Serverless platforms in Alibaba
    • Alibaba Application Platform (EDAS)
    • distributed Java application platform
    • Public product
    • Alibaba Operator Platform
    • Stateful application platform
    • Internal platform
    • Alibaba Serverless Platform
    • Serverless (event driven) application platform
    • Both public product and internal platform

    View Slide

  25. OAM + Alibaba Application PaaS (EDAS)
    Feature OAM Category Description
    • ContainerizedWorkload • component • Stateless app
    • Virtual Machine • component • Legacy app
    • traffic rollout • trait • Traffic shifting
    • monitoring • trait • Prometheus configuration
    • logging • trait • SLS log collecting policy
    • auto-scaler • trait • HPA
    • manual-scaler • trait • Manual scaling
    • rollout • trait • Blue-green deployment
    • ingress • trait • Routing policy
    Balance extensibility of k8s
    with best end-user experience
    by OAM based abstractions
    Reusable components and
    traits to save engineering time
    Discoverability for capabilities
    in the system, detect strategy
    conflict at deploy
    • Before: traditional PaaS with close ecosystem
    • After: OAM as core application model, provide simplified end user experience but
    still keep the extensibility of k8s, make EDAS become into an open ecosystem
    Share with other platforms

    View Slide

  26. OAM + Alibaba Operator Platform
    feature OAM Category Description
    • OpenKruise • component • Alternative to StatefulSet
    • Operators • component • Workloads based on operator
    • In-place rollout • trait • In-place rollout
    • canary • trait • Canary deployment
    • khpa • Trait • HPA to scale Operators
    • pvc • trait • Claim persistent storage
    • backup • trait • Data backup
    • restore • trait • Data restore
    • ingress • trait • Routing policy
    • monitoring • trait • Integrated internal monitoring
    policy
    All speak OAM API, easy to
    integrate with other OAM
    systems
    OAM to decouple application
    from runtime, easy to upgrade
    infrastructure
    Reusable components and
    traits to save engineering time
    • Before: in-house app definition, lack of portability and interoperability across
    clusters
    • After: OAM as core application model to manage stateful applications, e.g. Kafka,
    Zookeeper and Database on any cluster
    Share with other platforms

    View Slide

  27. OAM + Alibaba Serverless Platform
    • Before: in-house app definition, reinvent every wheel for serverless
    • After: OAM as unified definition of various workload types, and reuse traits from may
    other systems like EDAS.
    feature OAM Category Description
    • ContainerizedWorkload • component • Serverless container
    • Function • component • FaaS workload
    • Knative service (WIP) • component • Knative serving workload
    • traffic-mgmt • trait • Istio based Traffic shifting
    • cron-scaler • trait • Cron HPA
    • manual-scaler • trait • Manual scale
    • auto-scaler • trait • HPA
    • sls-project, sls-logstore, sls-
    index
    • component • Alibaba Cloud Resources as
    BaaS
    Separate concerns make
    developers accept serverless
    model way easier
    Migrate from K8s/PaaS to
    Serverless only need to update
    its trait
    Reusable components and
    traits to save engineering time
    Connect and consume cloud
    resources thru OAM based
    BaaS
    Share with other platforms

    View Slide

  28. Traits/Scopes
    Scale:
    - 10,000 nodes/cluster
    - 100,000 apps/cluster
    - 1,000,000 containers/cluster
    Applications:
    - 100,000 deploys/day
    - 500~1000 replicas/app
    Function
    Job Container
    Virtual
    Machine
    Gateway
    Route
    Traffic
    Alert
    Monitor
    Service
    Binding
    Rollout
    Ingress
    interpretability
    Infra Ops
    Application
    Configuration
    Application
    Configuration
    Application
    Configuration
    PaaS A PaaS B Serverless C
    Components
    Traits Components Traits Components Traits Components
    Application Management At Alibaba Today
    Developers
    Operators

    View Slide

  29. Kubernetes in Alibaba with OAM
    Modular, reusable, standardized, and
    manageable platform capabilities, best
    engineering efficiency
    Clear separate of concerns in the workflow of
    app management, best communication and
    collaboration efficiency
    For Platform
    • Simple, team-centric and unified experience
    • Enjoy “unlimited” capabilities provided by k8s
    ecosystem - the “open” PaaS
    For end users (Dev and Ops)
    Minimal migration effort from PaaS/K8s to OAM
    platform
    The simplest and unified approach to integrate
    and consume Alibaba Cloud Resources

    View Slide

  30. Thank You
    Application is the next hype!

    View Slide