$30 off During Our Annual Pro Sale. View Details »

Orchestructure: Building Kubernetes Operators with Ansible Talk

Orchestructure: Building Kubernetes Operators with Ansible Talk

An Operator is a method of packaging, deploying and automating the complete lifecycle management activities of a Kubernetes application. Put more simply, an Operator encodes human operational knowledge. It is designed to watch and respond to the resources in your cluster and enable your application to run as desired.

While powerful, Operators are usually written in Go and requires expertise in the advanced libraries and patterns used to write Kubernetes controllers. Ansible can help. It is a first class citizen in the Operator SDK providing a means of automating the deployment and management of Kubernetes applications on a cluster in an Ansible-native way. Ansible-based Operators provide a lower barrier to entry, faster iterations, and the power of Ansible and its ecosystem.

Combining Ansible and Kubernetes frees up application engineers to minimize the new skill sets required and maximize time to delivery. Furthermore, using the same tried and trusted Ansible tooling lets you automate and orchestrate your applications across both new and existing platforms with one simple language.

This talk introduces Operators with Ansible and demonstrates how you develop and deploy them to automate the management of complex Kubernetes application on a cluster.

Chris Short

August 28, 2019
Tweet

More Decks by Chris Short

Other Decks in Technology

Transcript

  1. Building Kubernetes
    Operators with
    Ansible
    CHRIS SHORT
    PRINCIPAL TECHNICAL MARKETING
    MANAGER, CLOUD PLATFORMS
    @ChrisShort

    View Slide

  2. The Ansible Operator SDK makes it
    easier to deploy and manage Kubernetes
    applications with native Ansible support

    View Slide

  3. Manage how containerized apps interact
    with other apps or the outside world
    Run distributed systems resiliently across
    a cluster of nodes
    Perform health checks
    Kubernetes Operators with Ansible
    What does Kubernetes help do?
    Scale your services up or down
    Perform graceful rolling updates
    Networking and routing of traffic
    Manages ephemeral and persistent
    storage volumes

    View Slide

  4. Pod
    Deployment
    DaemonSet
    ReplicaSet
    StatefulSet
    Job
    CronJob
    Kubernetes Operators with Ansible
    Kubernetes objects are persistent entities that represent the state
    of your cluster that you can mange with the K8s API
    EndPoints
    Ingress
    Service
    ConfigMap
    Secret
    StorageClass
    Volume
    Namespace
    NetworkPolicy
    PersistentVolume
    Role
    RoleBinding
    ServiceAccount
    Understanding Kubernetes objects

    View Slide

  5. Kubernetes Operators with Ansible
    apiVersion: v1
    kind: Pod
    metadata:
    name: example-app
    labels:
    app: example-app
    spec:
    containers:
    - name: example
    image: companyname/example:v1.2.0
    ports:
    - containerPort: 8000
    apiVersion: v1
    kind: Service
    metadata:
    name: example-service
    spec:
    selector:
    app: example-app
    ports:
    - protocol: TCP
    port: 80
    targetPort: 8000
    Kubernetes Object Definitions

    View Slide

  6. Extending Kubernetes with a
    Custom Resource Definition (CRD)
    apiVersion:
    cache.example.com/v1alpha1
    kind: Memcached
    metadata:
    name: example-memcached
    spec:
    size: 3
    Custom resources definition (CRD) is a
    powerful feature introduced into Kubernetes
    which enables users to add their own/custom
    objects to the Kubernetes cluster and use it
    like any other native Kubernetes objects.
    Kubernetes Operators with Ansible

    View Slide

  7. Ansible fits naturally into a
    Kubernetes environment

    View Slide

  8. Kubernetes and Ansible
    Kubernetes Operators with Ansible
    ● Both help make hard things easier through automation and orchestration
    ● Both are very active and widely used open source projects
    ● Both have vibrant communities working to solve common problems
    ● Both use YAML to describe the desired state of the world

    View Slide

  9. Kubernetes Operators with Ansible
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: foo
    namespace: default
    data:
    color: red
    - name: create foo configmap
    k8s:
    definition:
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: foo
    namespace: default
    data:
    color: “{{ color }}”
    YAML to describe the desired state of the world
    KUBERNETES/KUBECTL ANSIBLE

    View Slide

  10. Templating Kubernetes resource definitions
    with Ansible
    ---
    - name: create foo configmap
    k8s:
    definition: "{{ lookup('template', '/foo.yml') | from_yaml
    }}"

    View Slide

  11. What are
    Kubernetes Operators?

    View Slide

  12. Stateless is easy,
    Stateful is hard

    View Slide

  13. ● Encode human operational knowledge
    ● Automatically patch, upgrade, recover, and tune
    container-based apps and services
    ● Kubernetes-native
    ● Purpose-built for a specific application or service
    ● Enable “day 2” management
    Kubernetes Operators with Ansible
    Kubernetes Operators
    Operators simplify management of complex applications
    on Kubernetes

    View Slide

  14. Encoding and automating Ops knowledge
    WITH OPERATORS: PROACTIVE
    Continually adjusts to optimal state
    Automatically acts in milliseconds
    WITHOUT OPERATORS: REACTIVE
    Continually checks for anomalies
    Alert humans for response
    Requires manual change to fix
    Kubernetes Operators with Ansible

    View Slide

  15. K8S API CUSTOM
    RESOURCE & MY
    APP STATE
    WATCH
    EVENTS
    RECONCILE MY K8S
    APPLICATION
    The Operator Pattern
    Controller
    Kubernetes Operators with Ansible

    View Slide

  16. github.com/operator-framework
    Install, update, and manage
    Operators and their
    dependencies
    An open source toolkit to manage application instances
    on Kubernetes in an automated, scalable way
    Build Operators without
    specialized knowledge of
    the Kubernetes API
    Enable usage reporting
    for Operators
    Operator Framework
    Kubernetes Operators with Ansible

    View Slide

  17. Why build Operators
    with Ansible?

    View Slide

  18. Making it easier to deploy and manage Kubernetes apps in an Ansible-native way
    Ansible Operator SDK
    Kubernetes Operators with Ansible
    Use the Operator SDK to
    create a new skeleton
    Operator.
    operator-sdk new Add Ansible Content
    Use Ansible Roles and
    playbooks to manage
    lifecycle events for your
    containerized applications.
    operator-sdk build
    Use the Operator SDK to
    build and deploy your
    Operator to Kubernetes.

    View Slide

  19. EXISTING SKILLS & ECOSYSTEM
    Same tried & trusted Ansible tooling
    Utilize existing skills
    Supports cloud-native & traditional IT automation
    with one simple language
    Leverages vibrant existing ecosystem
    Why build Operators with Ansible?
    LOWER BARRIER OF ENTRY
    No programming required
    Faster iterations and easier maintenance
    Declarative state definitions like K8s
    Templating of resources
    Abstraction layer & helpers that reduces
    necessary K8s API experience
    Kubernetes Operators with Ansible

    View Slide

  20. Operator capability level
    Kubernetes Operators with Ansible
    Phase I Phase II Phase III Phase IV Phase V
    Basic Install
    Automated application
    provisioning and
    configuration management
    Seamless Upgrades
    Patch and minor version
    upgrades supported
    Full Lifecycle
    App lifecycle, storage
    lifecycle (backup, failure
    recovery)
    Deep Insights
    Metrics, alerts, log
    processing and workload
    analysis
    Auto Pilot
    Horizontal/vertical scaling,
    auto config tuning, abnormal
    detection, scheduling tuning

    View Slide

  21. K8S API MY K8S
    APPLICATION
    K8s Operator with Ansible
    WATCH
    FILE
    PLAYBOOK
    OR ROLE
    OPERATOR
    SDK
    BINARY
    Design overview
    Kubernetes Operators with Ansible
    CUSTOM
    RESOURCE & MY
    APP STATE

    View Slide

  22. K8S API
    CUSTOM
    RESOURCE
    MY K8S
    APPLICATION
    automatically
    updates resource
    status
    OPERATOR SDK BINARY
    K8s Operator with Ansible
    WATCHES.YML
    ANSIBLE
    RUNNER
    executes
    & reports
    Reconciliation with K8s Cluster
    event monitoring
    manages
    Kubernetes Operators with Ansible
    INFORMER
    REVERSE
    PROXY
    reads

    View Slide

  23. Developing your first
    Operator with Ansible

    View Slide

  24. Developing your first Operator with Ansible
    ● Initialize Your Operator With Ansible
    ○ $ operator-sdk new foo-operator
    --api-version=cache.example.com/v1alpha1 --kind=Foo --type=ansible
    ● Automate With Ansible
    ○ Create new roles and playbooks or reuse an existing one
    ● Define a watches file
    ○ Map a Kubernetes object to your Ansible content
    ● Build Your Operator
    ○ $ operator-sdk build foo-operator:v0.0.1
    ● Deploy Your Operator to a Kubernetes Cluster
    Kubernetes Operators with Ansible

    View Slide

  25. Initialize your Operator with Ansible
    $ operator-sdk new memcached-operator
    --api-version=cache.example.com/v1alpha1
    --kind=Memcached --type=ansible
    The new command in the Operator SDK using
    type=ansible will create an Ansible Role
    skelton, watches.yaml mapping file, CRD,
    deploy manifest for the Operator and basic
    tests using Molecule.
    Kubernetes Operators with Ansible

    View Slide

  26. Automate with Ansible
    ● Develop Ansible playbook or roles to reconcile your Kubernetes application
    and manage its lifecycle
    ● Data from the associated Custom Resource will be passed into the Ansible run
    by extra_vars for use with your tasks, templates, conditionals etc.
    ● Typically your automation will leverage the Ansible k8s module and other
    associated ones to interact with the Kubernetes cluster itself
    Kubernetes Operators with Ansible

    View Slide

  27. Ansible Roles
    Roles are a package of closely related
    Ansible content that can be shared more
    easily than plays alone:
    Improves readability & maintainability of complex plays
    Eases sharing, reuse and standardization of automation
    processes
    Enables Ansible content to exist independently of
    playbooks, projects -- even organizations
    Provides functional conveniences such as file path
    resolution and default values
    memcached/
    ├── defaults
    │ └── main.yml
    ├── files
    ├── handlers
    │ └── main.yml
    ├── meta
    │ └── main.yml
    ├── tasks
    │ └── main.yml
    ├── templates
    ├── tests
    └── vars
    └── main.yml
    Kubernetes Operators with Ansible

    View Slide

  28. Mapping Kubernetes events to Ansible automation
    Maps a Kubernetes Group Version Kind (GVK) to a role or playbook
    # watches.yaml
    ---
    version: v1alpha1
    group: cache.example.com
    kind: Memcached
    playbook:
    /path/to/playbook
    Kubernetes Operators with Ansible
    The “watches” file (watches.yaml)
    maps a Kubernetes object to your
    Ansible automation
    Associates the Kubernetes Group, Version, Kind
    (GVK) to an Ansible Role or Playbook
    The Operator SDK binary watches the cluster for
    matching events defined in the watches.yml
    Executes the associated Ansible content when an
    event occurs

    View Slide

  29. Custom resource spec to Ansible extra variables
    apiVersion:
    kind:
    metadata:
    name:
    spec:
    :
    ….
    status:
    :
    ….
    spec: values will be translated to
    Ansible extra_vars.
    status: will be a generic status
    recorded by the Ansible Operator
    SDK operator. This will use
    ansible-runner output to generate
    meaningful information for the user.
    Kubernetes Operators with Ansible

    View Slide

  30. Anatomy of Ansible-enabled Operator image
    your Operator image
    Ansible Role
    Ansible Role
    watches.yaml
    base Ansible Operator SDK image
    Operator SDK Binary
    ansible-runner
    Ansible
    Python + Libraries
    Kubernetes Operators with Ansible

    View Slide

  31. Build your Operator image
    $ operator-sdk build memcached-operator:v0.0.1

    ….
    Digest: sha256:6d3b4e6c9eca300277655f5bd2dcbcc33d12c3ac297a71aff4d5f723e09d606a
    Status: Downloaded newer image for quay.io/water-hole/ansible-operator:latest
    ---> ff7b5533bfd2
    Step 2/3 : COPY roles/ ${HOME}/roles/
    ---> d6ec895c8a6d
    Removing intermediate container eac3560f3dab
    Step 3/3 : COPY watches.yaml ${HOME}/watches.yaml
    ---> 060f91e9a843
    Removing intermediate container f680881ec972
    Successfully built 060f91e9a843
    Kubernetes Operators with Ansible

    View Slide

  32. Deploy your Operator
    ● Create RBAC and Service Account objects for the Operator
    ● Create the Operator Custom Resource Definition (CRD)
    ● Deploy the Operator to the cluster
    ● Create a Custom Resource (CR) to initialize the Operator

    View Slide

  33. Explore Operators
    OperatorHub.io is a home for
    the Kubernetes community to
    share Operators.
    Find an existing Operator or
    list your own today.
    Kubernetes Operators with Ansible

    View Slide

  34. Advanced
    Operator patterns

    View Slide

  35. Get started with Ansible: Get started with Operators:
    ansible.com/get-started
    ansible.com/community
    github.com/operator-framework/getting-started
    ansible.com/operators
    Next steps

    View Slide

  36. etcd Operator
    A great example of a sophisticated Kubernetes
    Operator using Ansible:
    github.com/water-hole/etcd-ansible-operator
    Memcached Operator
    Simple walkthrough for building an Operators using the
    Ansible Operator SDK and Kubernetes CRDs
    github.com/operator-framework/operator-sdk-sample
    s/tree/master/memcached-operator
    More resources

    View Slide

  37. Red Hat Certified Operators
    Red Hat Partners that certify their
    Operators have special status due to
    their backing of the Operator(s) with
    formal support, testing, and joint
    go-to-market activities.
    Kubernetes Operators with Ansible
    ● Register to Red Hat Connect
    ● Join TSANet
    ● For Each Operator Version Release:
    ○ Test
    ○ Submit
    ○ Release
    ○ Publish
    See the Operator Developer Guide for Red Hat
    Partners for more details on the Red Hat
    Certified Operators program.

    View Slide

  38. linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHat
    Thank you

    View Slide

  39. A SMARTER KUBERNETES PLATFORM
    Automated installation, patching, and updates from the OS on up*
    APPLICATIONS AND SERVICES
    ISV Operators
    Custom Operators (built w/Operator SDK)
    PLATFORM AND CLUSTER MANAGEMENT
    Automated updates for Kubernetes,
    monitoring, security, registry and more
    LINUX HOST
    Atomic, over-the-air updates for
    Red Hat CoreOS
    ANY INFRASTRUCTURE
    *coming soon
    ACROSS HYBRID / MULTI CLOUD DEPLOYMENTS
    Automated operations

    View Slide

  40. ● Supported model for running
    Playbooks in an Operator fashion
    ● Great for Ops teams that aren’t
    traditional devs
    ● Takes the human out of the loop
    ● Connects the playbooks to
    Kubernetes events like Node failures Tomcat Object
    apiVersion: apache.org/v1
    kind: Tomcat
    metadata:
    name: example-app
    Namespace: production
    spec:
    replicaCount: 10
    maxActiveSessions: 500
    Container Build
    Ansible
    Playbooks
    Operator Container
    v1.2.7 Tomcat Object
    apiVersion: apache.org/v1
    kind: Tomcat
    metadata:
    name: prod-1.2.7
    Namespace: production
    spec:
    replicaCount: 10
    maxActiveSessions: 1000
    Running Operator
    v1.2.7
    $ oc get Tomcats --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE
    production prod-1.2.7 1/1 Running 0 4d
    staging staging-v1.2.8 1/1 Running 1 2h
    ANSIBLE OPERATOR SDK Playbook
    Mapping

    View Slide