Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys

KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys

Join Carlos Camacho Gonzalez and Chris Short for a walkthrough about how to automate the preparation steps for deploying an HA OpenShift/OKD cluster using KubeInit.

https://github.com/kubeinit/kubeinit

This automation is based on reusing the best practices from the OpenStack/TripleO upstream community, using Ansible as the deployment vehicle.

Red Hat Livestreaming

September 24, 2020
Tweet

More Decks by Red Hat Livestreaming

Other Decks in Technology

Transcript

  1. KubeInit: Bringing good practices from the
    OpenStack ecosystem to improve the way
    OKD/OpenShift deploys
    Carlos Camacho - Sept. 24th, 2020 - Red Hat
    1

    View Slide

  2. https://github.com/kubeinit/kubeinit 2
    2
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Carlos Camacho
    Red Hatter
    Ph.D. in Computer Engineering
    Software Engineer
    Upstream OpenStack contributor
    Digging into the k8s ecosystem
    ---
    blog: www.anstack.com
    IRC (freenode): ccamacho
    slack (k8s): ccamacho
    GitHub: ccamacho
    Who am I?

    View Slide

  3. https://github.com/kubeinit/kubeinit 3
    3
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Intro

    View Slide

  4. https://github.com/kubeinit/kubeinit 4
    KubeInit provides Ansible
    playbooks and roles for the
    deployment and configuration of
    multiple Kubernetes distributions.
    The main goal of KubeInit is to have
    a fully automated way to deploy in
    a single command a curated list of
    prescribed architectures.
    About KubeInit
    KubeInit inherits
    some best practices
    from the OpenStack
    (TripleO) ecosystem
    4
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    OpenStack is a an open
    source cloud operating
    system managing compute,
    storage, and networking
    resources throughout a
    datacenter using APIs.

    View Slide

  5. https://github.com/kubeinit/kubeinit 5
    5
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Why?
    Working in other research topics I
    had the necessity to easily deploy
    complex Kubernetes architectures,
    hitting the k8s learning curve, and
    having to deal with all the steps
    prior to the cluster deployment.
    Science?
    To be convincing, a scientific paper needs to provide evidence that the results are reproducible.
    - FIRE AND FORGET -
    This evidence might come from repeating the whole experiment independently
    several times, or from performing the experiment in such a way that independent
    data are obtained and a formal procedure of statistical inference can be applied —
    usually confidence intervals (CIs) or statistical significance testing.

    View Slide

  6. https://github.com/kubeinit/kubeinit 6
    6
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    When?
    I started to work on the cluster automation back in June 2018,
    for a research project called Pystol, where the
    installation/deployment project grew that much that it became
    its own project.
    Submitted to: IEEE Transactions on Cloud Computing
    Status: Under review
    Title: Chaos as a Software Product Line - A platform for
    improving open hybrid-cloud systems resiliency

    View Slide

  7. https://github.com/kubeinit/kubeinit 7
    7
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    The logo
    Do you remember, Fire and forget ?
    The logo mimics a ‘cubic’ kamehameha from the mythical
    Japanese anime TV series Dragon Ball.
    The Kamehameha is formed when the ki is concentrated into a
    single point for then releasing all the energy at once (our
    deployment command).
    The demo
    starts now!

    View Slide

  8. https://github.com/kubeinit/kubeinit 8
    8
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    The demo
    -- Steps --
    Clone the repo:
    git clone https://github.com/Kubeinit/kubeinit.git
    cd kubeinit
    Run the playbook:
    time ansible-playbook \
    --user root \
    -v -i ./hosts/okd/inventory \
    --become \
    --become-user root \
    ./playbooks/okd.yml

    View Slide

  9. https://github.com/kubeinit/kubeinit 9
    9
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    tripleo-ansible/openstack-ansible
    automatic documentation, roles skeleton, features
    distribution as roles
    tripleo-upgrade
    molecule tests architecture
    os-migrate
    linters architecture, unit tests structure,
    downstream CI automation
    tripleo-validations
    validations organization based on components
    Reduce, Reuse,
    and Refactor
    from the
    OpenStack
    ecosystem.

    View Slide

  10. https://github.com/kubeinit/kubeinit 10
    10
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Components

    View Slide

  11. https://github.com/kubeinit/kubeinit 11
    11
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Arch
    Infrastructure drivers
    k8s distribution
    Apache WS
    FreeIPA
    NFS
    HAProxy
    Bind9
    ...
    Validations
    KubeVirt
    Supported services:
    ● Host OS: CentOS/Fedora, Debian/Ubuntu
    ● Infrastructure drivers: Libvirt
    ● Kubernetes distributions: OKD
    ● External services: Bind9, HAProxy, Apache WS, Validations

    View Slide

  12. https://github.com/kubeinit/kubeinit 12
    12
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Arch
    Infrastructure drivers
    k8s distribution
    Apache WS
    FreeIPA
    NFS
    HAProxy
    Bind9
    ...
    Validations
    Supported services:
    ● Host OS: CentOS/Fedora, Debian/Ubuntu
    ● Infrastructure drivers: Libvirt + AWS
    ● Kubernetes distributions: OKD + Kubernetes
    ● External services: Bind9, HAProxy, Apache WS, Validations + FreeIPA + SDN +
    Offline deployments
    KubeVirt

    View Slide

  13. https://github.com/kubeinit/kubeinit 13
    13
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Docs
    docs.kubeinit.com
    ● Based in Sphinx (reStructuredText).
    ● Based in Read the Docs Sphinx Theme.
    ● Automatically render the docs from the
    roles and modules with an extension.
    ● Built on each merge.
    ● Merged in the gh-pages branch.
    ● Integrated in GitHub actions.
    ● Linted

    View Slide

  14. https://github.com/kubeinit/kubeinit 14
    14
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Roles ansible-playbook \
    -i 'localhost,' \
    role-addition.yml \
    -e ansible_connection=local \
    -e role_name=kubeinit-example
    Defaults Vars
    _skeleton_role_
    Tasks
    Meta Molecule Docs

    View Slide

  15. https://github.com/kubeinit/kubeinit 15
    15
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    CI
    Unit tests EndToEnd tests Molecule tests Docs build tests
    Linters

    View Slide

  16. https://github.com/kubeinit/kubeinit 16
    16
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    CI
    Unit tests
    Molecule tests
    Docs build tests
    Linters
    ● Based in GitHub actions.
    ● They run on [push|pull_request].
    ● Results in between 2 and 4 minutes.
    ● All the code in theory can be covered:
    ○ Ansible automation (functional): Molecule
    ○ Python modules: Unit test
    ○ Code style: Linters
    ○ Documentation: Build test

    View Slide

  17. https://github.com/kubeinit/kubeinit 17
    17
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    CI
    EndToEnd tests
    ● They run in an internal GitLab instance (runner).
    ● They run on demand by adding custom tags.
    ● The check pipeline runs each 15 minutes
    ● For security reasons they are outside GitHub
    ○ Users can run malicious code on PR, or when
    forking the code,so it runs externally triggered by
    tags.
    1. Two simple scripts: ci/launch_e2e.py & ci/run.sh
    2. A check pipeline executes launch_e2e.py each ~15 minutes (from GitLab).
    3. If there is a PR with a custom tag i.e. okd-libvirt-deploy, then:
    a. Configure the job.
    b. Launch the job.
    c. Write back the result to the specific PR.
    How it works?

    View Slide

  18. https://github.com/kubeinit/kubeinit 18
    18
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Validations
    ● A role to:
    ○ Run pre deployment checks.
    ○ Run post deployment checks.
    ● Examples:
    ○ RAM availability.
    ○ Disk availability.
    ○ Endpoints check.
    ○ Nodes readiness.
    ○ Persistent storage status.
    ○ Test application deployments.

    View Slide

  19. https://github.com/kubeinit/kubeinit 19
    19
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Deep dive

    View Slide

  20. https://github.com/kubeinit/kubeinit 20
    20
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Net

    View Slide

  21. https://github.com/kubeinit/kubeinit 21
    21
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Net

    View Slide

  22. https://github.com/kubeinit/kubeinit 22
    22
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Bind
    External view
    DNS internal view

    View Slide

  23. https://github.com/kubeinit/kubeinit 23
    23
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Bind
    /etc/named/zones/internals/db.watata.kubeinit.local
    api.watata.kubeinit.local. IN A 10.0.0.100
    api-int.watata.kubeinit.local. IN A 10.0.0.100
    *.apps.watata.kubeinit.local. IN A 10.0.0.100
    console-openshift-console.apps.watata.kubeinit.local. IN A 10.0.0.100
    oauth-openshift.apps.watata.kubeinit.local. IN A 10.0.0.100

    View Slide

  24. https://github.com/kubeinit/kubeinit 24
    24
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    HAProxy
    Note: All the internal cluster traffic is routed directly
    ● openshift-api-server (port 6443)
    ● machine-config-server (port 22623)
    ● ingress-http (port 80)
    ● ingress-https (port 443)

    View Slide

  25. https://github.com/kubeinit/kubeinit 25
    25
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    3rd-party
    Any 3rd party software should be
    deployed using k8s objects as
    additional roles.
    Currently integrated: KubeVirt

    View Slide

  26. https://github.com/kubeinit/kubeinit 26
    26
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Conclusions

    View Slide

  27. https://github.com/kubeinit/kubeinit 27
    27
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Next steps/ Q&A

    View Slide

  28. https://github.com/kubeinit/kubeinit 28
    28
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    ● Find people interested in giving
    feedback, testing it, hacking it,
    raising issues, pushing PRs.
    ● Add other k8s distributions, like,
    plain HA kubernetes.
    ● Offline deployments (WIP).
    ● Improving testing coverage, unit
    tests, molecule tests.
    ● Roles refactor.
    ● Paths relative to the collection.
    ● Run the playbook from a container.
    ● Push mode -> PR mode.
    Next steps?
    Integration with
    other third party
    services, like,
    FreeIPA

    View Slide

  29. Thank you!
    For more details, doubts,
    requests, contributions or
    anything you might need,
    please keep in touch! :)
    QUESTIONS
    ?
    Where to look:
    www.kubeinit.com
    docs.kubeinit.com
    https://github.com/kubeinit
    IRC (freenode): #kubeinit
    29
    If you liked this presentation or if you are
    interested in the project, please help us to
    give it more visibility by starring it!
    https://github.com/kubeinit/kubeinit
    blog: www.anstack.com
    IRC (freenode): ccamacho
    slack (k8s): ccamacho
    GitHub: ccamacho

    View Slide