KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys

Join Carlos Camacho Gonzalez and Chris Short for a walkthrough about how to automate the preparation steps for deploying an HA OpenShift/OKD cluster using KubeInit.

https://github.com/kubeinit/kubeinit

This automation is based on reusing the best practices from the OpenStack/TripleO upstream community, using Ansible as the deployment vehicle.

Transcript

1. KubeInit: Bringing good practices from the
   OpenStack ecosystem to improve the way
   OKD/OpenShift deploys
   Carlos Camacho - Sept. 24th, 2020 - Red Hat
   1
   

   View Slide

2. https://github.com/kubeinit/kubeinit 2
   2
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   Carlos Camacho
   Red Hatter
   Ph.D. in Computer Engineering
   Software Engineer
   Upstream OpenStack contributor
   Digging into the k8s ecosystem
   ---
   blog: www.anstack.com
   IRC (freenode): ccamacho
   slack (k8s): ccamacho
   GitHub: ccamacho
   Who am I?
   

   View Slide

3. https://github.com/kubeinit/kubeinit 3
   3
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   Intro
   

   View Slide

4. https://github.com/kubeinit/kubeinit 4
   KubeInit provides Ansible
   playbooks and roles for the
   deployment and configuration of
   multiple Kubernetes distributions.
   The main goal of KubeInit is to have
   a fully automated way to deploy in
   a single command a curated list of
   prescribed architectures.
   About KubeInit
   KubeInit inherits
   some best practices
   from the OpenStack
   (TripleO) ecosystem
   4
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   OpenStack is a an open
   source cloud operating
   system managing compute,
   storage, and networking
   resources throughout a
   datacenter using APIs.
   

   View Slide

5. https://github.com/kubeinit/kubeinit 5
   5
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   Why?
   Working in other research topics I
   had the necessity to easily deploy
   complex Kubernetes architectures,
   hitting the k8s learning curve, and
   having to deal with all the steps
   prior to the cluster deployment.
   Science?
   To be convincing, a scientific paper needs to provide evidence that the results are reproducible.
   - FIRE AND FORGET -
   This evidence might come from repeating the whole experiment independently
   several times, or from performing the experiment in such a way that independent
   data are obtained and a formal procedure of statistical inference can be applied —
   usually confidence intervals (CIs) or statistical significance testing.
   

   View Slide

6. https://github.com/kubeinit/kubeinit 6
   6
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   When?
   I started to work on the cluster automation back in June 2018,
   for a research project called Pystol, where the
   installation/deployment project grew that much that it became
   its own project.
   Submitted to: IEEE Transactions on Cloud Computing
   Status: Under review
   Title: Chaos as a Software Product Line - A platform for
   improving open hybrid-cloud systems resiliency
   

   View Slide

7. https://github.com/kubeinit/kubeinit 7
   7
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   The logo
   Do you remember, Fire and forget ?
   The logo mimics a ‘cubic’ kamehameha from the mythical
   Japanese anime TV series Dragon Ball.
   The Kamehameha is formed when the ki is concentrated into a
   single point for then releasing all the energy at once (our
   deployment command).
   The demo
   starts now!
   

   View Slide

8. https://github.com/kubeinit/kubeinit 8
   8
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   The demo
   -- Steps --
   Clone the repo:
   git clone https://github.com/Kubeinit/kubeinit.git
   cd kubeinit
   Run the playbook:
   time ansible-playbook \
   --user root \
   -v -i ./hosts/okd/inventory \
   --become \
   --become-user root \
   ./playbooks/okd.yml
   

   View Slide

9. https://github.com/kubeinit/kubeinit 9
   9
   KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
   tripleo-ansible/openstack-ansible
   automatic documentation, roles skeleton, features
   distribution as roles
   tripleo-upgrade
   molecule tests architecture
   os-migrate
   linters architecture, unit tests structure,
   downstream CI automation
   tripleo-validations
   validations organization based on components
   Reduce, Reuse,
   and Refactor
   from the
   OpenStack
   ecosystem.
   

   View Slide

10. https://github.com/kubeinit/kubeinit 10
    10
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Components
    

    View Slide

11. https://github.com/kubeinit/kubeinit 11
    11
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Arch
    Infrastructure drivers
    k8s distribution
    Apache WS
    FreeIPA
    NFS
    HAProxy
    Bind9
    ...
    Validations
    KubeVirt
    Supported services:
    ● Host OS: CentOS/Fedora, Debian/Ubuntu
    ● Infrastructure drivers: Libvirt
    ● Kubernetes distributions: OKD
    ● External services: Bind9, HAProxy, Apache WS, Validations
    

    View Slide

12. https://github.com/kubeinit/kubeinit 12
    12
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Arch
    Infrastructure drivers
    k8s distribution
    Apache WS
    FreeIPA
    NFS
    HAProxy
    Bind9
    ...
    Validations
    Supported services:
    ● Host OS: CentOS/Fedora, Debian/Ubuntu
    ● Infrastructure drivers: Libvirt + AWS
    ● Kubernetes distributions: OKD + Kubernetes
    ● External services: Bind9, HAProxy, Apache WS, Validations + FreeIPA + SDN +
    Offline deployments
    KubeVirt
    

    View Slide

13. https://github.com/kubeinit/kubeinit 13
    13
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Docs
    docs.kubeinit.com
    ● Based in Sphinx (reStructuredText).
    ● Based in Read the Docs Sphinx Theme.
    ● Automatically render the docs from the
    roles and modules with an extension.
    ● Built on each merge.
    ● Merged in the gh-pages branch.
    ● Integrated in GitHub actions.
    ● Linted
    

    View Slide

14. https://github.com/kubeinit/kubeinit 14
    14
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Roles ansible-playbook \
    -i 'localhost,' \
    role-addition.yml \
    -e ansible_connection=local \
    -e role_name=kubeinit-example
    Defaults Vars
    _skeleton_role_
    Tasks
    Meta Molecule Docs
    

    View Slide

15. https://github.com/kubeinit/kubeinit 15
    15
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    CI
    Unit tests EndToEnd tests Molecule tests Docs build tests
    Linters
    

    View Slide

16. https://github.com/kubeinit/kubeinit 16
    16
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    CI
    Unit tests
    Molecule tests
    Docs build tests
    Linters
    ● Based in GitHub actions.
    ● They run on [push|pull_request].
    ● Results in between 2 and 4 minutes.
    ● All the code in theory can be covered:
    ○ Ansible automation (functional): Molecule
    ○ Python modules: Unit test
    ○ Code style: Linters
    ○ Documentation: Build test
    

    View Slide

17. https://github.com/kubeinit/kubeinit 17
    17
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    CI
    EndToEnd tests
    ● They run in an internal GitLab instance (runner).
    ● They run on demand by adding custom tags.
    ● The check pipeline runs each 15 minutes
    ● For security reasons they are outside GitHub
    ○ Users can run malicious code on PR, or when
    forking the code,so it runs externally triggered by
    tags.
    1. Two simple scripts: ci/launch_e2e.py & ci/run.sh
    2. A check pipeline executes launch_e2e.py each ~15 minutes (from GitLab).
    3. If there is a PR with a custom tag i.e. okd-libvirt-deploy, then:
    a. Configure the job.
    b. Launch the job.
    c. Write back the result to the specific PR.
    How it works?
    

    View Slide

18. https://github.com/kubeinit/kubeinit 18
    18
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Validations
    ● A role to:
    ○ Run pre deployment checks.
    ○ Run post deployment checks.
    ● Examples:
    ○ RAM availability.
    ○ Disk availability.
    ○ Endpoints check.
    ○ Nodes readiness.
    ○ Persistent storage status.
    ○ Test application deployments.
    

    View Slide

19. https://github.com/kubeinit/kubeinit 19
    19
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Deep dive
    

    View Slide

20. https://github.com/kubeinit/kubeinit 20
    20
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Net
    

    View Slide

21. https://github.com/kubeinit/kubeinit 21
    21
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Net
    

    View Slide

22. https://github.com/kubeinit/kubeinit 22
    22
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Bind
    External view
    DNS internal view
    

    View Slide

23. https://github.com/kubeinit/kubeinit 23
    23
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Bind
    /etc/named/zones/internals/db.watata.kubeinit.local
    api.watata.kubeinit.local. IN A 10.0.0.100
    api-int.watata.kubeinit.local. IN A 10.0.0.100
    *.apps.watata.kubeinit.local. IN A 10.0.0.100
    console-openshift-console.apps.watata.kubeinit.local. IN A 10.0.0.100
    oauth-openshift.apps.watata.kubeinit.local. IN A 10.0.0.100
    

    View Slide

24. https://github.com/kubeinit/kubeinit 24
    24
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    HAProxy
    Note: All the internal cluster traffic is routed directly
    ● openshift-api-server (port 6443)
    ● machine-config-server (port 22623)
    ● ingress-http (port 80)
    ● ingress-https (port 443)
    

    View Slide

25. https://github.com/kubeinit/kubeinit 25
    25
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    3rd-party
    Any 3rd party software should be
    deployed using k8s objects as
    additional roles.
    Currently integrated: KubeVirt
    

    View Slide

26. https://github.com/kubeinit/kubeinit 26
    26
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Conclusions
    

    View Slide

27. https://github.com/kubeinit/kubeinit 27
    27
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    Next steps/ Q&A
    

    View Slide

28. https://github.com/kubeinit/kubeinit 28
    28
    KubeInit: Bringing good practices from the OpenStack ecosystem to improve the way OKD/OpenShift deploys
    ● Find people interested in giving
    feedback, testing it, hacking it,
    raising issues, pushing PRs.
    ● Add other k8s distributions, like,
    plain HA kubernetes.
    ● Offline deployments (WIP).
    ● Improving testing coverage, unit
    tests, molecule tests.
    ● Roles refactor.
    ● Paths relative to the collection.
    ● Run the playbook from a container.
    ● Push mode -> PR mode.
    Next steps?
    Integration with
    other third party
    services, like,
    FreeIPA
    

    View Slide

29. Thank you!
    For more details, doubts,
    requests, contributions or
    anything you might need,
    please keep in touch! :)
    QUESTIONS
    ?
    Where to look:
    www.kubeinit.com
    docs.kubeinit.com
    https://github.com/kubeinit
    IRC (freenode): #kubeinit
    29
    If you liked this presentation or if you are
    interested in the project, please help us to
    give it more visibility by starring it!
    https://github.com/kubeinit/kubeinit
    blog: www.anstack.com
    IRC (freenode): ccamacho
    slack (k8s): ccamacho
    GitHub: ccamacho
    

    View Slide