Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why is Kubernetes On-Premises So Hard?!

Why is Kubernetes On-Premises So Hard?!

Discussion about what makes On-Prem so much harder than Cloud deployments using Kubernetes as a reference example.

Video of presentation:
https://www.youtube.com/watch?v=S9KQn5-wq6o

Rob Hirschfeld

May 03, 2019
Tweet

More Decks by Rob Hirschfeld

Other Decks in Technology

Transcript

  1. @zehicle
    on-Prem: the Hard way
    Why is On-Prem Kubernetes So Much Harder?
    DevOpsDays Austin, Spring 2019

    View Slide

  2. @zehicle
    Cloud
    On-Prem

    View Slide

  3. @zehicle
    @zehicle
    CEO of RackN
    Co-Founder Digital Rebar
    Co-Host of L8istSh9y podcast
    Focused on improving data center operations with zero-touch
    automation of distributed bare metal infrastructure.
    Rob Hirschfeld

    View Slide

  4. @zehicle
    It’s a simple architecture
    Etcd
    Host Host Host
    API

    View Slide

  5. @zehicle
    Openstack: “hold my beer”

    View Slide

  6. @zehicle
    Kubernetes is not hard.

    View Slide

  7. @zehicle
    Running systems adds complexity
    Etcd
    Host Host Host
    API
    Load
    Balancer
    Multi-Node
    Cluster
    Docker
    Persistent
    Storage
    Overlay
    Networking
    TLS
    RBAC
    Autoscale
    Provision

    View Slide

  8. @zehicle
    Kubernetes is not hard.
    Operations is hard.

    View Slide

  9. @zehicle
    On-Prem lacks Consistent support services
    Etcd
    Host Host Host
    API
    Load
    Balancer
    Multi-Node
    Cluster
    Docker
    Persistent
    Storage
    Overlay
    Networking
    TLS
    RBAC
    Autoscale
    Provision

    View Slide

  10. @zehicle
    Kubernetes is not hard.
    Operations is hard.
    Operations on-Prem is very hard.

    View Slide

  11. @zehicle
    Load Balancer (API Driven)
    Multiple Layers of Load Balancer Required
    Need to provide for API and applications
    Candidates:
    ● Metal LB
    ● Istio
    ● External LB

    View Slide

  12. @zehicle
    Networking
    Kubernetes makes a lot of networking assumptions
    You may need VMs for isolation
    Prepare to fight with Docker about networking
    Candidates:
    ● VMs for isolation
    ● Pay for SDN layers
    ● FFS! Use Kubernetes for the right things

    View Slide

  13. @zehicle
    Pace of Change
    Kubernetes has quarterly releases
    AND components it depends on are also releasing
    Expect clusters to have different versions
    Candidates:
    ● Plan for Day 2 upgrades
    ● Consider immutable deployments
    ● Reduce blast radius with smaller clusters

    View Slide

  14. @zehicle
    TLS - certification management
    Kubernetes REQUIRES TLS
    WTF! Why is not standard?
    But, now you must manage/rotate certs
    Candidates:
    ● Name servers + LetsEncrypt?
    ● Certificate Management Tooling

    View Slide

  15. @zehicle
    DNS
    Systems really expect names! (TLS, LB, etc)
    Dynamic apps mean dynamic registrations
    Oh… are you thinking about IPv6 too?
    Candidates:
    ● OpenDNS integrated into Kubernetes
    ● Need to integrate & delegate domains

    View Slide

  16. @zehicle
    Storage
    Multiple concerns!
    Local (ephemeral) storage is important
    Remote storage is evolving with multiple approaches
    Reminder: storage = networking
    Candidates:
    ● Plan for solution to be short term

    View Slide

  17. @zehicle
    Docker/Containerd are relatively new
    Plan for latest O/S to be required
    Firmware will actually matter
    Stuffing in VMs may not be enough
    Candidates:
    ● Automate, automate, automate
    ● Immutable imaging helps
    ● Consult your distro requirements
    Operating System & Host Provisioning

    View Slide

  18. @zehicle
    Distro & Support
    Distros have tentacles
    If using a distro / appliance it needs
    to fit into your infrastructure.
    Expect to pay for distro AND consulting
    Candidates:
    ● Gingerbread man…
    ● Look at your broader needs
    ● Trunk distros are OK, but...

    View Slide

  19. @zehicle
    What are we
    going to DO
    about it?

    View Slide

  20. @zehicle
    There are some solutions...
    ● Kubernetes (any platform) is not an island
    ● Integration is key - make sure to plan for it
    ● Fix root cause issues (bare metal / provisioning)
    ● Start with Day 2 / Lifecycle questions
    ● Worth Repeating:
    FFS! Use Kubernetes for the right things
    If you want to see how RackN is thinking about this, please
    check out our Kubernetes + Rebar Integration (KRIB).

    View Slide

  21. @zehicle
    @zehicle
    RackN.com
    Rebar.Digital
    L8istSh9y.com
    Thanks! & Questions?
    Rob Hirschfeld

    View Slide

  22. @zehicle
    BONUS: Does Size Matter? Big vs Small Clusters
    Smaller clusters are more
    common; however this may
    reflect small is the
    starting point.
    For on-prem, small
    clusters on VMs makes a
    lot of sense; however, I
    expect a shift to large
    clusters to replace cloud
    for the data center
    underlay.

    View Slide