Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why is Kubernetes On-Premises So Hard?!

Why is Kubernetes On-Premises So Hard?!

Discussion about what makes On-Prem so much harder than Cloud deployments using Kubernetes as a reference example.

Video of presentation:

Rob Hirschfeld

May 03, 2019

More Decks by Rob Hirschfeld

Other Decks in Technology


  1. @zehicle on-Prem: the Hard way Why is On-Prem Kubernetes So

    Much Harder? DevOpsDays Austin, Spring 2019
  2. @zehicle @zehicle CEO of RackN Co-Founder Digital Rebar Co-Host of

    L8istSh9y podcast Focused on improving data center operations with zero-touch automation of distributed bare metal infrastructure. Rob Hirschfeld
  3. @zehicle Running systems adds complexity Etcd Host Host Host API

    Load Balancer Multi-Node Cluster Docker Persistent Storage Overlay Networking TLS RBAC Autoscale Provision
  4. @zehicle On-Prem lacks Consistent support services Etcd Host Host Host

    API Load Balancer Multi-Node Cluster Docker Persistent Storage Overlay Networking TLS RBAC Autoscale Provision
  5. @zehicle Load Balancer (API Driven) Multiple Layers of Load Balancer

    Required Need to provide for API and applications Candidates: • Metal LB • Istio • External LB
  6. @zehicle Networking Kubernetes makes a lot of networking assumptions You

    may need VMs for isolation Prepare to fight with Docker about networking Candidates: • VMs for isolation • Pay for SDN layers • FFS! Use Kubernetes for the right things
  7. @zehicle Pace of Change Kubernetes has quarterly releases AND components

    it depends on are also releasing Expect clusters to have different versions Candidates: • Plan for Day 2 upgrades • Consider immutable deployments • Reduce blast radius with smaller clusters
  8. @zehicle TLS - certification management Kubernetes REQUIRES TLS WTF! Why

    is not standard? But, now you must manage/rotate certs Candidates: • Name servers + LetsEncrypt? • Certificate Management Tooling
  9. @zehicle DNS Systems really expect names! (TLS, LB, etc) Dynamic

    apps mean dynamic registrations Oh… are you thinking about IPv6 too? Candidates: • OpenDNS integrated into Kubernetes • Need to integrate & delegate domains
  10. @zehicle Storage Multiple concerns! Local (ephemeral) storage is important Remote

    storage is evolving with multiple approaches Reminder: storage = networking Candidates: • Plan for solution to be short term
  11. @zehicle Docker/Containerd are relatively new Plan for latest O/S to

    be required Firmware will actually matter Stuffing in VMs may not be enough Candidates: • Automate, automate, automate • Immutable imaging helps • Consult your distro requirements Operating System & Host Provisioning
  12. @zehicle Distro & Support Distros have tentacles If using a

    distro / appliance it needs to fit into your infrastructure. Expect to pay for distro AND consulting Candidates: • Gingerbread man… • Look at your broader needs • Trunk distros are OK, but...
  13. @zehicle There are some solutions... • Kubernetes (any platform) is

    not an island • Integration is key - make sure to plan for it • Fix root cause issues (bare metal / provisioning) • Start with Day 2 / Lifecycle questions • Worth Repeating: FFS! Use Kubernetes for the right things If you want to see how RackN is thinking about this, please check out our Kubernetes + Rebar Integration (KRIB).
  14. @zehicle BONUS: Does Size Matter? Big vs Small Clusters Smaller

    clusters are more common; however this may reflect small is the starting point. For on-prem, small clusters on VMs makes a lot of sense; however, I expect a shift to large clusters to replace cloud for the data center underlay.