Why is Kubernetes On-Premises So Hard?!

Discussion about what makes On-Prem so much harder than Cloud deployments using Kubernetes as a reference example.

Video of presentation:
https://www.youtube.com/watch?v=S9KQn5-wq6o

Transcript

1. @zehicle on-Prem: the Hard way Why is On-Prem Kubernetes So Much Harder? DevOpsDays Austin, Spring 2019
2. @zehicle Cloud On-Prem
3. @zehicle @zehicle CEO of RackN Co-Founder Digital Rebar Co-Host of L8istSh9y podcast Focused on improving data center operations with zero-touch automation of distributed bare metal infrastructure. Rob Hirschfeld
4. @zehicle It’s a simple architecture Etcd Host Host Host API
5. @zehicle Openstack: “hold my beer”
6. @zehicle Kubernetes is not hard.
7. @zehicle Running systems adds complexity Etcd Host Host Host API Load Balancer Multi-Node Cluster Docker Persistent Storage Overlay Networking TLS RBAC Autoscale Provision
8. @zehicle Kubernetes is not hard. Operations is hard.
9. @zehicle On-Prem lacks Consistent support services Etcd Host Host Host API Load Balancer Multi-Node Cluster Docker Persistent Storage Overlay Networking TLS RBAC Autoscale Provision
10. @zehicle Kubernetes is not hard. Operations is hard. Operations on-Prem is very hard.
11. @zehicle Load Balancer (API Driven) Multiple Layers of Load Balancer Required Need to provide for API and applications Candidates: • Metal LB • Istio • External LB
12. @zehicle Networking Kubernetes makes a lot of networking assumptions You may need VMs for isolation Prepare to fight with Docker about networking Candidates: • VMs for isolation • Pay for SDN layers • FFS! Use Kubernetes for the right things
13. @zehicle Pace of Change Kubernetes has quarterly releases AND components it depends on are also releasing Expect clusters to have different versions Candidates: • Plan for Day 2 upgrades • Consider immutable deployments • Reduce blast radius with smaller clusters
14. @zehicle TLS - certification management Kubernetes REQUIRES TLS WTF! Why is not standard? But, now you must manage/rotate certs Candidates: • Name servers + LetsEncrypt? • Certificate Management Tooling
15. @zehicle DNS Systems really expect names! (TLS, LB, etc) Dynamic apps mean dynamic registrations Oh… are you thinking about IPv6 too? Candidates: • OpenDNS integrated into Kubernetes • Need to integrate & delegate domains
16. @zehicle Storage Multiple concerns! Local (ephemeral) storage is important Remote storage is evolving with multiple approaches Reminder: storage = networking Candidates: • Plan for solution to be short term
17. @zehicle Docker/Containerd are relatively new Plan for latest O/S to be required Firmware will actually matter Stuffing in VMs may not be enough Candidates: • Automate, automate, automate • Immutable imaging helps • Consult your distro requirements Operating System & Host Provisioning
18. @zehicle Distro & Support Distros have tentacles If using a distro / appliance it needs to fit into your infrastructure. Expect to pay for distro AND consulting Candidates: • Gingerbread man… • Look at your broader needs • Trunk distros are OK, but...
19. @zehicle What are we going to DO about it?
20. @zehicle There are some solutions... • Kubernetes (any platform) is not an island • Integration is key - make sure to plan for it • Fix root cause issues (bare metal / provisioning) • Start with Day 2 / Lifecycle questions • Worth Repeating: FFS! Use Kubernetes for the right things If you want to see how RackN is thinking about this, please check out our Kubernetes + Rebar Integration (KRIB).
21. @zehicle @zehicle RackN.com Rebar.Digital L8istSh9y.com Thanks! & Questions? Rob Hirschfeld
22. @zehicle BONUS: Does Size Matter? Big vs Small Clusters Smaller clusters are more common; however this may reflect small is the starting point. For on-prem, small clusters on VMs makes a lot of sense; however, I expect a shift to large clusters to replace cloud for the data center underlay.