Extending Kubernetes With Elixir

Transcript

1. EXTENDING K8S WITH ELIXIR* 1 — *or any other language you are comfortable with
2. PRESENTER ROMAN HEINRICH TEAMLEAD AT TALON.ONE Devops/SRE/Elixir/Sleep ENTHUSIAST @mindreframer TWITTER/GITHUB 2
3. GOALS OF THIS TALK ๏ explain why K8S plays such a critical role nowadays ๏ share interesting and surprising facts about K8S ๏ encourage to play with K8S ⚽ 3
4. DISCLAIMER Due to time constaints (20 mins) and the broad scope of the topic there will be quite some hand-waving involved. ⾠ There will be information gaps and unclarities. ⾠ That's OK. The goal is to trigger your curiosity! 4
5. ALSO... There is no YAML/source code on slides! 5
6. OUTLINE ๏ WHY? ๏ K8S history ๏ K8S architecture - Basic principles ๏ K8S Operator pattern ๏ Elixir tooling 6
7. WHO ... 7
8. ... HAS HEARD ABOUT KUBERNETES? 8
9. ... IS USING KUBERNETES IN PRODUCTION? 9
10. ... HAS IMPLEMENTED AUTOMATION ON TOP OF KUBERNETES? 10
11. WHY SHOULD YOU CARE? 11
12. Kubernetes eliminates infrastructure lock-in by providing core capabilities for containers without imposing restrictions. 12
13. In the future, choosing a cloud provider could be just a matter of which one offers the best performance/cost 13
14. “SUPERPOWERS” OF KUBERNETES - EXTENSIBILITY AND AUTOMATION 14
15. KUBERNETES: VERY BRIEF HISTORY 15
16. 2003-2004: BIRTH OF THE BORG SYSTEM @ GOOGLE ๏ large-scale internal cluster management system ๏ hundreds of thousands of jobs ๏ from many thousands of applications ๏ across many clusters, each with up to tens of thousands of machines 16 — https://blog.risingstack.com/the-history-of-kubernetes/
17. 2013: FROM BORG TO OMEGA ๏ cluster management system ๏ flexible, scalable scheduler for large compute clusters 17 — https://blog.risingstack.com/the-history-of-kubernetes/
18. 2014: GOOGLE INTRODUCES KUBERNETES ๏ June 7: Initial release - first github commit for Kubernetes ๏ July 10: Microsoft, RedHat, IBM, Docker joins the Kubernetes community. 18 — https://blog.risingstack.com/the-history-of-kubernetes/
19. K8S V0.1 WAS... ๏ a lean yet powerful open-source container manager ๏ deploys containers into a fleet of machines ๏ provides health management and replication capabilities ๏ makes it easy for containers to connect to one another and the outside world 19 — https://cloudplatform.googleblog.com/2014/06/an-update-on-container-support-on-google-cloud-platform.html
20. WHAT IS K8S? (EMPHASIS MINE) 20 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
21. Additionally, Kubernetes is not a mere orchestration system. 21 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
22. In fact, it eliminates the need for orchestration. 22 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
23. The technical definition of orchestration is execution of a defined workflow: first do A, then B, then C. 23 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
24. In contrast, Kubernetes is comprised of a set of independent, composable control processes that continuously drive the current state towards the provided desired state. 24 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
25. It shouldn’t matter how you get from A to C. 25 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
26. Centralized control is also not required. 26 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
27. This results in a system that is easier to use and more powerful, robust, resilient, and extensible. 27 — https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
28. K8S ARCHITECTURE 28
29. 29 — https://www.youtube.com/watch?v=uUNck0TEVPY - Kubernetes Architecture
30. IS THIS DIAGRAM HELPFUL? Probably not... 30
31. MAYBE THE FOLLOWING MENTAL MODEL HELPS 31
33. If you start thinking about Kubernetes as a fully event-driven system, there's answers to so many "Why"'s 33 — https://twitter.com/embano1/status/1067537816324845569
34. 34 — https://twitter.com/embano1/status/1067537816324845569
35. Think of the API server as an immutable (replicated) log (queue) and stream of events. 35 — https://twitter.com/embano1/status/1067537816324845569
36. Events are facts that can be causally related ("happened-before") or not related at all (then we say they happened "concurrently"). etcd is an implementation detail. 36 — https://twitter.com/embano1/status/1067537816324845569
37. All controllers, e.g. the scheduler, deployment controller, endpoint controller, Kubelet, etc. can be understood as consumers and/or producers (consumers can be producers as well, and vice versa). 37 — https://twitter.com/embano1/status/1067537816324845569
38. Consumers specify the objects (and optionally namespace) they want to receive events from the API server. 38 — https://twitter.com/embano1/status/1067537816324845569
39. This is called a "watch" in Kubernetes. Think of the combination of object+namespace as a dedicated (virtual) event queue the API server handles. 39 — https://twitter.com/embano1/status/1067537816324845569
40. Consumers and producers don't know about each other as they're fully decoupled (by the log) and autonomous. 40 — https://twitter.com/embano1/status/1067537816324845569
41. This makes the whole system extremely scalable, robust and extensible (flexible). 41 — https://twitter.com/embano1/status/1067537816324845569
42. Thus, by design, it's a fully asynchronous and eventual consistent platform. 42
43. Information takes time to propagate from consumer(s) to producer(s). The picture above shows this where the HPA hasn't caught up with the metrics. 43
44. 44 — https://twitter.com/embano1/status/1067537816324845569
45. There's NO guarantee that the system will converge to the desired state (even if you got an "OK"/ACK from the control plane). 45 — https://twitter.com/embano1/status/1067537816324845569
46. Eg: kubectl scale deploy --replicas n (with cpu requests > cluster_capacity). You'll get a HTTP 200, even though there is no capacity left. 46 — https://twitter.com/embano1/status/1067537816324845569
47. Controllers are stateless. 47 — https://twitter.com/embano1/status/1067537816324845569
48. For efficiency and speed received events are placed in an in-memory buffer (typically also modelled as a queue) per controller. 48 — https://twitter.com/embano1/status/1067537816324845569
49. What if a controller crashes as it does not persist state? 49 — https://twitter.com/embano1/status/1067537816324845569
50. This is not needed! The event-driven design will replay all (appropriate) events when the controller starts, also known as event-sourcing. 50 — https://twitter.com/embano1/status/1067537816324845569
51. This is also very useful as events are "at most once" delivered, i.e. could be lost during transmission. 51 — https://twitter.com/embano1/status/1067537816324845569
52. Kubernetes is level-triggered. If an event gets lost during transmission (e.g. network issue), next time there's a sync you're going to receive the desired state 52 — https://twitter.com/embano1/status/1067537816324845569
53. Since information can get delivered more than once (e.g. after failure, resync, etc.) and controllers don't talk to each other directly, ... 53 — https://twitter.com/embano1/status/1067537816324845569
54. ... there's a potential of race conditions when state is to be changed (e.g. a write to same object from different controllers). 54 — https://twitter.com/embano1/status/1067537816324845569
55. This is called optimistic concurrency and needs to be handled in the application layer (i.e. in each controller logic). 55 — https://twitter.com/embano1/status/1067537816324845569
56. Idempotency and compare and set (based on monotonically increasing resource versions) are patterns used here. 56 — https://twitter.com/embano1/status/1067537816324845569
57. THIS ARCHITECTURE MAKES K8S EXTREMELY EXTENSIBLE 57
58. KUBERNETES CONTROL PLANE 58
59. The Control Plane maintains a record of all of the Kubernetes Objects in the system, and runs continuous control loops to manage those objects’ state. 59 — https://kubernetes.io/docs/concepts/#kubernetes-control-plane
60. At any given time, the Control Plane’s control loops will respond to changes in the cluster and work to make the actual state of all the objects in the system match the desired state that you provided. 60 — https://kubernetes.io/docs/concepts/#kubernetes-control-plane
61. CONTROL LOOPS ๏ respond to changes (watch-ing) ๏ mutate the "world" towards desired state ๏ cause any kind of side-effects ! 61
62. K8S OPERATOR PATTERN 62
63. Why Kubernetes Operators are a game changer 63 — https://blog.couchbase.com/kubernetes-operators-game-changer/
64. Kubernetes 1.7 has added an important feature called Custom Controllers 64 — https://blog.couchbase.com/kubernetes-operators-game-changer/
65. CUSTOM CONTROLLERS ENABLE DEVELOPERS TO ๏ extend and add new functionalities ๏ replace existent ones (like replacing kube-proxy for instance) ๏ automate administration tasks as if they were a native Kubernetes component 65 — https://blog.couchbase.com/kubernetes-operators-game-changer/
66. An Operator is nothing more than a set of application-specific custom controllers. So, why is it a game changer? 66
67. Well, controllers have direct access to Kubernetes API 67
68. which means they can monitor the cluster, change pods/services, scale up/ down and call endpoints of the running applications, all according to custom rules written inside those controllers. 68
69. That is the real power of Operators, they allow you to write an application to fully manage another 69
70. SIMILARITIES BETWEEN ELIXIR/ERLANG AND K8S ๏ best practices for reliable / highly available apps (OTP) ๏ used as control plane ๏ control loops == Erlang processes / GenServers ? 70
71. ELIXIR LIBRARIES 71
72. coryodaniel/k8s ๏ A Kubernetes library and HTTP client in Elixir (since Jan 11, 2019) ๏ sympathetic newcomer ๏ quite capable and written with the specific goal for writing K8S operators 72
73. obmarg/kazan ๏ Kubernetes API client for Elixir (since 26 Dec 2016) ๏ generates Modules via macros, hard (impossible?) to use with CRD 73
74. NOT IMPLEMENTED ๏ Custom resources ๏ Other forms of authentication ๏ Patching with application/json-patch+json or application/ strategic-merge-patch+jsoncontent types ๏ (... and more) 74
75. coryodaniel/bonny ๏ Kubernetes Operator SDK written in Elixir. Extend the Kubernetes API and implement CustomResourceDefinitions in Elixir ๏ very interesting, but quite bleeding-edge 75
76. https://metacontroller.app/ ๏ 100% language agnostic ๏ real framework ๏ HTTP + JSON reading / generating to start shipping operators ๏ would be my first recommendation to get your feet wet and quickly ship POC 76
77. ✂ CUT ✂ 77
78. THERE CAN BE PART #2 WITH MULTITENANT APPS RELATED DETAILS 78
79. ✌ THANKS FOR YOUR ATTENTION! 79
80. Q & A 80