Kubernetes Chaos Engineering: Lessons Learned in Networking

When you deploy an application in Kubernetes, your code ends up running on one or more worker nodes. A node may be a physical machine or VM such as AWS EC2 or Google Compute Engine and having several of them means you can run and scale your application across instances efficiently. When there is an incoming request, the cluster routes the traffic to one of the nodes using a network proxy. But what happens when network proxy crashes? Does the cluster still work? Can Kubernetes recover from the failure?
In this talk, you'll learn how the traffic is distributed within a Kubernetes cluster and what happens when the network proxy is misbehaving.

Transcript

1. kubernetes chaos engineering: lessons learned in networking @danielepolencic
3. once upon a time…
6. MASTER
7. MASTER NODE NODE
8. MASTER NODE NODE NODE
9. CPU Memory
10. CPU Memory
11. CPU Memory
12. CPU Memory
14. deployments in k8s
15. APP Pod 1
16. APP Pod 1 Pod 2 APP
17. APP Pod 1 Pod 2 Pod 3 APP APP
18. APP Pod 1 Pod 2 Pod 3 Pod 4 APP APP APP
19. Node 2 Node 1 Node 3
20. Node 2 Pod 1 APP APP Pod 2 APP Pod 3 APP Pod 4 Node 1 Node 3
21. designed to be fault tolerant #1
22. Node 2 APP Pod 1 APP Pod 2 APP Pod 3 APP Pod 4 Node 1 Node 3
23. Node 2 APP Pod 1 APP Pod 2 APP Pod 3 APP Pod 4 Node 1 Node 3
24. Node 2 APP Pod 1 APP Pod 5 APP Pod 3 APP Pod 4 Node 1 Node 3
25. designed to be fault tolerant #2
26. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic APP Pod 3 APP Pod 4 Node 1 Node 3
27. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic APP Pod 3 APP Pod 4 Node 1 Node 3
28. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic APP Pod 3 APP Pod 4 APP Pod 5 Node 1 Node 3
29. designed to scale
30. Ingress Pod 1 Pod 2 Service Incoming traffic
31. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic Node 1 Node 3
32. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic Node 1 Node 3
33. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic Node 1 Node 3
34. Node 2 APP Pod 1 APP Pod 2 Load balancer Incoming traffic Node 1 Node 3 ?
35. is the traffic lost?
36. is the traffic lost? times out?
37. is the traffic lost? times out? 404?
38. it works™
39. 1. the load balancer is app aware? !
40. RED RED
41. RED RED
42. RED RED
43. RED RED
44. 1. cloud vendor specific
45. 1. cloud vendor specific 2. single point of failure
46. 1. cloud vendor specific 2. single point of failure 3. hard to scale
47. 2. the master node routes the traffic? !
48. RED RED
49. RED RED
50. RED RED
51. 1. cloud vendor agnostic 2. single point of failure 3. hard to scale
52. 1. cloud vendor agnostic 2. single point of failure 3. hard to scale
53. 3. node is app aware? !
54. RED RED
55. RED RED Load balancer Load balancer Load balancer
56. RED RED Load balancer Load balancer Load balancer
57. RED RED Load balancer Load balancer Load balancer
58. RED RED Load balancer Load balancer Load balancer
59. RED RED Load balancer Load balancer Load balancer
60. 1. cloud vendor agnostic 2. redundancy built-in 3. scales with nodes
61. 1. cloud vendor agnostic 2. redundancy built-in 3. scales with nodes
62. kube-proxy
64. kube-proxy
65. how does it know that the app is not on the node?
66. Create a 2 Pods deployment
67. API SERVER
68. API SERVER
69. API SERVER
70. API SERVER Deployment #1
71. CONTROLLER MANAGER API SERVER Deployment #1
72. CONTROLLER MANAGER API SERVER Deployment #1
73. CONTROLLER MANAGER API SERVER Deployment #1 Pod #1 Pod #2
74. CONTROLLER MANAGER API SERVER Deployment #1 Pod #1 PENDING Pod #2 PENDING
75. CONTROLLER MANAGER SCHEDULER API SERVER Deployment #1 Pod #1 PENDING Pod #2 PENDING
76. CONTROLLER MANAGER SCHEDULER API SERVER Deployment #1 Pod #1 PENDING Pod #2 PENDING
77. CONTROLLER MANAGER SCHEDULER API SERVER Deployment #1 Pod #1 SCHEDULED Pod #2 SCHEDULED
78. the node creates the container
79. Deployment with 2 replicas please kubelet kubelet
80. Deployment with 2 replicas please Is there anything for me? kubelet kubelet
81. Docker daemon Deployment with 2 replicas please
82. Deployment with 3 replicas please RED RED
83. Deployment with 3 replicas please RED RED
84. the kubelet keeps the control plane informed
85. CONTROLLER MANAGER SCHEDULER API SERVER Deployment #1 Pod #1 RUNNING Pod #2 RUNNING
86. Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2
87. Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2
88. the list is up to date
89. Pod name Status Node Pod 1 RUNNING worker1
90. Pod name Status Node Pod 1 RUNNING worker1 Pod 2 RUNNING worker2
91. Pod name Status Node Pod 1 RUNNING worker1 Pod 2 RUNNING worker2
92. Pod name Status Node Pod 1 RUNNING worker1 Pod 2 RUNNING worker2 Pod 3 RUNNING worker3
93. Pod name Status Node Pod 1 RUNNING worker1 Pod 2 RUNNING worker2 Pod 3 RUNNING worker3
94. Pod name Status Node Pod 1 RUNNING worker1 Pod 2 RUNNING worker2
95. more lists
96. Ingress Pod 1 Pod 2 Service Incoming traffic 10.0.1.1:3000 10.0.1.2:3000
97. Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Service name IP Endpoints Service 1 172.17.0.1 10.0.1.1:3000, 10.0.1.2:3000, 10.0.2.1:3000 Service 2 172.17.0.2 10.0.2.2:8080 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2
98. kube-proxy reroutes the traffic
99. ? RED RED
100. ? RED RED
101. ? RED RED Pod name IP Node Pod 1 10.0.1.1 worker2 Pod 2 10.0.2.1 worker3 Service name Endpoints Service 1 Pod1, Pod2
102. RED RED Pod name IP Node Pod 1 10.0.1.1 worker2 Pod 2 10.0.2.1 worker3 Service name Endpoints Service 1 Pod1, Pod2
103. that was quick! !
104. !
105. ! what if… split brain
106. RED RED
107. RED RED
108. RED RED
109. 1. almost works
110. 1. almost works 2. stale routing table
111. 1. almost works 2. stale routing table 3. can recover
112. ! what if… kube-proxy crashes
113. RED
114. RED RED Pod name Status Node Pod 1 RUNNING worker1 Old routing table
115. RED
116. RED Old routing table Pod name Status Node Pod 1 RUNNING worker1
117. kube-proxy as daemonset
118. 1. ☸ respawns it
119. 1. ☸ respawns it 2. almost works
120. 1. ☸ respawns it 2. almost works 3. stale routing table
121. ! what if… routing table is lost
122. "lost"
124. RED
125. 1. ssh into node
126. 1. ssh into node 2. drop routing table
127. 1. ssh into node 2. drop routing table 3. observe
128. monitor the app ~$ while sleep 1 do date +%X curl -sS http://<balancer_ip>/ done
129. 14:39:41 Hello world!
130. 14:39:41 Hello world! 14:39:42 Hello world!
131. 14:39:41 Hello world! 14:39:42 Hello world! 14:39:43 Hello world!
132. 14:39:41 Hello world! 14:39:42 Hello world! 14:39:43 Hello world! 14:39:44 Hello world!
133. 14:39:41 Hello world! 14:39:42 Hello world! 14:39:43 Hello world! 14:39:44 Hello world! 14:39:45 Hello world!
134. 14:39:41 Hello world! 14:39:42 Hello world! 14:39:43 Hello world! 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world!
135. drop routing table ~$ ssh <node ip> ~$ iptables -F
136. what would you expect?
137. 14:39:43 Hello world! 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world! 14:39:47 Hello world! # nothing...
138. 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world! 14:39:47 Hello world! # nothing... # nothing...
139. 14:39:45 Hello world! 14:39:46 Hello world! 14:39:47 Hello world! # nothing... # nothing... # nothing...
140. 14:39:45 Hello world! 14:39:46 Hello world! 14:39:47 Hello world! # nothing... # nothing... # nothing... 14:40:14 Hello world!
141. 14:39:46 Hello world! 14:39:47 Hello world! # nothing... # nothing... # nothing... 14:40:14 Hello world! 14:40:15 Hello world!
142. ~30 seconds of void
143. what if you skip the lb!
144. RED
145. monitor the app ~$ while sleep 1 do date +%X curl -sS http://<node_ip>/ done
146. 14:39:41 Hello world! 14:39:42 Hello world! 14:39:43 Hello world! 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world!
147. drop the routing table!
148. 14:39:42 Hello world! 14:39:43 Hello world! 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world! # nothing...
149. 14:39:43 Hello world! 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world! # nothing... curl: (28) Connection timed out after 10003 milliseconds
150. 14:39:44 Hello world! 14:39:45 Hello world! 14:39:46 Hello world! # nothing... curl: (28) Connection timed out after 10003 milliseconds curl: (28) Connection timed out after 10004 milliseconds
151. 14:39:45 Hello world! 14:39:46 Hello world! # nothing... curl: (28) Connection timed out after 10003 milliseconds curl: (28) Connection timed out after 10004 milliseconds 14:40:15 Hello world!
152. 14:39:46 Hello world! # nothing... curl: (28) Connection timed out after 10003 milliseconds curl: (28) Connection timed out after 10004 milliseconds 14:40:15 Hello world! 14:40:16 Hello world!
153. 1. curl times out at 10s
154. 1. curl times out at 10s 2. lb must be timing out > 10s
155. 1. curl times out at 10s 2. lb must be timing out > 10s 3. something fixed the routing table
156. 1. curl times out 10s 2. lb must be timing out > 10s 3. something fixed the routing table 4. why 30 seconds?
157. is it you, kube-proxy?
158. --iptables-sync-period --iptables-min-sync-period
159. --iptables-sync-period how often routing rules are refreshed (default 30s)
160. --iptables-min-sync-period minimum interval for refresh (default 10s)
161. ah!
162. RED Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2 Routing table
163. RED Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2 Routing table
164. RED Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2 Routing table
165. RED Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2 Routing table !
166. RED Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2 Routing table !
167. RED Pod name Status Node Endpoint Pod 1 RUNNING worker1 10.0.1.1 Pod 2 RUNNING worker1 10.0.1.2 Pod 3 RUNNING worker2 10.0.2.1 Pod 4 RUNNING worker2 10.0.2.2 Routing table
168. you can tweak the flags
169. ! lessons learned
170. 1. understand how works
171. ? TO: Service (172.17.0.1) FROM: Anywhere
172. ? TO: FROM: Anywhere Service (172.17.0.1)
173. ? Pod name IP Node Pod 1 10.0.1.1 worker2 Pod 2 10.0.2.1 worker3 Service name Endpoints Service 1 Pod1, Pod2 TO: FROM: Anywhere Service (172.17.0.1)
174. ? Pod name IP Node Pod 1 10.0.1.1 worker2 Pod 2 10.0.2.1 worker3 Service name Endpoints Service 1 Pod1, Pod2 TO: Pod1 (10.0.1.1) FROM: Anywhere Service (172.17.0.1)
175. 192.168.0.2 192.168.0.3 ? 10.0.1.1 TO: Pod1 (10.0.1.1) FROM: Anywhere Service (172.17.0.1)
176. 192.168.0.1 Destination Next hop 10.0.0.0/24 10.0.1.0/24 10.0.2.0/24 192.168.0.2 192.168.0.3 192.168.0.4 10.0.3.0/24 192.168.0.5 192.168.0.2 192.168.0.3 ? 10.0.1.1 TO: Pod1 (10.0.1.1) FROM: Anywhere Service (172.17.0.1)
177. 192.168.0.2 192.168.0.3 10.0.1.1 TO: FROM: Anywhere Service (172.17.0.1) Pod1 (10.0.1.1)
179. 2. learn by doing · https://github.com/DennyZhang/challenges- kubernetes · https://github.com/arush-sal/cka-practice- environment
180. 3. monitoring and alerting
181. thanks
184. QUESTIONS? @danielepolencic