Real-World Kubernetes Deployments @ OSCON 2016

Transcript

1. Real World Kubernetes Deployments failure domains, upgrades, high-availability @coreoslinux @brandonphilips
2. Follow Along Instructions http://bit.ly/1XeUbMW Stickers Upfront Decorate your laptop, dog, kid, phone.
3. Brandon Philips CTO, CoreOS github.com/philips
5. Build, Store and Distribute your Containers quay.io
6. Linux
7. Secure the Internet MISSION
8. Separate Apps from OS STRATEGY
9. Make Servers Consistent STRATEGY
10. Tolerate Machine Failures STRATEGY
11. Make Servers Easy to Upgrade STRATEGY
12. Simplify Application Upgrades STRATEGY
18. Application Packaging 1
19. Abstract away app from the OS OS App
22. Linux at Scale 2
23. Patches to the OS and kernel are hard Retest after updates No automation SECURITY Dependency breakage Uptime risk APPLICATION
25. Auto-updating browsers fixed security We got HTML5 at the same time
26. Clustering 3
27. Operations Paradise Easy scale out Painless app upgrades Tolerant of machine failure
28. App Req/sec: 6,000 App Healthy: True
29. App Req/sec: 6,000 App Healthy: True
30. App Req/sec: 7,000 App Healthy: True
31. App Req/sec: 8,000 App Healthy: True
32. App Req/sec: 7,000 App Healthy: True
33. App Req/sec: 6,000 App Healthy: True
34. App Req/sec: 8,000 App Healthy: True
35. App Req/sec: 7,000 App Healthy: True
36. App Req/sec: 8,000 App Healthy: True
37. App Req/sec: 8,000 App Healthy: True
38. 3 Application packaging Clustering Linux at scale
39. 3 Application packaging Clustering Linux at scale
40. Follow Along Instructions https://github.com/philips/repositories 2016-OSCON-containers-at-scale
41. CoreOS+Kubernetes vagrant, aws, bare metal, etc coreos.com/kubernetes/docs/latest/
42. kubernetes architecture in practice
43. worker kubelet worker kubelet worker kubelet scheduler & API worker kubelet w ku t worker kubelet
44. worker kubelet worker kubelet scheduler & API
45. worker & API works on 1 node too
46. kube-aws Initial Cluster Setup
47. worker kubelet worker kubelet controller scheduler, etcd & API
48. Demo Boot up a Cluster
49. Demo Run an App
50. Demo Understand the Network
51. Domains Let's Talk About Failure
52. Failure domains are regions or components of the infrastructure which contain a potential for failure.
53. These regions can be physical or logical boundaries, and each has its own risks and challenges to architect for.
54. Failure Feud - Machine Failure - Network/Disks/RAM/Processor/Power Supply - Rack Failure - Network/Power - Data Center Failure - Network/Power/Fire/Semi-trucks - Internet Failure - Network/Political/Natural
55. Failure Analysis Kid Celebrating
57. Kid Hitting His Eye Failure Analysis - Failure is caused by human error - Celebration continues; eye unnecessary - Kid has two eyes can continue seeing - Brain elects new eye automatically
58. Primary Datastore etcd operations
59. /etc distributed hence, the name...
60. a clustered key-value store GET and SET operations
61. a building block for higher order systems primitives for building reliable distributed systems
62. Demo play.etcd.io
70. Failure Analysis etcd
71. worker kubelet worker kubelet scheduler & API
72. kube-aws high availability in cloud
73. scheduler & API EBS { ASG
74. etcd protects against - Machine Failure - Replication, automatic leader election - Flakey Disk Failure - CRC checksums on WAL files - Network Failure - Timeouts and linearized state machine
75. etcd does not protect against - Denial of Service - Future work on proxies - Lying etcd Peers - We do a ton of functional testing a hedge - Buggy or Broken Clients - Client deleting all keys requires restore from backup
76. Demo etcd restore backup
77. 1 2 3 4 { Log
78. 1 2 3 4 Entries
79. 1 2 3 4 Indexes
80. Kubernetes Control API Service, Scheduler, Controller Manager
81. Failure Analysis Kubernetes
82. Demo etcd down for API server
83. worker kubelet worker kubelet scheduler & API
84. scheduler & API
85. Demo etcd restore for API server
86. scheduler & API
87. Demo node partition from API
88. worker kubelet worker kubelet scheduler & API
89. Demo node scaling up
90. worker kubelet worker kubelet scheduler & API worker kubelet
91. Demo node scheduled outage API
92. worker kubelet worker kubelet scheduler & API
93. Demo node unplanned outage
94. worker kubelet worker kubelet scheduler & API
95. Demo node downgrade/upgrade outage
96. worker kubelet worker kubelet scheduler & API
97. Future Work Upstream Kubernetes and Elsewhere
98. Upstream rktnetes Auth/OIDC Node self-signed TLS
99. Scaling 15x scheduler performance 30k pods on 1k nodes SIG-scale
100. Automatic Node Drain Locksmith Design Doc
102. Performance etcd3 /ZooKeeper snapshot disabled
103. Performance etcd3 /ZooKeeper snapshot disabled
104. Memory 10GB 2.4GB 0.8GB 512MB data - 2M 256B keys
105. Sounds good, but... Is anyone successful with all this in prod?
106. Publically traded options exchange
107. Containers on CoreOS are powering ISE's high- throughput, low-latency financial exchange Running in production Bare metal & AWS Billions of transactions a day 150 million req/sec
109. Secure the Internet MISSION
110. Separate Apps from OS STRATEGY
111. Make Servers Consistent STRATEGY
112. Tolerate Machine Failures STRATEGY
113. Make Servers Easy to Upgrade STRATEGY
114. Simplify Application Upgrades STRATEGY
116. Thank you! Brandon Philips @brandonphilips | brandon@coreos.com | coreos.com We’re hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers