Stateful systems in the time of orchestrators

Transcript

1. Orchestrating Persistence Danielle Lancashire she/her @endocrimes @endocrimes 1
2. $ whoami @endocrimes 2
3. Lets talk about containers @endocrimes 3
4. they are ephemeral @endocrimes 4
5. but I want my state and to burn it too ! 5
6. if you liked it... you should've put a mount on it? @endocrimes 6
7. Persistent State + Ephemeral Runtime @endocrimes 7
8. State is hard @endocrimes 8
9. It is possible! @endocrimes 9
10. It's still early @endocrimes 10
11. This is a talk about failure & reliability @endocrimes 11
12. @endocrimes 12
13. Cluster Orchestrators? • Nomad • Kubernetes • Mesos • CloudFoundry @endocrimes 13
14. What do orchestrators do? • Map work to compute resources • Coordinate deployments • Provide observability into workloads @endocrimes 14
15. What is Nomad? • Cluster Orchestrator • Pluggable drivers (containers/vms/plain binaries) • Single Binary • Scalable (10k+ node deployments, with 5k+ scheduling decisions/second) @endocrimes 15
16. @endocrimes 16
17. CAP Theorem @endocrimes 17
18. @endocrimes 18
19. AP Clients @endocrimes 19
20. Container Storage Interface (CSI) @endocrimes 20
21. A Common API @endocrimes 21
22. Block Devices & File Storage @endocrimes 22
23. These plugins are external processes @endocrimes 23
24. Plugin Services • Identity • Controller • Node @endocrimes 24
25. Controller Service interacts with the control plane @endocrimes 25
26. from any node @endocrimes 26
27. to Provision, Attach, and Delete Volumes @endocrimes 27
28. and need tokens for the storage provider @endocrimes 28
29. Node Service performs local managegment @endocrimes 29
30. handles formatting, mounting, and local volume actions @endocrimes 30
31. must run on every node that can use volumes @endocrimes 31
32. and need special privileges @endocrimes 32
33. Architecture @endocrimes 33
34. @endocrimes 34
35. @endocrimes 35
36. @endocrimes 36
37. Lifecycle of a GCE Persistent Disk @endocrimes 37
38. @endocrimes 38
39. ControllerPublishVolume -> NodeStageVolume -> NodePublishVolume @endocrimes 39
40. @endocrimes 40
41. ControllerPublishVolume -> NodeStageVolume -> NodePublishVolume @endocrimes 41
42. ControllerPublishVolume -> NodeStageVolume -> NodePublishVolume @endocrimes 42
43. CSI doesn't solve everything @endocrimes 43
44. CSI is (currently) lowest common denominator @endocrimes 44
45. Topology @endocrimes 45
46. API Correctness @endocrimes 46
47. Failure Recovery is Hard @endocrimes 47
48. What does that mean? @endocrimes 48
49. Modes of failure • Human/Application • Node • Control Plane • Storage Plugin @endocrimes 49
50. 1. Application failure @endocrimes 50
51. @endocrimes
52. Rescheduling @endocrimes 52
53. Adding storage into the mix @endocrimes 53
54. 3. Node Failure @endocrimes 54
55. @endocrimes 55
56. Duplicate Workloads @endocrimes 56
57. Exclusive Access @endocrimes 57
58. Safe Shutdown @endocrimes 58
59. like still providing write paths to data after a volume has been detached. @endocrimes 59
60. Shut down tasks when losing connection? @endocrimes 60
61. 3. Control Plane Failure @endocrimes 61
62. Conflicting Goals @endocrimes 62
63. Deferring to the operator @endocrimes 63
64. Running at the edge @endocrimes 64
65. Running in a data centre @endocrimes 65
66. Kubernetes StatefulSets @endocrimes 66
67. 4. Plugin Failure @endocrimes 67
68. Controller Plugins @endocrimes 68
69. An operation that has no additional effects if invoked multiple times with the same inputs. — Idempotent (adj) @endocrimes 69
70. Node Plugins @endocrimes 70
71. Crashing Plugins @endocrimes 71
72. Most apps weren't built for containers @endocrimes 72
73. We have to accommodate them @endocrimes 73
74. Operators @endocrimes 74
75. Extending the scheduler @endocrimes 75
76. Nothing quite gives you "run your own RDS" yet @endocrimes 76
77. But I want RDS! @endocrimes 77
78. Use RDS. @endocrimes 78
79. The Future @endocrimes 79
80. Thanks! ! twitter.com/endocrimes ✉ dani@builds.terrible.systems ! Recommended Talk: • Preetha Appan: Who guards the guardians? Designing for resilience in cluster orchestrators (Velocity NYC, 2018) @endocrimes 80