CD for infrastructure

If you're doing devops, you're almost certainly doing CD. Developers get their changes in front of customers quicker, operations have consistency, repeatability, and auditability, and the business gets measurable value, faster.

But most of the literature and case studies on CD are around applications. What about infrastructure? The evidence clearly shows that CD of applications reduces failures and increases reliability, but operations teams generally don't manage infrastructure services like PaaS or monitoring with CD - and they're missing out on the same benefits.

For the last 6 months, we have been building our next generation monitoring metrics storage platform to handle over 350,000 metrics updated every 10 seconds, with at least 6 months retention.

Because monitoring infrastructure needs to be at least as reliable (if not more reliable) than the things being monitored, we opted to design this infrastructure service to be Continuous Deployed. By designing CD-first, we have been able to adapt to changing business and engineering requirements, and scale the infrastructure faster and more reliably than would have otherwise been possible.

In this talk we'll cover a toolchain for Continuously Deploying changes to infrastructure services, compare and contrast the CD challenges for applications vs infrastructure, and explore how to get good feedback on changes to infrastructure that compliments and assists monitoring.

Transcript

1. Continuous Deployment of infrastructure services Lindsay Holmwood
2. How to CD successfully 1. Optimise for fast feedback 2. Chunk your changes
3. Background
4. CD vs CD
5. deploy to production acceptance tests integrate unit tests code done Continuous Deployment Auto Auto Auto Auto deploy to production acceptance tests integrate unit tests code done Continuous Delivery Manual Auto Auto Auto
6. What is an infrastructure service?
7. • CI & • DNS & • Metrics storage & • Database & • PaaS
8. What is Continuously Deployed infrastructure?
9. code change CI running service
10. The pipeline
11. test deploy build commit test
12. Executing the build: make build || ./cibuild.sh make deploy || ./cideploy.sh
13. Push build definition into repo
14. API
15. API
16. The service
17. Decide what guarantees you are providing
18. • Consistency • Availability • Partition tolerance • Pick 2
19. Define your SLAs
20. Throughput
21. Availability
22. Data consistency
23. Examples:
24. 95e response time for monitoring metric queries in a one hour window is < 1 second.
25. No gaps in metric data returned from queries.
26. A single storage node failure does not result 
 in unavailability.
27. Codify your SLAs as tests and checks
28. Know your data
29. How does data flow?
30. Eliminate the state
31. Define interfaces
32. Examples:
33. Requests to the API should conform to JSONAPI spec.
34. Metrics are received on UDP 25826 in collectd network protocol format.
35. How to CD successfully 1. Optimise for fast feedback 2. Chunk your changes
36. Making it fast
37. Fast feedback 1. Validate quickly 2. Limit technical debt 3. Make it work, make it fast, make it right
38. Constantly identify & eliminate bottlenecks
39. Get iteration time down
40. test deploy build commit Measure individual duration test Measure total duration
41. < 5 minutes
42. Track cycle time (min, max, median, 95e)
43. time(1)
44. Get CI close to the action
45. Eliminate latency
46. API
47. API
48. 1. Does the thing exist? 2. Maybe make a change 3. Get info about the thing
49. API These steps are mandatory
50. Push all changes through CI
51. 1 workflow
52. API
53. Divergence & Confidence
54. This will be painful! 1. (But worth it for audit trail & reduced latency)
55. jenkins-cli from Netflix Skunkworks
56. git push && jenkins start && jenkins tail
58. One-off jobs
59. • Data fixups • Software patches • Benchmarks
60. Add one-off jobs to build or deploy steps
61. test deploy build commit test Inject one-off jobs here
62. Push build definition into repo
63. Chunking the changes
64. Chunk your changes 1. Change one, test one 2. Limit WIP
65. Change 1, Test 1
66. Ordering matters
67. 1. Change app 2.Change DB 3.Change proxy 4.Test app 5.Test DB 6.Test proxy 4.Test app 5.Test DB 6.Test proxy
68. 1. Change app 2.Test app 3.Change DB 4.Test DB 5.Change proxy 6.Test proxy 1. Change app 2.Test app 3.Change DB 4.Test DB 5.Change proxy 6.Test proxy fail early
69. Tests finish quickly
70. < 10 seconds
71. 1. Service running? 2. Can I do a simple query? 3. Obviously bad log messages? 4. Significant statistical deviation in metrics?
72. twitter/BreakoutDetection 1. Goodness of fit tests: 2. Kolmogorov-Smirnov 3. Kuiper’s 4. Anderson-Darling
74. Make feedback visual
76. 1. checkout 2. build 3. test 4. notify Continuous Integration Monitoring
77. Level up: Run tests constantly
79. { "service": { "name": "collectd", "tags": [], "port": 25826, "checks": [ … ] } }
80. { "id": "coco-expvars", "name": "Coco expvars at :9090", "http": "http://127.0.0.1:9090/debug/vars", "interval": "10s", "timeout": "1s" }, { "id": "lookup", "name": "Coco hash lookup at :9090", "http": "http://127.0.0.1:9090/lookup?name=hello", "interval": "10s", "timeout": "1s" }, { "id": "anomalous_coco_errors", "name": "anomalous_coco_errors", "script": "anomalous_coco_errors --host coco.example --window 10m", "interval": "10s", "timeout": "5s" }
81. 1. Change app 2.Test app 3.Change DB 4.Test DB 5.Change proxy 6.Test proxy
83. CD of apps vs CD of infrastructure
84. • Faster builds? • Apps: More compute " • Infrastructure: Change IaaS
85. • Testing? • Apps: xUnit • Infrastructure: Serverspec & DIY
86. • Latency? • Apps: More compute, SSDs, memory. $ • Infrastructure: Network proximity.
87. • Lingering state? • Apps: Database transactions. Wipe database contents. • Infrastructure: Enjoy your complete rebuild.
89. • How to CD successfully • Optimise for fast feedback • Chunk your changes • Constantly eliminate bottlenecks • Get iteration time down
90. I’m Lindsay @auxesis
91. Thank you! Questions? '()*+ the talk? Let @auxesis know!