Heartbeats and Healthchecks

In this presentation, I explain what patterns to keep in mind to build fault-tolerant edge computing devices.

This version of the talk was given at ConFoo Montreal in February 2024.

---

Companion Code: github.com/workloads/baedge

Transcript

1. Heartbeats & Healthchecks (at the Edge of Human and Compute)
2. Sr. Developer Advocate at HashiCorp for Infrastructure and Orchestration @ksatirli Kerim Satirli
3. computing that takes place at or near the physical location of the producer or consumer of data. Similar: noun point of presence mobile datacenter edge com·put·ing
4. ConFoo Attendee Kerim Satirli plastic frame low-powered screen conference branding
5. ConFoo Attendee Kerim Satirli back front
6. 01 Noisy Neighbors Don't expect connectivity at the edge.
7. ! reception okay'ish frequent disconnects
8. Noise on the Net Spectrum Scan (2.4 GHz) 08:24 08:48 09:12 09:36 10:00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Channels
9. Challenge: no connectivity
10. Suggested Solutions store data with device- specific IDs to prevent reconciling conflicts retry and back-off with sensible intervals, don't just increase the delay implement code that allows a control server to push config changes unique IDs back-off push config implement code that runs in offline / delayed connectivity situations local-first
11. job "baedge" { group "server" { disconnect { ttl = "4h" stop_after = "2h" replace = true reconcile = "keep_original" } } } baedge.nomad.hcl Offline "Easy-Mode"
12. 02 Sims aren't real Don't expect mocks to tell the full story.
13. digital twin real device
14. Fake isn't Real.
15. Suggested Solutions get feedback from people not involved in the building process subject to physical stress to understand operational impact buy devices from multiple vendors to account for revisions don't train throw 'em test broadly deploy devices in real environments often, avoid mocked stages field early
16. 03 Build for Breaches Expect leaks and compromises.
17. Control goes down, and Risk goes up.
18. bbc.com/news/technology-48743043 Device Security
19. Challenge: remain in control
20. Suggested Solutions limit access credentials on devices to absolute bare minimum automatically rotate secrets, and expire rotated secrets audit access logs early and often, use data to make informed choices limit rotate audit physically seal and disconnect ports you don't actively use. seal
21. job "baedge" { group "server" { identity { name = "baedge" aud = [ "oidc.baedge.local", ] file = true ttl = "30m" } } } baedge.nomad.hcl Rotating Credentials
22. This talk wasn't about conference badges.
23. This talk was about fault tolerance.
24. "Nomad" screen "Baedge" screen ConFoo Attendee Allocation: 1a2b3c4d Address: 192.168.0.23 Version: 1.7.5 Nomad Runtime ConFoo Attendee Model: 2in7b Revision: v2 {Ba,E}dge Hardware
25. Demo Code
26. Thank you speakerdeck.com/ksatirli