Heartbeats & Healthchecks (at the Edge of Human and Compute)

Sr. Developer Advocate at HashiCorp for Infrastructure and Orchestration @ksatirli Kerim Satirli

computing that takes place at or near the physical location of the producer or consumer of data. Similar: noun point of presence mobile datacenter edge com·put·ing

ConFoo Attendee Kerim Satirli plastic frame low-powered screen conference branding

ConFoo Attendee Kerim Satirli back front

01 Noisy Neighbors Don't expect connectivity at the edge.

! reception okay'ish frequent disconnects

Noise on the Net Spectrum Scan (2.4 GHz) 08:24 08:48 09:12 09:36 10:00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Channels

Challenge: no connectivity

Suggested Solutions store data with device- specific IDs to prevent reconciling conflicts retry and back-off with sensible intervals, don't just increase the delay implement code that allows a control server to push config changes unique IDs back-off push config implement code that runs in offline / delayed connectivity situations local-first

job "baedge" { group "server" { disconnect { ttl = "4h" stop_after = "2h" replace = true reconcile = "keep_original" } } } baedge.nomad.hcl Offline "Easy-Mode"

02 Sims aren't real Don't expect mocks to tell the full story.

digital twin real device

Fake isn't Real.

Suggested Solutions get feedback from people not involved in the building process subject to physical stress to understand operational impact buy devices from multiple vendors to account for revisions don't train throw 'em test broadly deploy devices in real environments often, avoid mocked stages field early

03 Build for Breaches Expect leaks and compromises.

Control goes down, and Risk goes up.

bbc.com/news/technology-48743043 Device Security

Challenge: remain in control

Suggested Solutions limit access credentials on devices to absolute bare minimum automatically rotate secrets, and expire rotated secrets audit access logs early and often, use data to make informed choices limit rotate audit physically seal and disconnect ports you don't actively use. seal

job "baedge" { group "server" { identity { name = "baedge" aud = [ "oidc.baedge.local", ] file = true ttl = "30m" } } } baedge.nomad.hcl Rotating Credentials

This talk wasn't about conference badges.

This talk was about fault tolerance.

"Nomad" screen "Baedge" screen ConFoo Attendee Allocation: 1a2b3c4d Address: 192.168.0.23 Version: 1.7.5 Nomad Runtime ConFoo Attendee Model: 2in7b Revision: v2 {Ba,E}dge Hardware

Demo Code

Thank you speakerdeck.com/ksatirli