Formal Specifications and Other People's Tech

How do you use formal specification like TLA+ to model real world engineering challenges?

Transcript

1. auth0.com Formal Specification Taming Other People's Tech &
2. SPECS! BY BEGINNERS FOR BEGINNERS @bellmar bit.ly/bellmar-auth0
3. TLA+ Conf Finding Bugs Without Running or Even Looking at Code Correctness Proofs of Distributed Systems with Isabelle
4. WHAT IS FORMAL SPECIFICATION? @bellmar bit.ly/bellmar-auth0
5. -Inputs -Expected behavior -Legal outputs -Illegal behavior
6. Tell me the future, wise computer @bellmar bit.ly/bellmar-auth0
7. @bellmar bit.ly/bellmar-auth0
8. !! @bellmar bit.ly/bellmar-auth0
9. @bellmar bit.ly/bellmar-auth0 Mostly I write bad specs, and this talk is about how that still adds a lot of value The more people write specs the faster we all learn
10. S3 Design Session @bellmar bit.ly/bellmar-auth0
11. MOST ENGINEERS WILL NEVER BUILD A MESSAGE QUEUE You’ll buy one from AWS
12. BUT SPECIFICATION CAN BE APPLIED TO MANY DIFFERENT THINGS! @bellmar bit.ly/bellmar-auth0
13. @bellmar bit.ly/bellmar-auth0 Identity as a Service
14. @bellmar bit.ly/bellmar-auth0
15. Built by other teams
16. In taking over a service: - Defining SLOs - Evaluating debt - Studying existing problems and rethinking architecture
17. EMAILS - validate email - password reset - principal mode of communication We do logins not emails!
18. EMAILS - validate email - password reset - principal mode of communication @bellmar bit.ly/bellmar-auth0
19. X per hour @bellmar bit.ly/bellmar-auth0
20. @bellmar bit.ly/bellmar-auth0
21. @bellmar bit.ly/bellmar-auth0
22. Email all my users! Password Reset Anomaly Detection Welcome Email Validation @bellmar bit.ly/bellmar-auth0
23. @bellmar bit.ly/bellmar-auth0
24. HOW DO WE KNOW IF THIS CHANGE IS SAFE? @bellmar bit.ly/bellmar-auth0
25. LET’S WRITE A SPEC! @bellmar bit.ly/bellmar-auth0
26. STEPS WITH SPECIFICATIONS YOU THINK IN STATES WITH ALGORITHMS YOU THINK IN @bellmar bit.ly/bellmar-auth0
27. @bellmar bit.ly/bellmar-auth0
28. @bellmar bit.ly/bellmar-auth0
29. \ 40% → ✅ 41% → ✅ 42% → ✅ 43% → ✅ 44% → ✅ 45% → ✅ 46% → ✅ 47% → ✅ 48% → ✅ 49% → ✅ 50% → ✅
30. VM 50% VM 20% VM 70%
31. VM 50% VM 20% VM 70% Test all possible states to make sure the load balancer does the right thing
32. 40% → ✅ 41% → ✅ 42% → ✅ 43% → ✅ 44% → ✅ 45% → ✅ 46% → ✅ 47% → ✅ 48% → ✅ 49% → ✅ 50% → ✅ 40% → ✅ 50% → ✅ 60% → ✅ 70% → ❌ ✅ ❌
33. Normal Idle Unhealthy @bellmar bit.ly/bellmar-auth0
34. Must always be true: 0 < servers > 10 Goto! Party like it’s 1979…
35. LEAVE YOUR PROBABILITY AT HOME
36. Sometimes we kill a server, sometimes we create one Literally the one time we just keep killing servers @bellmar bit.ly/bellmar-auth0
37. Sometimes we kill a server, sometimes we create one Literally the one time we just keep killing servers ARGH! BUT WHY? THAT WILL NEVER HAPPEN!!
38. SYSTEM NOT SYSTEM LGTM SAFETY @bellmar bit.ly/bellmar-auth0
39. Normal Idle Unhealthy Health Check @bellmar bit.ly/bellmar-auth0
40. - Several unhealthy servers == Create new - All servers healthy and more than 3 == Shut one down - Else assume we kill and refresh unhealthy
41. - Several unhealthy servers == Create new - All servers healthy and more than 3 == Shut one down - Else assume we kill and refresh unhealthy
42. WHAT STATES SHOULD BE IMPOSSIBLE? (NOT UNDESIRABLE?) @bellmar bit.ly/bellmar-auth0
43. Servers keep restarting Too many servers Too few servers Healthcheck action inconsistency @bellmar bit.ly/bellmar-auth0
44. Servers keep restarting Too many servers Too few servers Healthcheck action inconsistency Undesirable but not impossible states @bellmar bit.ly/bellmar-auth0
45. Maybe you should turn yourself rather than the puzzle
46. Anyway… @bellmar bit.ly/bellmar-auth0
47. x per hour @bellmar bit.ly/bellmar-auth0
48. One please! @bellmar bit.ly/bellmar-auth0
49. One please! Sorry no more tokens. Sit in the retry queue and come back later @bellmar bit.ly/bellmar-auth0
50. One please! Sorry no more tokens. Sit in the retry queue and come back later @bellmar bit.ly/bellmar-auth0
51. STATE HOW DO WE MAKE THIS LOOK LIKE A MACHINE @bellmar bit.ly/bellmar-auth0
52. Queued → Worker → Sent Has Tokens → No Tokens @bellmar bit.ly/bellmar-auth0
53. Queued Worker Sent @bellmar bit.ly/bellmar-auth0
54. Queued Worker Sent Dropped @bellmar bit.ly/bellmar-auth0
55. WHAT’S STATES SHOULD BE IMPOSSIBLE? @bellmar bit.ly/bellmar-auth0
56. Has Token → Deleted Sent → Deleted Deleted → Sent No Token → Sent @bellmar bit.ly/bellmar-auth0
57. HOW DOES OUR IMPLEMENTATION PREVENT THESE STATES? @bellmar bit.ly/bellmar-auth0
58. JUST TRYING TO BUILD THE MODEL OFTEN REVEALS BUGS @bellmar bit.ly/bellmar-auth0
59. Email all my users! Password Reset Anomaly Detection Welcome Email Validation @bellmar bit.ly/bellmar-auth0
60. Email all my users! Password Reset Anomaly Detection Welcome Email Validation @bellmar bit.ly/bellmar-auth0
61. Queued Worker @bellmar bit.ly/bellmar-auth0
62. GREAT, SO WHAT SHOULD OUR MODEL LOOK LIKE? @bellmar bit.ly/bellmar-auth0
63. Tenant Tenant Tenant @bellmar bit.ly/bellmar-auth0
64. Tenant @bellmar bit.ly/bellmar-auth0 For each tenant For each bucket
65. LOOPS WILL F— YOU UP LOOPS @bellmar bit.ly/bellmar-auth0
66. x in [a, b, c] print x “a” “b” “c” @bellmar bit.ly/bellmar-auth0
67. x in [a, b, c] l=append(l, x) l = [ ] print l @bellmar bit.ly/bellmar-auth0
68. x in [a, b, c] l=append(l, x) [“a”] [“a”, “b”] [“a”, “b”, “c”] l = [ ] print l @bellmar bit.ly/bellmar-auth0
69. x in [a, b, c] l = [ ] [“a”] [“b”] [“c”] l=append(l, x) print l @bellmar bit.ly/bellmar-auth0
70. A B C then then A B C or or Steps States @bellmar bit.ly/bellmar-auth0
71. Tenant Tenant Tenant @bellmar bit.ly/bellmar-auth0
72. Tenant Tenant Tenant @bellmar bit.ly/bellmar-auth0
73. @bellmar bit.ly/bellmar-auth0
74. Alice Bob Oh… that’s easy!
75. Undesirable -vs- Impossible
76. Queue Publishers
77. “SHOULD THE PUBLISHERS BE PUTTING THINGS DIRECTLY IN THE QUEUE?” @bellmar bit.ly/bellmar-auth0 WAIT…
78. 1) Make it look like a state machine 2) Define the impossible states 3) Model what makes them impossible To Summarize: @bellmar bit.ly/bellmar-auth0
79. Things that already look like state machines: @bellmar bit.ly/bellmar-auth0
80. Things that already look like state machines: - Redux @bellmar bit.ly/bellmar-auth0
81. Things that already look like state machines: - Redux - AWS Lambda @bellmar bit.ly/bellmar-auth0
82. Things that already look like state machines: - Redux - AWS Lambda - Workflow Orchestration (Airflow, Conductor, Luigi) @bellmar bit.ly/bellmar-auth0
83. Bit.ly/bellmar-auth0