When testing just doesn't cut it

Writing unit tests is pretty much standard practice these days. Otherwise, how would you make sure that your code does what you expect? Yet, some software is mission-critical and merely testing a few examples – or even randomized testing – is not enough. To reach higher levels of assurance, we need proof: mathematical, formal proof. This session will be based on an example from industry, where we successfully verified the core of a financial application. I will describe the core architecture of the system and the mathematical foundations behind the verification, including the classes of problems that we can (or cannot) discover with this approach.

Transcript

1. When testing just doesn't cut it Lars Hupel BOB Konferenz 2023-03-17
2. Where would this line be used? int mid = (low + high) / 2
3. … and what’s wrong with it? int mid = (low + high) / 2
6. Sorting in Java 6 list.sort((x, y) -> x.beard.compareTo(y.beard) )
7. 7 CAV 2015
8. Programming & Bugs 8
9. Requirements Design/Architecture Implementation Testing Operation
10. Bugs: We don’t like them Yet, they keep cropping up … 10
11. Requirements Design/Architecture Implementation Testing Operation “Debugging”
12. Debugging is a core skill 12
13. 13 OSDI 2014
14. 14 Empirical Software Engineering 2015
16. “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence”
17. Formal Methods 17
18. “Formal Methods refers to mathematically rigorous techniques and tools for the specification, design and verification of software and hardware systems” 18
19. Specification What are Formal Methods? Coverage Rigor Implementation Type system First-order logic Model checking State machines Theorem prover Property testing Flowchart
20. ISO 5807 Flowchart 20
21. ISO 5807:1985 21 Syntax Semantics
22. What is verification? Specification Implementation Proof
23. What is verification? Abstract specification Implementation Proof Executable specification Proof
24. Formal Methods in practice 24
26. Central Bank Digital Currency 26 CBDC Banknotes Bank deposits and e-money Issued by the central bank Digital money
27. Our customers • central banks • commercial/retail banks • payment service providers 27
29. How money is represented in G+D Filia® 29
34. Isabelle to the rescue! 34
35. “Isabelle/HOL = Functional Programming + Logic”
36. G+D Filia® in Isabelle/HOL • mathematical model of “coins” and their evolution • graph-theoretic considerations • high-level correctness properties • reference implementation (executable in Scala)
37. Example: Money in circulation definition graph_balance :: nat where ‹graph_balance = (∑N ∈ unspent. value N)› lemma graph_balance_alt_def: ‹graph_balance = ¦(∑c ∈ graph. value_difference c)¦› 37
38. It’s not just us 38
39. Proof-Driven Development (PDD) 39
43. Designing a new feature • Can the feature work correctly? • Are there any undesirable feature interactions? • How can we implement the feature? 43
44. Requirements Design/Architecture Implementation Testing Operation “PDD”
45. PDD works for us • we found some flaws in our initial design of a feature • … including a feature interaction bug • after iterative improvement, the feature is now better than an alternative design • changed the internal (simpler) data model, but we established a mapping • feature has been shipped to production 45
46. Roadmap 46
48. There’s always more to do … • expanding the scope of our formalization • adding model checking to our toolbox • closing the gap between executable specification and implementation 48
49. Closing the gap Abstract specification Implementation Proof Executable specification Proof
50. Questions? Answers! Lars Hupel https://lars.hupel.info lars.hupel@gi-de.com
51. Image sources • Edsger W. Dijskstra: Hamilton Richards, CC-BY-SA 3.0, https://commons.wikimedia.org/w/index.php?title=File:Edsger_Wybe_Dijkstra.jpg&oldid=710250 942 • César A. Muñoz: https://shemesh.larc.nasa.gov/people/cam/ • BPMN: Mikelo Skarabo, CC-BY-SA 4.0, https://commons.wikimedia.org/w/index.php?title=File:BPMN- AProcessWithNormalFlow.svg&oldid=734511959