When testing just doesn't cut it

Writing unit tests is pretty much standard practice these days. Otherwise, how would you make sure that your code does what you expect? Yet, some software is mission-critical and merely testing a few examples – or even randomized testing – is not enough. To reach higher levels of assurance, we need proof: mathematical, formal proof. This session will be based on an example from industry, where we successfully verified the core of a financial application. I will describe the core architecture of the system and the mathematical foundations behind the verification, including the classes of problems that we can (or cannot) discover with this approach.

Transcript

1. When testing just
   doesn't cut it
   Lars Hupel
   BOB Konferenz
   2023-03-17
   

   View Slide

2. Where would this line be used?
   int mid = (low + high) / 2
   

   View Slide

3. … and what’s wrong with it?
   int mid = (low + high) / 2
   

   View Slide

4. View Slide

5. View Slide

6. Sorting in Java
   6
   list.sort((x, y) ->
   x.beard.compareTo(y.beard)
   )
   

   View Slide

7. 7
   CAV 2015
   

   View Slide

8. Programming & Bugs
   8
   

   View Slide

9. Requirements
   Design/Architecture
   Implementation
   Testing
   Operation
   

   View Slide

10. Bugs: We don’t like them
    Yet, they keep cropping up …
    10
    

    View Slide

11. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation
    “Debugging”
    

    View Slide

12. Debugging is a core skill
    12
    

    View Slide

13. 13
    OSDI 2014
    

    View Slide

14. 14
    Empirical Software Engineering 2015
    

    View Slide

15. View Slide

16. “Program testing can be a very effective way to show
    the presence of bugs, but it is hopelessly inadequate
    for showing their absence”
    

    View Slide

17. Formal Methods
    17
    

    View Slide

18. “Formal Methods refers to mathematically rigorous
    techniques and tools for the specification, design and
    verification of software and hardware systems”
    18
    

    View Slide

19. Specification
    What are Formal Methods?
    Coverage
    Rigor
    Implementation
    Type system
    First-order logic Model checking
    State machines
    Theorem prover
    Property testing
    Flowchart
    

    View Slide

20. ISO 5807 Flowchart
    20
    

    View Slide

21. ISO 5807:1985
    21
    Syntax
    Semantics
    

    View Slide

22. What is verification?
    Specification Implementation
    Proof
    

    View Slide

23. What is verification?
    Abstract
    specification
    Implementation
    Proof
    Executable
    specification
    Proof
    

    View Slide

24. Formal Methods in practice
    24
    

    View Slide

25. View Slide

26. Central Bank Digital Currency
    26
    CBDC
    Banknotes Bank deposits
    and e-money
    Issued by the
    central bank
    Digital money
    

    View Slide

27. Our customers
    ● central banks
    ● commercial/retail banks
    ● payment service providers
    27
    

    View Slide

28. View Slide

29. How money is represented in G+D Filia®
    29
    

    View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. Isabelle to the rescue!
    34
    

    View Slide

35. “Isabelle/HOL = Functional Programming + Logic”
    

    View Slide

36. G+D Filia® in Isabelle/HOL
    ● mathematical model of “coins” and their evolution
    ● graph-theoretic considerations
    ● high-level correctness properties
    ● reference implementation (executable in Scala)
    

    View Slide

37. Example: Money in circulation
    definition graph_balance :: nat where
    ‹graph_balance = (∑N ∈ unspent. value N)›
    lemma graph_balance_alt_def:
    ‹graph_balance = ¦(∑c ∈ graph. value_difference c)¦›
    37
    

    View Slide

38. It’s not just us
    38
    

    View Slide

39. Proof-Driven Development (PDD)
    39
    

    View Slide

40. View Slide

41. View Slide

42. View Slide

43. Designing a new feature
    ● Can the feature work correctly?
    ● Are there any undesirable feature interactions?
    ● How can we implement the feature?
    43
    

    View Slide

44. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation
    “PDD”
    

    View Slide

45. PDD works for us
    ● we found some flaws in our initial design of a feature
    ● … including a feature interaction bug
    ● after iterative improvement, the feature is now better than an alternative design
    ● changed the internal (simpler) data model, but we established a mapping
    ● feature has been shipped to production
    45
    

    View Slide

46. Roadmap
    46
    

    View Slide

47. View Slide

48. There’s always more to do …
    ● expanding the scope of our formalization
    ● adding model checking to our toolbox
    ● closing the gap between executable specification and implementation
    48
    

    View Slide

49. Closing the gap
    Abstract
    specification
    Implementation
    Proof
    Executable
    specification
    Proof
    

    View Slide

50. Questions?
    Answers!
    Lars Hupel
    https://lars.hupel.info
    lars.hupel@gi-de.com
    

    View Slide

51. Image sources
    ● Edsger W. Dijskstra: Hamilton Richards, CC-BY-SA 3.0,
    https://commons.wikimedia.org/w/index.php?title=File:Edsger_Wybe_Dijkstra.jpg&oldid=710250
    942
    ● César A. Muñoz: https://shemesh.larc.nasa.gov/people/cam/
    ● BPMN: Mikelo Skarabo, CC-BY-SA 4.0,
    https://commons.wikimedia.org/w/index.php?title=File:BPMN-
    AProcessWithNormalFlow.svg&oldid=734511959
    

    View Slide