Upgrade to Pro — share decks privately, control downloads, hide ads and more …

When testing just doesn't cut it

When testing just doesn't cut it

Writing unit tests is pretty much standard practice these days. Otherwise, how would you make sure that your code does what you expect? Yet, some software is mission-critical and merely testing a few examples – or even randomized testing – is not enough. To reach higher levels of assurance, we need proof: mathematical, formal proof. This session will be based on an example from industry, where we successfully verified the core of a financial application. I will describe the core architecture of the system and the mathematical foundations behind the verification, including the classes of problems that we can (or cannot) discover with this approach.

Lars Hupel

March 17, 2023
Tweet

More Decks by Lars Hupel

Other Decks in Programming

Transcript

  1. When testing just
    doesn't cut it
    Lars Hupel
    BOB Konferenz
    2023-03-17

    View Slide

  2. Where would this line be used?
    int mid = (low + high) / 2

    View Slide

  3. … and what’s wrong with it?
    int mid = (low + high) / 2

    View Slide

  4. 4

    View Slide

  5. View Slide

  6. Sorting in Java
    6
    list.sort((x, y) ->
    x.beard.compareTo(y.beard)
    )

    View Slide

  7. 7
    CAV 2015

    View Slide

  8. Programming & Bugs
    8

    View Slide

  9. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation

    View Slide

  10. Bugs: We don’t like them
    Yet, they keep cropping up …
    10

    View Slide

  11. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation
    “Debugging”

    View Slide

  12. Debugging is a core skill
    12

    View Slide

  13. 13
    OSDI 2014

    View Slide

  14. 14
    Empirical Software Engineering 2015

    View Slide

  15. 15

    View Slide

  16. “Program testing can be a very effective way to show
    the presence of bugs, but it is hopelessly inadequate
    for showing their absence”

    View Slide

  17. Formal Methods
    17

    View Slide

  18. “Formal Methods refers to mathematically rigorous
    techniques and tools for the specification, design and
    verification of software and hardware systems”
    18

    View Slide

  19. Specification
    What are Formal Methods?
    Coverage
    Rigor
    Implementation
    Type system
    First-order logic Model checking
    State machines
    Theorem prover
    Property testing
    Flowchart

    View Slide

  20. ISO 5807 Flowchart
    20

    View Slide

  21. ISO 5807:1985
    21
    Syntax
    Semantics

    View Slide

  22. What is verification?
    Specification Implementation
    Proof

    View Slide

  23. What is verification?
    Abstract
    specification
    Implementation
    Proof
    Executable
    specification
    Proof

    View Slide

  24. Formal Methods in practice
    24

    View Slide

  25. 25

    View Slide

  26. Central Bank Digital Currency
    26
    CBDC
    Banknotes Bank deposits
    and e-money
    Issued by the
    central bank
    Digital money

    View Slide

  27. Our customers
    ● central banks
    ● commercial/retail banks
    ● payment service providers
    27

    View Slide

  28. 28

    View Slide

  29. How money is represented in G+D Filia®
    29

    View Slide

  30. 30

    View Slide

  31. 31

    View Slide

  32. 32

    View Slide

  33. 33

    View Slide

  34. Isabelle to the rescue!
    34

    View Slide

  35. “Isabelle/HOL = Functional Programming + Logic”

    View Slide

  36. G+D Filia® in Isabelle/HOL
    ● mathematical model of “coins” and their evolution
    ● graph-theoretic considerations
    ● high-level correctness properties
    ● reference implementation (executable in Scala)

    View Slide

  37. Example: Money in circulation
    definition graph_balance :: nat where
    ‹graph_balance = (∑N ∈ unspent. value N)›
    lemma graph_balance_alt_def:
    ‹graph_balance = ¦(∑c ∈ graph. value_difference c)¦›
    37

    View Slide

  38. It’s not just us
    38

    View Slide

  39. Proof-Driven Development (PDD)
    39

    View Slide

  40. 40

    View Slide

  41. 41

    View Slide

  42. 42

    View Slide

  43. Designing a new feature
    ● Can the feature work correctly?
    ● Are there any undesirable feature interactions?
    ● How can we implement the feature?
    43

    View Slide

  44. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation
    “PDD”

    View Slide

  45. PDD works for us
    ● we found some flaws in our initial design of a feature
    ● … including a feature interaction bug
    ● after iterative improvement, the feature is now better than an alternative design
    ● changed the internal (simpler) data model, but we established a mapping
    ● feature has been shipped to production
    45

    View Slide

  46. Roadmap
    46

    View Slide

  47. 47

    View Slide

  48. There’s always more to do …
    ● expanding the scope of our formalization
    ● adding model checking to our toolbox
    ● closing the gap between executable specification and implementation
    48

    View Slide

  49. Closing the gap
    Abstract
    specification
    Implementation
    Proof
    Executable
    specification
    Proof

    View Slide

  50. Questions?
    Answers!
    Lars Hupel
    https://lars.hupel.info
    [email protected]

    View Slide

  51. Image sources
    ● Edsger W. Dijskstra: Hamilton Richards, CC-BY-SA 3.0,
    https://commons.wikimedia.org/w/index.php?title=File:Edsger_Wybe_Dijkstra.jpg&oldid=710250
    942
    ● César A. Muñoz: https://shemesh.larc.nasa.gov/people/cam/
    ● BPMN: Mikelo Skarabo, CC-BY-SA 4.0,
    https://commons.wikimedia.org/w/index.php?title=File:BPMN-
    AProcessWithNormalFlow.svg&oldid=734511959

    View Slide