Upgrade to Pro — share decks privately, control downloads, hide ads and more …

When testing just doesn't cut it

When testing just doesn't cut it

Writing unit tests is pretty much standard practice these days. Otherwise, how would you make sure that your code does what you expect? Yet, some software is mission-critical and merely testing a few examples – or even randomized testing – is not enough. To reach higher levels of assurance, we need proof: mathematical, formal proof. This session will be based on an example from industry, where we successfully verified the core of a financial application. I will describe the core architecture of the system and the mathematical foundations behind the verification, including the classes of problems that we can (or cannot) discover with this approach.

Lars Hupel

March 17, 2023
Tweet

More Decks by Lars Hupel

Other Decks in Programming

Transcript

  1. When testing just
    doesn't cut it
    Lars Hupel
    BOB Konferenz
    2023-03-17

    View full-size slide

  2. Where would this line be used?
    int mid = (low + high) / 2

    View full-size slide

  3. … and what’s wrong with it?
    int mid = (low + high) / 2

    View full-size slide

  4. Sorting in Java
    6
    list.sort((x, y) ->
    x.beard.compareTo(y.beard)
    )

    View full-size slide

  5. Programming & Bugs
    8

    View full-size slide

  6. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation

    View full-size slide

  7. Bugs: We don’t like them
    Yet, they keep cropping up …
    10

    View full-size slide

  8. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation
    “Debugging”

    View full-size slide

  9. Debugging is a core skill
    12

    View full-size slide

  10. 14
    Empirical Software Engineering 2015

    View full-size slide

  11. “Program testing can be a very effective way to show
    the presence of bugs, but it is hopelessly inadequate
    for showing their absence”

    View full-size slide

  12. Formal Methods
    17

    View full-size slide

  13. “Formal Methods refers to mathematically rigorous
    techniques and tools for the specification, design and
    verification of software and hardware systems”
    18

    View full-size slide

  14. Specification
    What are Formal Methods?
    Coverage
    Rigor
    Implementation
    Type system
    First-order logic Model checking
    State machines
    Theorem prover
    Property testing
    Flowchart

    View full-size slide

  15. ISO 5807 Flowchart
    20

    View full-size slide

  16. ISO 5807:1985
    21
    Syntax
    Semantics

    View full-size slide

  17. What is verification?
    Specification Implementation
    Proof

    View full-size slide

  18. What is verification?
    Abstract
    specification
    Implementation
    Proof
    Executable
    specification
    Proof

    View full-size slide

  19. Formal Methods in practice
    24

    View full-size slide

  20. Central Bank Digital Currency
    26
    CBDC
    Banknotes Bank deposits
    and e-money
    Issued by the
    central bank
    Digital money

    View full-size slide

  21. Our customers
    ● central banks
    ● commercial/retail banks
    ● payment service providers
    27

    View full-size slide

  22. How money is represented in G+D Filia®
    29

    View full-size slide

  23. Isabelle to the rescue!
    34

    View full-size slide

  24. “Isabelle/HOL = Functional Programming + Logic”

    View full-size slide

  25. G+D Filia® in Isabelle/HOL
    ● mathematical model of “coins” and their evolution
    ● graph-theoretic considerations
    ● high-level correctness properties
    ● reference implementation (executable in Scala)

    View full-size slide

  26. Example: Money in circulation
    definition graph_balance :: nat where
    ‹graph_balance = (∑N ∈ unspent. value N)›
    lemma graph_balance_alt_def:
    ‹graph_balance = ¦(∑c ∈ graph. value_difference c)¦›
    37

    View full-size slide

  27. It’s not just us
    38

    View full-size slide

  28. Proof-Driven Development (PDD)
    39

    View full-size slide

  29. Designing a new feature
    ● Can the feature work correctly?
    ● Are there any undesirable feature interactions?
    ● How can we implement the feature?
    43

    View full-size slide

  30. Requirements
    Design/Architecture
    Implementation
    Testing
    Operation
    “PDD”

    View full-size slide

  31. PDD works for us
    ● we found some flaws in our initial design of a feature
    ● … including a feature interaction bug
    ● after iterative improvement, the feature is now better than an alternative design
    ● changed the internal (simpler) data model, but we established a mapping
    ● feature has been shipped to production
    45

    View full-size slide

  32. There’s always more to do …
    ● expanding the scope of our formalization
    ● adding model checking to our toolbox
    ● closing the gap between executable specification and implementation
    48

    View full-size slide

  33. Closing the gap
    Abstract
    specification
    Implementation
    Proof
    Executable
    specification
    Proof

    View full-size slide

  34. Questions?
    Answers!
    Lars Hupel
    https://lars.hupel.info
    [email protected]

    View full-size slide

  35. Image sources
    ● Edsger W. Dijskstra: Hamilton Richards, CC-BY-SA 3.0,
    https://commons.wikimedia.org/w/index.php?title=File:Edsger_Wybe_Dijkstra.jpg&oldid=710250
    942
    ● César A. Muñoz: https://shemesh.larc.nasa.gov/people/cam/
    ● BPMN: Mikelo Skarabo, CC-BY-SA 4.0,
    https://commons.wikimedia.org/w/index.php?title=File:BPMN-
    AProcessWithNormalFlow.svg&oldid=734511959

    View full-size slide