Lecture 8: Software negligence and testing coverage

"Software Testing for Complex Intelligent Systems and Autonomous Vehicles" online course.

The lecture is available via the link: https://youtu.be/MoYDDHMfHbk
--
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software exactpro.com
   Lecture 8
   Software negligence and test coverage
   Software Testing for
   Complex Intelligent Systems
   and Autonomous Vehicles
   Online Course
   

   View Slide

2. 2 Build Software to Test Software exactpro.com
   Lecture Structure
   The concepts of professional malpractice and
   criminal negligence
   The issue of liability for software malfunction.
   The principle that exhaustive testing is impossible.
   Test coverage. Types of test coverage.
   The relation between coverage and liability for
   software malfunction
   

   View Slide

3. 3 Build Software to Test Software exactpro.com
   Reading Materials
   Cem Kaner
   Software Negligence and Testing Coverage
   SOFTWARE NEGLIGENCE AND
   TESTING COVERAGE
   Cem Kaner, kaner@kaner.com
   Copyright, 1995, all rights reserved.
   Portions of this paper were originally published in C. Kaner, “Software Negligence &
   Testing Coverage”, Software QA Quarterly, Vol. 2, #2, p. 18, 1995.
   1
   1
   

   View Slide

4. 4 Build Software to Test Software exactpro.com
   Reading Materials
   Iosif Itkin for FOW
   Assuring compliance with pending AI liability regulation
   

   View Slide

5. 5 Build Software to Test Software exactpro.com
   Reading Materials
   ISTQB Foundation Level Syllabus 2018
   

   View Slide

6. 6 Build Software to Test Software exactpro.com
   Cem Kaner: Case Study
   Cem Kaner
   Software Negligence and Testing Coverage
   

   View Slide

7. 7 Build Software to Test Software exactpro.com
   Negligence in
   Relation to
   Software
   MISREPRESENTATION
   

   View Slide

8. 8 Build Software to Test Software exactpro.com
   The Question of Responsibility for Test Results
   VS.
   

   View Slide

9. 9 Build Software to Test Software exactpro.com
   The Subject
   The Field of IT
   The Field of Law
   

   View Slide

10. 10 Build Software to Test Software exactpro.com
    Negligence/Malpractice
    Intent
    Adequate Measures
    Intent
    Adequate Measures
    Professionalism
    VS.
    

    View Slide

11. 11 Build Software to Test Software exactpro.com
    Negligence/Malpractice (Kaner)
    Intent
    Adequate Measures
    Intent
    Adequate Measures
    Professionalism
    VS.
    

    View Slide

12. 12 Build Software to Test Software exactpro.com
    Negligence/Malpractice (Kaner)
    Intent
    Adequate Measures
    Intent
    Adequate Measures
    Professionalism
    VS.
    Standard?
    

    View Slide

13. 13 Build Software to Test Software exactpro.com
    Negligence/Malpractice (Kaner)
    

    View Slide

14. 14 Build Software to Test Software exactpro.com
    Negligence/Malpractice (Kaner)
    Intent
    Adequate Measures
    Intent
    Adequate Measures
    Professionalism
    VS.
    ???
    ???
    

    View Slide

15. 15 Build Software to Test Software exactpro.com
    Reasonable measures
    Cem Kaner
    Software Negligence and Testing Coverage
    Costs to prevent
    Cost of the error
    

    View Slide

16. 16 Build Software to Test Software exactpro.com
    Reasonable measures
    Cem Kaner
    Software Negligence and Testing Coverage
    Cost of the error
    Costs to prevent
    + the cost of
    finding the
    bug
    “What would it have cost to make the testing process
    thorough enough that you would have found this bug
    during testing?”
    

    View Slide

17. 17 Build Software to Test Software exactpro.com
    Test Coverage
    

    View Slide

18. 18 Build Software to Test Software exactpro.com
    Line coverage
    Line coverage measures the number / percentage of
    lines of code that have been executed
    Cem Kaner
    Software Negligence and Testing Coverage
    

    View Slide

19. 19 Build Software to Test Software exactpro.com
    Black box testing vs. white box testing
    ISTQB Foundation Level Syllabus 2018
    

    View Slide

20. 20 Build Software to Test Software exactpro.com
    Cem Kaner: 101 coverage metrics
    “This appendix lists 101 coverage measures. Coverage
    measures the amount of testing done of a certain type.
    Because testing is done to find bugs, coverage is also a
    measure of your effort to detect a certain class of potential
    errors. For example, 100% line coverage doesn’t just mean
    that you’ve executed every line of code; it also means that
    you’ve tested for every bug that can be revealed by simple
    execution of a line of code.”
    Cem Kaner
    Software Negligence and Testing Coverage
    

    View Slide

21. 21 Build Software to Test Software exactpro.com
    Test Coverage: All Message Fields
    8=FIXT1.1
    | 9=90 |
    35=D | 49=User1 |
    56=FGW | 34=2 |
    55=ACME | 54=1 |
    40=2 | 38=40 |
    59=0 | 44=100 |
    11=ORD1 |
    10=197 |
    Test Coverage in FinTech:
    Protocol Logs and Discrete Values
    

    View Slide

22. 22 Build Software to Test Software exactpro.com
    Test Coverage: All System Events
    New
    Ack
    Working
    Fill
    Done
    Part Fill
    Canceled
    Expired
    Rejected
    Events:
    Requests:
    New/Amend/Cancel
    Market: Sessions/Trades
    Time: Triggers/Expiration
    ISTQB Foundation Level Certification Explained
    Chapter 4 (Part 1)
    

    View Slide

23. 23 Build Software to Test Software exactpro.com
    (Eng) Mind the Gap: Applying Process Mining
    https://focus.world-exchanges.org/articles/mind-gap-between-testing-and-
    production-applying-process-mining-test-resilience-exchange-platforms
    New (100)
    New (279)
    New (4784)
    END
    NEW
    Rejected
    Filled
    Expired
    Cancelled
    START
    New (38)
    Reject (3222)
    (3222)
    (100)
    (38)
    (279)
    (4784)
    Cancel (100)
    Cancel (38)
    Expired (279)
    Trade (4784)
    Trade (38)
    TradeCancel (38)
    Test Coverage: All Transitions between System
    States
    

    View Slide

24. 24 Build Software to Test Software exactpro.com
    Test Coverage: Requirement Coverage
    Test Coverage in FinTech:
    Protocol Logs and Discrete Values
    

    View Slide

25. 25 Build Software to Test Software exactpro.com
    Coverage Matrix: the Illusion of Exhaustive Test
    Coverage
    Test Coverage in FinTech:
    Protocol Logs and Discrete Values
    

    View Slide

26. 26 Build Software to Test Software exactpro.com
    Test Coverage: the Importance of Specific Values
    Fake Testing: "The Pentagon Wars" Film Review
    

    View Slide

27. 27 Build Software to Test Software exactpro.com
    AV AI Test Challenge: Simulation Coverage
    http://av-test-challenge.org/
    

    View Slide

28. 28 Build Software to Test Software exactpro.com
    https://www.svlsimulator.com/docs/python-api/python-api/#weather-
    and-time-of-day-control
    Test Coverage: Weather
    Weather:
    rain, fog, wetness, cloudiness
    

    View Slide

29. 29 Build Software to Test Software exactpro.com
    LGSVL - Vehicle Dynamics
    Test Coverage: Car Dynamics Parameters (the EGO
    vehicle)
    

    View Slide

30. 30 Build Software to Test Software exactpro.com
    https://www.svlsimulator.com/docs/python-api/python-api/#agents
    Test Coverage: Other Vehicles and Pedestrians
    NPC vehicles: Sedan, SUV, Jeep, Hatchback,
    SchoolBus, BoxTruck
    Pedestrians: Bob, EntrepreneurFemale, Howard,
    Johny, Pamela, Presley, Red, Robin, Stephen, Zoe
    

    View Slide

31. 31 Build Software to Test Software exactpro.com
    Coverage and
    Liability
    

    View Slide

32. 32 Build Software to Test Software exactpro.com
    Exhaustive Testing is Impossible
    

    View Slide

33. 33 Build Software to Test Software exactpro.com
    Other Offenses in Software Development
    
    VS.
    

    View Slide

34. 34 Build Software to Test Software exactpro.com
    Other Offenses in Software Development
    VS.
    MISREPRESENTATION
    

    View Slide

35. 35 Build Software to Test Software exactpro.com
    Software Responsibility in Artificial Intelligence
    Systems
    European Commission - Liability for Artificial Intelligence (2019)
    FCA and Bank of England - Machine learning in UK financial services
    (2019)
    US DoD - AI Principles: Recommendations on the Ethical Use of
    Artificial Intelligence (2019)
    US Whitehouse - Guidance for Regulation of Artificial Intelligence
    Applications (2019)
    FSB - Artificial intelligence and machine learning in financial services
    (2017)
    IOSCO - The use of artificial intelligence and machine learning by
    market intermediaries and asset managers (2020)
    

    View Slide

36. 36 Build Software to Test Software exactpro.com
    Product vs Service
    Product
    Service
    

    View Slide

37. 37 Build Software to Test Software exactpro.com
    AI Service
    - There is a duty to take adequate
    measures to ensure quality
    - There is a breach of duty
    - Violation is the cause of the damage
    service
    

    View Slide

38. 38 Build Software to Test Software exactpro.com
    AI Product
    - Product liability
    - Strict liability
    Product
    Intent
    Adequate Measures
    

    View Slide

39. 39 Build Software to Test Software exactpro.com
    Liability for AI Malfunction
    - Negligence
    - Product liability
    - Strict liability
    - Misrepresentation
    https://spectrum.ieee.org/cars-that-think/transportation/self-driving/why-
    you-shouldnt-worry-about-liability-for-selfdriving-car-accidents
    

    View Slide

40. 40 Build Software to Test Software exactpro.com
    Presenting Facts
    Volvo XC90 Drive Me Autonomous (self-driving) car
    interface (IntelliSafe Autopilot)
    

    View Slide

41. 41 Build Software to Test Software exactpro.com
    Presenting Facts
    Elon Musk on Twitter - 17 April
    Tesla (TSLA) Texas Crash: Two Men Dead After
    Accident With 'No One' Driving - Bloomberg, 18 April
    

    View Slide

42. 42 Build Software to Test Software exactpro.com
    Thank you!
    Join our Telegram chat, meet
    your peers, ask questions
    and get new software testing
    skills with Exactpro!
    Subscribe to our YouTube
    channel to see all the new
    videos!
    

    View Slide

43. 43 Build Software to Test Software exactpro.com
    Further Reading
    Cem Kaner - Legal Issues Related to Software
    Quality (1997)
    John Villasenor - Products Liability and Driverless
    Cars: Issues and Guiding Principles for Legislation
    (2014)
    Karen Mercedes Goertzel - Legal Liability for Bad
    Software (2016)
    John Kingston - Artificial Intelligence and Legal
    Liability (2018)
    Miriam Buiten - Towards Intelligent Regulation of
    Artificial Intelligence (2019)
    Darrell M. West - It is time to restore the US Office of
    Technology Assessment (2021)
    

    View Slide