1 Build Software to Test Software exactpro.com Lecture 4 Foundations of Software Testing Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course 

2 Build Software to Test Software exactpro.com Lecture structure Sources of information about testing What is testing? What is quality, errors, defects, failures? What testing is not: QA, unit testing, BDD, bad testing Principles and types of testing Testing objectives; what constitutes good testing Testing Components: processes, platforms, people The testing process: phases and results Is it the testers' fault that failures occur in production 

3 Build Software to Test Software exactpro.com Exactpro The company was founded on Tester Day, September 9, 2009 The company’s headquarters are in London Moscow Kostroma Saratov Obninsk Tbilisi London New York 

4 Build Software to Test Software exactpro.com What We Do Our company specializes in development and implementation of high load market infrastructures as well as systems that work with financial tools/platforms. Our solutions are used in more than 20 countries all over the world. 

5 Build Software to Test Software exactpro.com Sources of information - Absence-of-errors fallacy and other principles of software testing. - Types and levels of software testing. - Understanding of test objectives and methodologies in different schools of thought in the software testing domain. 

6 Build Software to Test Software exactpro.com Sources of Information https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/ 

7 Build Software to Test Software exactpro.com Sources of Information https://youtu.be/jZitXMQaXvE Watch Yegor Bugayenko 

8 Build Software to Test Software exactpro.com Sources of Information https://ingophilipp.medium.com/excellent-software-testing-6e0fd8f3e11e 

9 Build Software to Test Software exactpro.com Sources of Information http://cartoontester.blogspot.com/ Andy Glover 

10 Build Software to Test Software exactpro.com Software Testing: Definition ● Testing is about extracting information about software quality ● Testing is pain ● Testing is relentless learning ● Testing is about processes, platforms, people 

11 Build Software to Test Software exactpro.com West World TEST WORLD: REBOOT 

12 Build Software to Test Software exactpro.com Ray Dalio PAIN + REFLECTION = PROGRESS 

13 Build Software to Test Software exactpro.com Ray Dalio 

14 Build Software to Test Software exactpro.com Ray Dalio 

15 Build Software to Test Software exactpro.com How We Get Information 

16 Build Software to Test Software exactpro.com How We Get Information 

17 Build Software to Test Software exactpro.com https://blogs.scientificamerican.com/observations/the-line-between-science-and-journali sm-is-getting-blurry-again/ ● Journalism is communication of 'what's new'. A journalist is anyone who can say "I’m there, you’re not, let me tell you about it." ● Science is communication of 'how the world works'. A scientist is anyone who can say "I understand something about the world, you don't, let me explain it to you". ● Neither definition necessitates that what they say is True, just what they know to the best of their ability and understanding. 

18 Build Software to Test Software exactpro.com How We Get Information 

19 Build Software to Test Software exactpro.com How We Get Information 

20 Build Software to Test Software exactpro.com Quality Quality is value to some people Quality is the degree to which a component or system satisfies the stated and implied needs of its various stakeholders. Quality is inherently subjective. Different stakeholders will perceive the same product as having different levels of quality. 

21 Build Software to Test Software exactpro.com What is a Bug? Bug is a discrepancy between expectations and reality An attribute of a software product that reduces its value to a stakeholder or increases its value to a competitor. May or may not be a coding error or a functional error Design errors are bugs too. 

22 Build Software to Test Software exactpro.com What is a Bug? Error – a human action that produces an incorrect result Defect (bug) – an imperfection or deficiency in a work product where it does not meet its requirements or specifications Failure – an event in which a component or system does not perform a required function within specified limits 

23 Build Software to Test Software exactpro.com What is a Bug? 65,586 m/ h (~65.6 km/h) error defect/bug failure 

24 Build Software to Test Software exactpro.com What is a Bug? 65,586 m/ h (~65.6 km/h) error defect/bug failure 

25 Build Software to Test Software exactpro.com What is a Bug? Breaking path 

26 Build Software to Test Software exactpro.com What is a Bug? Breaking path https://cordis.europa.eu/arti cle/id/19509-ariane-5-explo sion-caused-by-fault-in-mai n-engine-cooling-system 

27 Build Software to Test Software exactpro.com https://www.snopes.com/fact-check/cone-of-silence PASS FAIL PASS + = = = + + What is a Bug? 

28 Build Software to Test Software exactpro.com What is Software Testing? Software Testing is an ● empirical ● technical ● investigation ● conducted to provide stakeholders ● with information ● about the quality ● of the product or service under test Software Testing is exploring software with the intent of finding bugs 

29 Build Software to Test Software exactpro.com Watch “Software is Eating the World.” 

30 Build Software to Test Software exactpro.com “Software is Eating the World.” http://www3.weforum.org/docs/WEF_ The_Global_Risks_Report_2021.pdf 

31 Build Software to Test Software exactpro.com Software Testing is not Quality Assurance Testers: Get Out of the Quality Assurance Business Quality Assurance Quality Control Testing Code review Unit testing Audit Process Improvements Change management Early reviews and testing 

32 Build Software to Test Software exactpro.com Software Testing is an Information Service 

33 Build Software to Test Software exactpro.com Testing is not Quality Assurance Testers: Get Out of the Quality Assurance Business https://www.developsense.com/presentations/2010-06-TestersGetO utOfTheQABusiness.pdf 

34 Build Software to Test Software exactpro.com Unit Testing Development of unit tests allows you to isolate individual parts of the program and test their performance - creating tests, executing them and obtaining reports using open source frameworks Pytest and PyUnit. 

35 Build Software to Test Software exactpro.com BDD Studying the Behavior Driven Development (BDD) software development methodology Using the Gherkin in the requirements definition process, use cases, and acceptance testing 

36 Build Software to Test Software exactpro.com Software Testing Not Software Testing 

37 Build Software to Test Software exactpro.com Examples of Obtaining Information 

38 Build Software to Test Software exactpro.com Examples of Obtaining Information 

39 Build Software to Test Software exactpro.com Examples of obtaining and disseminating information 

40 Build Software to Test Software exactpro.com Bad Testing https://youtu.be/2S--Ge3c-yA 

41 Build Software to Test Software exactpro.com Software Testing is not... https://mrslavchev.com/2016/03/28/software-testing-not/ 

42 Build Software to Test Software exactpro.com 

43 Build Software to Test Software exactpro.com Testing Principles Principle 1: Testing shows the presence of defects, not their absence Principle 2: Exhaustive testing is impossible Principle 3: Early testing saves time and money Principle 4: Defects cluster together Principle 5: Beware of the pesticide paradox Principle 6: Testing is context dependent Principle 7: Absence-of-errors is a fallacy 

44 Build Software to Test Software exactpro.com Testing shows the presence of defects, not their absence 

45 Build Software to Test Software exactpro.com Exhaustive testing is impossible 

46 Build Software to Test Software exactpro.com Autonomous Vehicle Testing 

47 Build Software to Test Software exactpro.com LGSVL Road Simulator - Introduction to the multi-agent road simulator developed by LG's Research and Development division based on the cross-platform environment for creating video games - Unity. - Overview of installation, launch, and user interface. 

48 Build Software to Test Software exactpro.com Pareto Principle 

49 Build Software to Test Software exactpro.com Beware of the Pesticide Paradox 

50 Build Software to Test Software exactpro.com Beware of the Pesticide Paradox 

51 Build Software to Test Software exactpro.com Context Driven Testing Consider: ● Software development lifecycle model and project methodologies being used ● Test levels and test types being considered ● Product and project risks ● Business domain ● Operational constraints, including but not limited to: a. Budgets and resources b. Timescales c. Complexity d. Contractual and regulatory requirements ● Organizational policies and practices ● Required internal and external standards 

52 Build Software to Test Software exactpro.com Context Driven Testing The Seven Basic Principles of the Context-Driven School 1. The value of any practice depends on its context. 2. There are good practices in context, but there are no best practices. 3. People, working together, are the most important part of any project’s context. 4. Projects unfold over time in ways that are often not predictable. 5. The product is a solution. If the problem isn’t solved, the product doesn’t work. 6. Good software testing is a challenging intellectual process. 7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products. https://context-driven-testing.com/ 

53 Build Software to Test Software exactpro.com 

54 Build Software to Test Software exactpro.com Absence-of-errors is a fallacy 

55 Build Software to Test Software exactpro.com Verification vs. Validation Verification – checks whether the system meets the specified requirements Validation – checks whether the system will meet user and other stakeholder needs ● https://techcrunch.com/2018/06/22/uber-safety-driver-of-fatal-self-driving-crash-was-watchin g-hulu-not-the-road ● https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf 

56 Build Software to Test Software exactpro.com https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf 

57 Build Software to Test Software exactpro.com https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf 

58 Build Software to Test Software exactpro.com Types of Software Testing ● Functional testing — WHAT ● Non-functional testing — HOW ● Testing with access to the source code — White Box 

59 Build Software to Test Software exactpro.com Software Testing at the confluence Functional Testing Nonfunctional Testing 

60 Build Software to Test Software exactpro.com Black Box Testing vs. White Box Testing 

61 Build Software to Test Software exactpro.com Testing Components 

62 Build Software to Test Software exactpro.com ● Test planning – involves defining the objectives of testing and the approach for meeting those objectives within project constraints and contexts ● Test monitoring and control – compares actual progress against the plan, takes actions which are necessary to meet the mission and objectives of the project, and/or adjust the plan ● Test analysis – determines ‘what to test’ ● Test design – addresses the question ‘how to test’ ● Test implementation – asks ‘do we now have everything in place to run the tests? ● Test execution – runs tests ● Test completion – summarizes the results of testing Testing Components 

63 Build Software to Test Software exactpro.com Test Objectives 1. To prevent defects by evaluating work products such as requirements, user stories, design, and code 2. To verify whether all specified requirements have been fulfilled 3. To check whether the test object is complete and validate if it works as the users and other stakeholders expect 4. To build confidence in the level of quality of the test object 5. To find defects and failures thus reduce the level of risk of inadequate software quality 6. To provide sufficient information to stakeholders to allow them to make informed decisions, especially regarding the level of quality of the test object 7. To comply with contractual, legal, or regulatory requirements or standards, and/or to verify the test object’s compliance with such requirements or standards 

64 Build Software to Test Software exactpro.com 

65 Build Software to Test Software exactpro.com Bad Testing Doctor Fauci 

66 Build Software to Test Software exactpro.com Testers vs. Activists 

67 Build Software to Test Software exactpro.com Test Objectives ● To prevent defects by evaluate work products such as requirements, user stories, design, and code ● To verify whether all specified requirements have been fulfilled ● To check whether the test object is complete and validate if it works as the users and other stakeholders expect ● To build confidence in the level of quality of the test object ● To find defects and failures thus reduce the level of risk of inadequate software quality ● To provide sufficient information to stakeholders to allow them to make informed decisions, especially regarding the level of quality of the test object ● To comply with contractual, legal, or regulatory requirements or standards, and/or to verify the test object’s compliance with such requirements or standards 

68 Build Software to Test Software exactpro.com 2021 IEEE Autonomous Driving AI Test Challenge http://av-test-challenge.org/ 

69 Build Software to Test Software exactpro.com Good Testing 

70 Build Software to Test Software exactpro.com Test Objectives 

71 Build Software to Test Software exactpro.com Testing is not Quality Assurance Testers: Get Out of the Quality Assurance Business Quality Assurance Quality Control Testing Code review Unit testing Audit Process Improvements Change management Early reviews and testing 

72 Build Software to Test Software exactpro.com Good Testing 

73 Build Software to Test Software exactpro.com Good Testing 

74 Build Software to Test Software exactpro.com Good Testing ● Ownership – taking care of your company. Treat others the way you would like to be treated. Take responsibility for company's needs. Remember: responsibility is power. It is not granted. It is taken. Extreme ownership is the way to lead and win. ● Deep Impact – exitus acta probat. The outcome justifies the effort. Make sure that your work results in a valuable contribution. If you want to go fast, go alone. If you want to go far, go together. Involve others to achieve scalability and increase the impact of our work. ● Hard Work – no pain, no gain. Sometimes putting in long hours does not help achieve any result. We do not measure time spent in the office. Having said that, we've never observed extraordinary results without hard work and deliberate effort. 

75 Build Software to Test Software exactpro.com ● Learning – software testing is relentless learning. Change is the end result of all true learning. The beautiful thing about learning is that nobody can take it away from you. The Mistakes and problems are inevitable. However, it is unacceptable not to learn from them. The best way to learn something is to teach it. ● Agility – the only constant is change. We embrace the values reflected in the Manifesto for Agile Software Development. Agility allows us to turn learning into action and action into change. From its inception, Exactpro has been able to adapt to change. It is important to keep this ability as we scale up our business. ● Communication – the medium is the message. A relationship is based on communication. We need to make sure others understand us and we understand them. ● Objectivity – you have nothing to fear from knowing the truth. It is at the core of our business. Our main service is to build software to test software and provide objective information about it to our clients. Our business benefits from openness. Everyone in our company has the right to understand what makes sense, and no one has the right to hold a critical opinion without speaking up. Good Testing 

76 Build Software to Test Software exactpro.com Thank you Exactpro Systems Exactpro Vlog