TMPA-2021: AUTOMATION IN SOFTWARE TESTING. HUMANS AND COMPLEX MODELS.

TMPA is an annual International Conference on Software Testing, Machine Learning and Complex Process Analysis. The conference will focus on the application of modern methods of data science to the analysis of software quality.

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. 1 25-27 NOVEMBER SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Automation in Software Testing. Humans and Complex Models. Iosif Itkin CEO and co-founder, Exactpro
2. 2 About Exactpro
3. 3 3
4. 4 4 MANAGED SUPPORT SERVICES PROVIDER OF THE YEAR In this Risk.net interview, Alexey Zverev, Exactpro co-CEO and co-founder, talks about mitigating risks while driving innovation during the time of high volatility and explains what made Exactpro stand out among the contestants. Watch the interview with Risk.net via the QR code above.
5. 5 • https://www.linkedin.com/in/iosifitkin/ • Exactpro representative at the Zero Outage Industry Standard association • Head of LOST (Laboratory of Software Testing) at Tomsk Polytechnic University • Co-chair of FIX Trading Community MOST Working Group (Monitoring, Onboarding and Software Testing) • Co-founder of TMPA (Tools & Methods of Program Analysis) • Software Testing Machine Learning and Complex Process Analysis About me
6. 6 Zero Outage Industry Standard (ZOIS) • Platforms • People • Processes • Security
7. 7 Monitoring, Onboarding and Software Testing Working Group within FIX Trading Community Exactpro Partners For over a year, the Monitoring, Onboarding and Software Testing (MOST) working group within the FIX Trading Community has been meeting regularly to grow a community focused on standardization and innovation in technology processes leveraging the FIX protocol. The group’s mission is to raise the quality and reliability of platforms operated by the FIX Trading Community Members. This is being implemented via developing a set of recommended practices for monitoring, client onboarding and software testing of FIX-related financial applications as well as creating technical guidelines for the assessment of their quality and reliability. The group has presented a detailed mind map of such a set of best practices and guidelines, and continues working on the content of the document. The MOST members meet on a monthly basis and also collaborate with the FIX Orchestra and the Cybersecurity working groups to ensure alignment across a number of related topics.
8. 8 Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course Software testing for complex intelligent systems and autonomous vehicles - Playlist on the Exactpro YouTube channel Laboratory of Software Testing
9. 9 Laboratory of Software Testing
11. 11 Laboratory of Software Testing - AI Testing
12. 12 Laboratory of Software Testing - AI Testing
13. 13 Laboratory of Software Testing - AI Testing
14. 14 What is Software Testing
15. 15 Art, Craft, Science, Engineering Where does software testing belong on this continuum?
16. 16 https://fs.blog/2013/07/the-difference-between-science-and-engineering/ Abstract Model = theory Concrete Description = data Physical System = object of study flow of information measure compare Abstract Model = design concept Concrete Description = specification Physical System = useful product flow of information design produce Scientific Inquiry Engineering Design The Antiparallel Structures of Scientific Inquiry and Engineering Design Science and Engineering
17. 17 Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software Testing
18. 18 Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development Software Testing
19. 19 Never invest in a business you cannot understand “ Warren Buffett
20. 20 Models Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development Understanding Your Technology Assets
21. 21 A model is a simplified, formal representation of a relationship, process or system. The simplification makes some aspects of the thing modeled clearer, more visible, and easier to work with. All tests are based on models, but many of those models are implicit. When the behavior of the program “feels wrong”, it is clashing with your internal model of the program and how it should behave. Black Box Software Testing Cem Kaner & James Bach All Testing is Model Based Testing
22. 22 BA DEV QA Given Everything is Broken When Time is Up Then Build Back Better Behavior Driven Development
23. 23 The best material model of a cat is another, or preferably the same, cat. Arturo Rosenblueth ”
24. 24 Parallel Run / Operational Day Replay
25. 25 Manual Using Tools Mechanization Automation A tool is an object used to extend the ability of an individual to modify features of the surrounding environment Automation is the technology by which a process or procedure is performed with minimal human assistance Manual vs. Automated
26. 26 Bill Gates The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency. ”
27. 27 Processes Platforms People Managed Software Testing
28. 28 Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model‑based test case generation and prioritization: a systematic literature review. 2021, Springer. State of the Art
29. 29 Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT System Adapters Execution Environment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5 • more formal models (expressed in some machine-readable, well-defined, notation) • more formal test generation (we are interested in test generation algorithms) • more formal checks for mechanical execution (the generated tests must be sufficiently precise to efficiently use humans’ involvement) Narrow Definition of Model Based Testing
30. 30 A great challenge of life: Knowing enough to think you are right, but not knowing enough to know you are wrong. Neil deGrasse Tyson “
31. 31 Humans and Complex Models
32. 32 George E. P. Box ...all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind…
33. 33 Humans and Complex Models
34. 34 Aleatoric vs. Epistemic Works 90% of the time
35. 35 Simplified Models .01 .09 .09 .81 weapons fail independently fails works weapon 1 weapon 2 fails works .10 0 0 .90 weapons fail from a common cause fails works weapon 1 weapon 2 fails works
36. 36 Snake? Spear? Wall? Rope? Tree? Cut your problems down to size * From Bernard L. Madoff Investment Securities LLC Simplified Models
37. 37 Defense in Depth
38. 38 Defense in Depth
39. 39 Simplified Models
40. 40 Scout mindset: the motivation to see things as they are, not as you wish they were. Julia Galef ”
41. 41 Microservices Platform SUT DW Diversity
42. 42 Where does software testing belong on this continuum? Art, Craft, Science, Engineering, Journalism, Magic
43. 43 The only way of discovering the limits of the possible is to venture a little way past them into the impossible. Any sufficiently advanced technology is indistinguishable from magic. Arthur Clarke
44. 44 Too many journalists see their work as an opportunity to promote their own pet political notions, rather than a responsibility to inform the public and let their readers and viewers decide for themselves. Thomas Sowell “
45. 45 45 FOLLOW US ON SOCIAL MEDIA: @exactprosystems TO #th2net
46. 46 Thank You Follow TMPA on Facebook TMPA-2021 Conference