1 25-27 NOVEMBER SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Automation in Software Testing. Humans and Complex Models. Iosif Itkin CEO and co-founder, Exactpro

2 About Exactpro

3 3

4 4 MANAGED SUPPORT SERVICES PROVIDER OF THE YEAR In this Risk.net interview, Alexey Zverev, Exactpro co-CEO and co-founder, talks about mitigating risks while driving innovation during the time of high volatility and explains what made Exactpro stand out among the contestants. Watch the interview with Risk.net via the QR code above.

5 ● https://www.linkedin.com/in/iosifitkin/ ● Exactpro representative at the Zero Outage Industry Standard association ● Head of LOST (Laboratory of Software Testing) at Tomsk Polytechnic University ● Co-chair of FIX Trading Community MOST Working Group (Monitoring, Onboarding and Software Testing) ● Co-founder of TMPA (Tools & Methods of Program Analysis) ● Software Testing Machine Learning and Complex Process Analysis About me

6 Zero Outage Industry Standard (ZOIS) • Platforms • People • Processes • Security

7 Monitoring, Onboarding and Software Testing Working Group within FIX Trading Community Exactpro Partners For over a year, the Monitoring, Onboarding and Software Testing (MOST) working group within the FIX Trading Community has been meeting regularly to grow a community focused on standardization and innovation in technology processes leveraging the FIX protocol. The group’s mission is to raise the quality and reliability of platforms operated by the FIX Trading Community Members. This is being implemented via developing a set of recommended practices for monitoring, client onboarding and software testing of FIX-related financial applications as well as creating technical guidelines for the assessment of their quality and reliability. The group has presented a detailed mind map of such a set of best practices and guidelines, and continues working on the content of the document. The MOST members meet on a monthly basis and also collaborate with the FIX Orchestra and the Cybersecurity working groups to ensure alignment across a number of related topics.

8 Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course Software testing for complex intelligent systems and autonomous vehicles - Playlist on the Exactpro YouTube channel Laboratory of Software Testing

9 Laboratory of Software Testing

10

11 Laboratory of Software Testing - AI Testing

12 Laboratory of Software Testing - AI Testing

13 Laboratory of Software Testing - AI Testing

14 What is Software Testing

15 Art, Craft, Science, Engineering Where does software testing belong on this continuum?

16 https://fs.blog/2013/07/the-difference-between-science-and-engineering/ Abstract Model = theory Concrete Description = data Physical System = object of study flow of information measure compare Abstract Model = design concept Concrete Description = specification Physical System = useful product flow of information design produce Scientific Inquiry Engineering Design The Antiparallel Structures of Scientific Inquiry and Engineering Design Science and Engineering

17 Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software Testing

18 Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development Software Testing

19 Never invest in a business you cannot understand “ Warren Buffett

20 Models Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development Understanding Your Technology Assets

21 A model is a simplified, formal representation of a relationship, process or system. The simplification makes some aspects of the thing modeled clearer, more visible, and easier to work with. All tests are based on models, but many of those models are implicit. When the behavior of the program “feels wrong”, it is clashing with your internal model of the program and how it should behave. Black Box Software Testing Cem Kaner & James Bach All Testing is Model Based Testing

22 BA DEV QA Given Everything is Broken When Time is Up Then Build Back Better Behavior Driven Development

23 The best material model of a cat is another, or preferably the same, cat. Arturo Rosenblueth ”

24 Parallel Run / Operational Day Replay

25 Manual Using Tools Mechanization Automation A tool is an object used to extend the ability of an individual to modify features of the surrounding environment Automation is the technology by which a process or procedure is performed with minimal human assistance Manual vs. Automated

26 Bill Gates The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency. ”

27 Processes Platforms People Managed Software Testing

28 Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model‑based test case generation and prioritization: a systematic literature review. 2021, Springer. State of the Art

29 Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT System Adapters Execution Environment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5 ● more formal models (expressed in some machine-readable, well-defined, notation) ● more formal test generation (we are interested in test generation algorithms) ● more formal checks for mechanical execution (the generated tests must be sufficiently precise to efficiently use humans’ involvement) Narrow Definition of Model Based Testing

30 A great challenge of life: Knowing enough to think you are right, but not knowing enough to know you are wrong. Neil deGrasse Tyson “

31 Humans and Complex Models

32 George E. P. Box ...all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind…

33 Humans and Complex Models

34 Aleatoric vs. Epistemic Works 90% of the time

35 Simplified Models .01 .09 .09 .81 weapons fail independently fails works weapon 1 weapon 2 fails works .10 0 0 .90 weapons fail from a common cause fails works weapon 1 weapon 2 fails works

36 Snake? Spear? Wall? Rope? Tree? Cut your problems down to size * From Bernard L. Madoff Investment Securities LLC Simplified Models

37 Defense in Depth

38 Defense in Depth

39 Simplified Models

40 Scout mindset: the motivation to see things as they are, not as you wish they were. Julia Galef ”

41 Microservices Platform SUT DW Diversity

42 Where does software testing belong on this continuum? Art, Craft, Science, Engineering, Journalism, Magic

43 The only way of discovering the limits of the possible is to venture a little way past them into the impossible. Any sufficiently advanced technology is indistinguishable from magic. Arthur Clarke

44 Too many journalists see their work as an opportunity to promote their own pet political notions, rather than a responsibility to inform the public and let their readers and viewers decide for themselves. Thomas Sowell “

45 45 FOLLOW US ON SOCIAL MEDIA: @exactprosystems TO #th2net

46 Thank You Follow TMPA on Facebook TMPA-2021 Conference