Data-driven Testing and Test Automation for Complex Intelligent Systems

Iosif Itkin, CEO and co-founder, Exactpro
Elena Treshcheva, Program Manager, Exactpro

The Third IEEE International Conference On Artificial Intelligence Testing
23 - 26 August 2021
____
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. The Third IEEE International Conference
   On Artificial Intelligence Testing
   Iosif Itkin, CEO and co-founder, Exactpro
   Elena Treshcheva, Program Manager, Exactpro
   DATA-DRIVEN TESTING AND TEST AUTOMATION
   FOR COMPLEX INTELLIGENT SYSTEMS
   

   View Slide

2. 2 The Third IEEE International Conference
   On Artificial Intelligence Testing
   BUILD SOFTWARE TO TEST SOFTWARE
   2
   The 2021 IEEE Autonomous Driving
   AI Test Challenge
   August 23-26, 2021 - Online, Worldwide
   August 23 - team demos
   August 26 - selected paper presentations
   August 26 - Award Ceremony (5:00 pm BST / 9:00 am PST)
   

   View Slide

3. BUILD SOFTWARE TO TEST SOFTWARE
   The Third IEEE International Conference
   On Artificial Intelligence Testing
   3
   Overview
   ● Fintech applications complexity
   ● Present-day requirements of
   performance and high availability
   ● Requirements for next-generation
   test automation tools
   ● HiVAT paradigm
   th2 architecture
   Open-source model
   What
   Why
   How
   

   View Slide

4. 4 The Third IEEE International Conference
   On Artificial Intelligence Testing
   BUILD SOFTWARE TO TEST SOFTWARE
   4
   A specialist firm focused on
   functional and non-functional
   testing services for financial
   market infrastructures, financial
   and non-financial institutions.
   Headquartered in the UK with
   operations in the US and Eastern
   Europe. Most of our clients are
   regulated market infrastructures.
   Incorporated in 2009 with 10
   people, Exactpro is now
   employing over 650 specialists.
   Part of the London Stock
   Exchange Group (LSEG) from
   May 2015 until January 2018,
   when the Exactpro management
   proceeded through the buyout of
   the company from LSEG
   About Exactpro
   Exactpro provides independent
   software testing services for
   mission critical technology that
   underpins global financial
   markets. The firm is experienced
   with trading and clearing &
   settlement platforms, risk
   management systems, central
   data warehouses, accounting
   and treasury systems, regulatory
   reporting.
   

   View Slide

5. 5 The Third IEEE International Conference
   On Artificial Intelligence Testing
   BUILD SOFTWARE TO TEST SOFTWARE
   5
   Global Client Network – Financial Market Infrastructures
   

   View Slide

6. BUILD SOFTWARE TO TEST SOFTWARE
   The Third IEEE International Conference
   On Artificial Intelligence Testing
   6
   WHY
   

   View Slide

7. 7 The Third IEEE International Conference
   On Artificial Intelligence Testing
   BUILD SOFTWARE TO TEST SOFTWARE
   7
   Software complexity
   Dynamic
   why?
   why?
   Component Coordinative
   

   View Slide

8. 8 The Third IEEE International Conference
   On Artificial Intelligence Testing
   BUILD SOFTWARE TO TEST SOFTWARE
   8
   Component complexity (AI, NLP)
   why?
   https://www.stateof.ai/
   why?
   

   View Slide

9. 9 The Third IEEE International Conference
   On Artificial Intelligence Testing
   BUILD SOFTWARE TO TEST SOFTWARE
   9
   why?
   Coordinative and dynamic complexity (fintech platforms)
   

   View Slide

10. 10 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    10
    Continued surge of technology advancements
    Data Volumes Computational Power
    why?
    

    View Slide

11. 11 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    11
    Recent Advances in Machine Learning
    Data Volumes
    why?
    https://www.stateof.ai/
    

    View Slide

12. 12 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    12
    Recent Advances in Machine Learning
    Computational Power
    why?
    https://www.stateof.ai/
    

    View Slide

13. 13 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    13
    ● Daily capacity – 300+ mln transactions
    ● Peakrates – 50k transactions per second per partition
    ● Average round-trip latency – <60 microseconds
    ● Availability – high nines
    2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024
    2
    5 6.5 9
    12.5
    15.5 18
    26
    33
    41
    59
    74
    94
    118
    149
    Data Volume in zettabytes
    why?
    Fintech platforms: requirements
    

    View Slide

14. 14 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    14
    Distributed, resilient latency sensitive systems
    

    View Slide

15. 15 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    15
    Non-determinism
    Traditional Technology AI-based Software
    why?
    

    View Slide

16. 16 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    16
    Testing tool to match the complexity of the system under test
    System
    under test
    Test
    Library
    Complex System –
    Simple Tools
    Complex System –
    Complex Tools
    why?
    

    View Slide

17. BUILD SOFTWARE TO TEST SOFTWARE
    The Third IEEE International Conference
    On Artificial Intelligence Testing
    17
    WHAT
    

    View Slide

18. 18 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    18
    Testing tool to match the complexity of the system under test
    what?
    

    View Slide

19. 19 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    19
    Requirements
    Functional
    Testing
    Non-Functional
    Testing
    what?
    

    View Slide

20. 20 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    20
    Requirements Diversity and
    Unification
    Scalability
    Resilience
    Data Analysis
    Modularity
    Performance
    Configuration
    Agility
    what?
    Explainability
    

    View Slide

21. 21 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    21
    Performance and Scalability
    SUT
    DW
    what?
    

    View Slide

22. 22 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    22
    Diversity and Unification
    SUT
    DW
    Diversity
    what?
    

    View Slide

23. BUILD SOFTWARE TO TEST SOFTWARE
    The Third IEEE International Conference
    On Artificial Intelligence Testing
    23
    HOW
    

    View Slide

24. 24 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    24
    Open-source model
    how? https://github.com/th2-net
    

    View Slide

25. 25 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    25
    Open-source model
    how?
    

    View Slide

26. BUILD SOFTWARE TO TEST SOFTWARE
    26 The Third IEEE International Conference
    On Artificial Intelligence Testing
    Open for Collaboration
    

    View Slide

27. 27 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    27
    TMPA - Tools & Methods of Program Analysis Conference
    https://tmpaconf.org/events/tmpa-2021/
    

    View Slide

28. 28 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    28
    Exactpro YouTube Channel
    Software Testing for Complex
    Intelligent Systems and
    Autonomous Vehicles
    Online Course
    Software testing for complex intelligent systems and autonomous vehicles
    - Playlist on the Exactpro YouTube channel
    

    View Slide

29. 29 The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    29
    References
    1. Jung, C., Mueller, H., Pedemonte, S., Plances, S., & Thew, O. Machine learning in UK financial services. Joint report by
    the FCA and the Bank of England. October 2019.
    2. LabCFTC (Commodities and Futures Trading Commission). A Primer on Artificial Intelligence in Financial Markets.
    October 2019.
    3. Zhang, J. M., Harman, M., Ma, L., & Liu, Y. Machine Learning Testing: Survey, Landscapes and Horizons. In: IEEE
    Transactions on Software Engineering. 2020.
    4. Treshcheva, E., Yavorsky, R., & Itkin, I. Toward Reducing the Operational Risk of Emerging Technologies Adoption in
    Central Counterparties through End-to-End Testing. In: Journal of Financial Market Infrastructures 8(3), 2020: 51-74.
    5. Wood, R.E. Task Complexity: Definition of a Construct. In: Organizational Behavior and Human Decision Processes,
    vol. 31, 1986: 60-82.
    6. Banker, R. D., Davis, G. B., & Slaughter, S. A. Software development practices, software complexity, and software
    maintenance performance: A field study. Management science 44(4), 1998: 433-45
    7. Protsenko, P., Khristenok, A., Lukina, A., Alexeenko, A., Pavlyuk, T., & Itkin, I. Trading day logs replay limitations and
    test tools applicability. In Proceedings. International Conference on Tools and Methods of Program Analysis (TMPA 2014),
    2014: 46-53.
    8. Benaich, N. & Hogarth, I. State of AI Report 2020. October, 2020. 177p.
    

    View Slide

30. Thank you!
    The Third IEEE International Conference
    On Artificial Intelligence Testing
    BUILD SOFTWARE TO TEST SOFTWARE
    

    View Slide