Тестирование машинного обучения: обзор, пейзажи и горизонты

Иосиф Иткин
Генеральный директор, Exactpro

29 апреля 2020
Online QA Meetup

Иосиф работает над созданием инструментов и методов для тестирования высоконагруженных систем с 2006 года.
На встрече он поделится своими мыслями о недавно опубликованном масштабном исследовании «Тестирование машинного обучения: обзор, пейзажи и горизонты», проведенном учеными из Университетского колледжа Лондона (UCL): https://arxiv.org/pdf/1906.10742.pdf.

Видео: https://youtu.be/u826L6ijvUo

---
Подписывайтесь на Exactpro в социальных сетях:

LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Подписывайтесь на YouTube канал Exactpro https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software
   exactpro.com
   Iosif Itkin
   co-CEO and co-founder
   Exactpro Systems
   29 April 2020
   Machine Learning Testing: Survey, Landscapes and Horizons
   

   View Slide

2. 2 Build Software to Test Software
   exactpro.com
   The Subject of Today’s Presentation
   https://arxiv.org/pdf/1906.10742.pdf
   

   View Slide

3. 3 Build Software to Test Software
   exactpro.com
   The Source of Today’s Presentation
   https://www.eventbrite.com/e/online-talk-intelligent-testing-bringing-transparency-to-ml-in-automation-tickets-103719598110
   

   View Slide

4. 4 Build Software to Test Software
   exactpro.com
   The Source of Today’s Presentation
   https://www.slideshare.net/HeemengFoo/machine-learning-testing-survey-landscapes-and-horizons-the-cliff-notes
   

   View Slide

5. 5 Build Software to Test Software
   exactpro.com
   Previous Presentation
   25 February 2020
   https://youtu.be/KGVQVBdjvCU
   

   View Slide

6. 6 Build Software to Test Software
   exactpro.com
   Introduction
   ● Started IT career 20 years ago as a software developer
   ● Exactpro co-CEO and co-founder
   ● Moved into performance and load testing in 2006
   ● Co-founded Exactpro on 9th September 2009
   ● EXTENT Software Testing and AI conference – 19 November 2020
   ● Working on th2.net next generation test tool
   ● ISTQB FL 2018, A4Q AI and Software Testing
   ● Build Software to Test Software
   

   View Slide

7. 7 Build Software to Test Software
   exactpro.com
   EXACTPRO
   A few quick facts about Exactpro
   • A specialist firm focused on functional and non-functional testing of exchanges,
   clearing houses, depositories and other market infrastructures.
   • Incorporated in 2009 with 10 people, our
   company has experienced significant growth as
   satisfied clients require more services; now
   employing 580 specialists worldwide
   • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018.
   Exactpro management buyout from LSEG in January 2018. Headquartered in
   the UK, with operations in the US, Georgia and Russia.
   • We provide software testing services for mission critical
   technology that underpins global financial markets. Our clients
   are regulated by FCA, Bank of England and their counterparts
   from other countries.
   

   View Slide

8. 8 Build Software to Test Software
   exactpro.com
   EXACTPRO
   Our clients are the leading exchange groups across the globe
   

   View Slide

9. 9 Build Software to Test Software
   exactpro.com
   Recent Press Releases with Clients and Partners
   

   View Slide

10. 10 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Exactpro Partners and Cross Industry Work
    

    View Slide

11. 11 Build Software to Test Software
    exactpro.com
    11 Build Software to Test Software
    exactpro.com
    • Platforms
    • People
    • Processes
    • Security
    

    View Slide

12. 12 Build Software to Test Software
    exactpro.com
    Challenging Non-Functional Requirements
    Daily capacity - 100+ mln transactions
    Peak rates - 40k+ transactions per second
    Average round-trip latency - <100 microseconds
    Availability - 100%
    3000 trx 2.5 cm <1 mm
    

    View Slide

13. 13 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Indirect Testing Methods
    Information
    extraction and
    Machine learning
    End-to-End
    Automated Test
    Library
    Whatever it
    takes!
    Test execution
    data and log
    analysis
    Passive Testing
    

    View Slide

14. 14 Build Software to Test Software
    exactpro.com
    TMPA-2013
    October
    TMPA-2014
    November
    TMPA-2015
    November
    TMPA-2017
    March
    TMPA-2019 |
    November
    TMPA 2019 program included the following topics:
    ● Applications of Computational Topology to Artificial Intelligence;
    ● Petri Nets and Their Extensions;
    ● Static Taint Analysis for JavaScript Programs;
    ● Machine Learning and Value Generation in Software Development;
    ● Generation of Testing Metrics by Using Cluster Analysis of Bug Reports;
    ● Partial Specifications of Libraries: Applications in Software Engineering;
    ● Multi-perspective Process Mining with Embedding Configurations into DB-
    based Event Logs;
    ● Pre-processing Network Messages of Trading Systems into Event Logs for
    Process Mining;
    ● Passive Testing Techniques in Practice;
    ● On DB-nets and Their Applications;
    ● Relation between Test Coverage and Timed Automata Model Structure;
    ● Chaotic Time Series Prediction: Run for the Horizon;
    ● Development of Test Suite with Formally Verified FSM Coverage: A Case Study
    ● Making Bounded Model Checking Interprocedural in (Static Analysis) Style
    TMPA-2021 |TBC
    

    View Slide

15. 15 Build Software to Test Software
    exactpro.com
    Exactpro IEEE publications: during testing, Exactpro uses knowledge and experience obtained over the years and reflected in a few
    papers published by IEEE:
    ● Raising the Quality of Bug Reports by Predicting Software Defect Indicators
    https://ieeexplore.ieee.org/document/885946
    ● Development of Intelligent Virtual Assistant for Software Testing Team
    https://ieeexplore.ieee.org/document/8859438
    ● Poster: ClearTH Test Automation Framework: A Running Example of a DLT-Based Post-Trade System
    https://ieeexplore.ieee.org/abstract/document/8730204
    ● User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications
    https://ieeexplore.ieee.org/abstract/document/8718213
    ● Special features of testing tools applicable for use in trading systems production:
    http://ieeexplore.ieee.org/document/7163719/
    ● Usage of exchange simulators and test exchanges as tools for Ticker Plant systems testing
    http://ieeexplore.ieee.org/document/7163720/
    ● High performance load generator for automated trading systems testing
    http://ieeexplore.ieee.org/document/7163721/
    Latest Exactpro Research and White papers:
    ● https://exactpro.com/ideas/research-papers
    ● https://exactpro.com/ideas/white-papers
    Testing is relentless learning - Exactpro research papers
    

    View Slide

16. 16 Build Software to Test Software
    exactpro.com
    EXACTPRO
    AI-based Systems in Finance
    Machine Learning in financial organizations:
    - already passed an initial development phase
    - the usage of live ML applications is about to
    dramatically increase over the next three years
    ● Market Surveillance Systems
    ● Conversational Assistants
    ● Algo Trading Systems
    ● Pricing Calculators
    ● Machine Readable News
    ● Insurance Claims
    https://www.bankofengland.co.uk/-
    /media/boe/files/report/2019/machine-learning-in-uk-
    financial-services.pdf
    

    View Slide

17. 17 Build Software to Test Software
    exactpro.com
    EXACTPRO
    AI-based Systems in Finance
    

    View Slide

18. 18 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Research Paper
    https://arxiv.org/pdf/1906.10742.pdf
    

    View Slide

19. 19 Build Software to Test Software
    exactpro.com
    EXACTPRO
    List of References
    

    View Slide

20. 20 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Terms Definition
    http://ieeeaitests.com/
    The Second IEEE International Conference On Artificial Intelligence Testing
    3rd-6th August 2020
    Keble College, Oxford University, UK
    Artificial Intelligence vs. Machine Learning
    

    View Slide

21. 21 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Difference between Machine Learning and AI
    https://twitter.com/matvelloso/status/1065778379612282885
    

    View Slide

22. 22 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Difference between Machine Learning and AI
    https://twitter.com/matvelloso
    

    View Slide

23. 23 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Terms Definition
    https://divis.io/en/2019/03/ai-for-laymen-part-2-symbolic-ai-neural-
    networks-and-deep-learning/#gref
    

    View Slide

24. 24 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Terms Definition
    https://divis.io/en/2019/03/ai-for-laymen-part-2-symbolic-ai-neural-
    networks-and-deep-learning/#gref
    

    View Slide

25. 25 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Symbolic AI
    https://youtu.be/GWy4OajL-Dc
    

    View Slide

26. 26 Build Software to Test Software
    exactpro.com
    EXACTPRO
    AI Testing vs. Testing AI
    (ML Testing vs. Testing ML)
    Machine Learning Testing
    Machine Learning-based Testing
    Terms Definition
    http://ieeeaitests.com/
    The Second IEEE International Conference On Artificial Intelligence Testing
    3rd-6th August 2020
    Keble College, Oxford University, UK
    

    View Slide

27. 27 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Machine Learning Testing Publication Stats
    

    View Slide

28. 28 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Machine Learning Testing Publication Stats
    

    View Slide

29. 29 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Research Distribution
    https://www.immuniweb.com/blog/machine-learning-AI-deep-learning-terms.html
    

    View Slide

30. 30 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Research Distribution
    

    View Slide

31. 31 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Structure of the Paper
    

    View Slide

32. 32 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Structure of the Paper
    

    View Slide

33. 33 Build Software to Test Software
    exactpro.com
    EXACTPRO
    How to Test
    

    View Slide

34. 34 Build Software to Test Software
    exactpro.com
    EXACTPRO
    How to Test
    

    View Slide

35. 35 Build Software to Test Software
    exactpro.com
    1
    Larva
    2
    Pupa
    3
    Adult
    Metamorphic Testing
    sin(x) = sin(x + 2)
    sin(x) = sin( - x)
    

    View Slide

36. 36 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Metamorphic Testing
    Donaldson, A. (2018). “Metamorphic Testing for Non-testable Systems.” https://www.testandverification.com/wp-
    content/uploads/2018/VF_2018/Alastair_DonaldsonGraphicsFuzz_ICL.pdf
    

    View Slide

37. 37 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Where to Test
    

    View Slide

38. 38 Build Software to Test Software
    exactpro.com
    EXACTPRO
    What to Test
    

    View Slide

39. 39 Build Software to Test Software
    exactpro.com
    EXACTPRO
    AI Systems Quality Characteristics
    - Ability to learn: The capacity of the system to learn from use for the
    system itself, or data and events it is exposed to.
    - Ability to generalize: The ability of the system to apply to different
    and previously unseen scenarios.
    - Trustworthiness: The degree to which the system is trusted by
    stakeholders, for example a health diagnostic
    A4Q AI and Software Testing
    Foundation
    Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html
    Testing the
    Intelligence
    of your AI
    

    View Slide

40. 40 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Comparison with Software Testing
    

    View Slide

41. 41 Build Software to Test Software
    exactpro.com
    Build Software to Test Software
    

    View Slide

42. 42 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Comparison with Software Testing
    https://www.satisfice.com/blog/archives/4947
    

    View Slide

43. 43 Build Software to Test Software
    exactpro.com
    EXACTPRO
    General Problems with Testing AI Systems
    Non-Deterministic Probabilistic Non-Testable Oracle Problem
    

    View Slide

44. 44 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Build Software to Test Software
    

    View Slide

45. 45 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Structure of the Paper
    

    View Slide

46. 46 Build Software to Test Software
    exactpro.com
    Style-based GANs – Generating and Tuning
    Realistic Artificial Faces:
    https://www.lyrn.ai/2018/12/26/a-style-
    based-generator-architecture-for-generative-
    adversarialnetworks/
    This Person Does Not Exist:
    https://thispersondoesnotexist.com
    Using AI for Test Data Generation
    

    View Slide

47. 47 Build Software to Test Software
    exactpro.com
    Using AI for Test Data Generation
    

    View Slide

48. 48 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Autonomous Driving
    https://www.slideshare.net/taoxiease/intelligent-software-engineering-synergy-between-ai-and-software-engineering-125417110
    

    View Slide

49. 49 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Structure of the Paper
    

    View Slide

50. 50 Build Software to Test Software
    exactpro.com
    No Testing for Conversational Assistants
    Chatbot
    

    View Slide

51. 51 Build Software to Test Software
    exactpro.com
    Public Relations Disaster vs. Research Breakthrough
    

    View Slide

52. 52 Build Software to Test Software
    exactpro.com
    Meena is Unavailable to Talk
    Quote:
    “Also, tackling safety and bias in
    the models is a key focus area for
    us, and given the challenges
    related to this, we are not
    currently releasing an external
    research demo”
    

    View Slide

53. 53 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Autonomous Driving
    https://blog.acolyer.org/2018/06/18/deeptest-automated-testing-of-deep-neural-network-driven-autonomous-cars/
    

    View Slide

54. 54 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Structure of the Paper
    

    View Slide

55. 55 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Data Sets
    

    View Slide

56. 56 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Data Sets
    

    View Slide

57. 57 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Data Sets
    

    View Slide

58. 58 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Data Sets
    

    View Slide

59. 59 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Open Source Tools
    

    View Slide

60. 60 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Open Source Tools
    https://medium.com/artificial-intelligence-ai-for-social-impact/how-might-data-science-students-consider-ethics-d45852e4a83a
    

    View Slide

61. 61 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Further Research
    • Test Input Generation
    • Test Assessment Criteria
    • The Oracle Problem
    • Testing Cost Reduction
    • Testing More Application Scenarios
    • Testing More ML Categories and Tasks
    • Testing Other Properties
    • Presenting More Testing Benchmarks
    • Mutating Investigation in Machine Learning System
    

    View Slide

62. 62 Build Software to Test Software
    exactpro.com
    https://www.youtube.com/c/exactprosystems
    

    View Slide