Тестирование машинного обучения: обзор, пейзажи и горизонты

Тестирование машинного обучения: обзор, пейзажи и горизонты

Иосиф Иткин
Генеральный директор, Exactpro

29 апреля 2020
Online QA Meetup

Иосиф работает над созданием инструментов и методов для тестирования высоконагруженных систем с 2006 года.
На встрече он поделится своими мыслями о недавно опубликованном масштабном исследовании «Тестирование машинного обучения: обзор, пейзажи и горизонты», проведенном учеными из Университетского колледжа Лондона (UCL): https://arxiv.org/pdf/1906.10742.pdf.

Видео: https://youtu.be/u826L6ijvUo

---
Подписывайтесь на Exactpro в социальных сетях:

LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Подписывайтесь на YouTube канал Exactpro https://www.youtube.com/c/exactprosystems

5206c19df417b8876825b5561344c1a0?s=128

Exactpro

April 29, 2020
Tweet

Transcript

  1. 1 Build Software to Test Software exactpro.com Iosif Itkin co-CEO

    and co-founder Exactpro Systems 29 April 2020 Machine Learning Testing: Survey, Landscapes and Horizons
  2. 2 Build Software to Test Software exactpro.com The Subject of

    Today’s Presentation https://arxiv.org/pdf/1906.10742.pdf
  3. 3 Build Software to Test Software exactpro.com The Source of

    Today’s Presentation https://www.eventbrite.com/e/online-talk-intelligent-testing-bringing-transparency-to-ml-in-automation-tickets-103719598110
  4. 4 Build Software to Test Software exactpro.com The Source of

    Today’s Presentation https://www.slideshare.net/HeemengFoo/machine-learning-testing-survey-landscapes-and-horizons-the-cliff-notes
  5. 5 Build Software to Test Software exactpro.com Previous Presentation 25

    February 2020 https://youtu.be/KGVQVBdjvCU
  6. 6 Build Software to Test Software exactpro.com Introduction • Started

    IT career 20 years ago as a software developer • Exactpro co-CEO and co-founder • Moved into performance and load testing in 2006 • Co-founded Exactpro on 9th September 2009 • EXTENT Software Testing and AI conference – 19 November 2020 • Working on th2.net next generation test tool • ISTQB FL 2018, A4Q AI and Software Testing • Build Software to Test Software
  7. 7 Build Software to Test Software exactpro.com EXACTPRO A few

    quick facts about Exactpro • A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories and other market infrastructures. • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 580 specialists worldwide • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG in January 2018. Headquartered in the UK, with operations in the US, Georgia and Russia. • We provide software testing services for mission critical technology that underpins global financial markets. Our clients are regulated by FCA, Bank of England and their counterparts from other countries.
  8. 8 Build Software to Test Software exactpro.com EXACTPRO Our clients

    are the leading exchange groups across the globe
  9. 9 Build Software to Test Software exactpro.com Recent Press Releases

    with Clients and Partners
  10. 10 Build Software to Test Software exactpro.com EXACTPRO Exactpro Partners

    and Cross Industry Work
  11. 11 Build Software to Test Software exactpro.com 11 Build Software

    to Test Software exactpro.com • Platforms • People • Processes • Security
  12. 12 Build Software to Test Software exactpro.com Challenging Non-Functional Requirements

    Daily capacity - 100+ mln transactions Peak rates - 40k+ transactions per second Average round-trip latency - <100 microseconds Availability - 100% 3000 trx 2.5 cm <1 mm
  13. 13 Build Software to Test Software exactpro.com EXACTPRO Indirect Testing

    Methods Information extraction and Machine learning End-to-End Automated Test Library Whatever it takes! Test execution data and log analysis Passive Testing
  14. 14 Build Software to Test Software exactpro.com TMPA-2013 October TMPA-2014

    November TMPA-2015 November TMPA-2017 March TMPA-2019 | November TMPA 2019 program included the following topics: • Applications of Computational Topology to Artificial Intelligence; • Petri Nets and Their Extensions; • Static Taint Analysis for JavaScript Programs; • Machine Learning and Value Generation in Software Development; • Generation of Testing Metrics by Using Cluster Analysis of Bug Reports; • Partial Specifications of Libraries: Applications in Software Engineering; • Multi-perspective Process Mining with Embedding Configurations into DB- based Event Logs; • Pre-processing Network Messages of Trading Systems into Event Logs for Process Mining; • Passive Testing Techniques in Practice; • On DB-nets and Their Applications; • Relation between Test Coverage and Timed Automata Model Structure; • Chaotic Time Series Prediction: Run for the Horizon; • Development of Test Suite with Formally Verified FSM Coverage: A Case Study • Making Bounded Model Checking Interprocedural in (Static Analysis) Style TMPA-2021 |TBC
  15. 15 Build Software to Test Software exactpro.com Exactpro IEEE publications:

    during testing, Exactpro uses knowledge and experience obtained over the years and reflected in a few papers published by IEEE: • Raising the Quality of Bug Reports by Predicting Software Defect Indicators https://ieeexplore.ieee.org/document/885946 • Development of Intelligent Virtual Assistant for Software Testing Team https://ieeexplore.ieee.org/document/8859438 • Poster: ClearTH Test Automation Framework: A Running Example of a DLT-Based Post-Trade System https://ieeexplore.ieee.org/abstract/document/8730204 • User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications https://ieeexplore.ieee.org/abstract/document/8718213 • Special features of testing tools applicable for use in trading systems production: http://ieeexplore.ieee.org/document/7163719/ • Usage of exchange simulators and test exchanges as tools for Ticker Plant systems testing http://ieeexplore.ieee.org/document/7163720/ • High performance load generator for automated trading systems testing http://ieeexplore.ieee.org/document/7163721/ Latest Exactpro Research and White papers: • https://exactpro.com/ideas/research-papers • https://exactpro.com/ideas/white-papers Testing is relentless learning - Exactpro research papers
  16. 16 Build Software to Test Software exactpro.com EXACTPRO AI-based Systems

    in Finance Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years • Market Surveillance Systems • Conversational Assistants • Algo Trading Systems • Pricing Calculators • Machine Readable News • Insurance Claims https://www.bankofengland.co.uk/- /media/boe/files/report/2019/machine-learning-in-uk- financial-services.pdf
  17. 17 Build Software to Test Software exactpro.com EXACTPRO AI-based Systems

    in Finance
  18. 18 Build Software to Test Software exactpro.com EXACTPRO Research Paper

    https://arxiv.org/pdf/1906.10742.pdf
  19. 19 Build Software to Test Software exactpro.com EXACTPRO List of

    References
  20. 20 Build Software to Test Software exactpro.com EXACTPRO Terms Definition

    http://ieeeaitests.com/ The Second IEEE International Conference On Artificial Intelligence Testing 3rd-6th August 2020 Keble College, Oxford University, UK Artificial Intelligence vs. Machine Learning
  21. 21 Build Software to Test Software exactpro.com EXACTPRO Difference between

    Machine Learning and AI https://twitter.com/matvelloso/status/1065778379612282885
  22. 22 Build Software to Test Software exactpro.com EXACTPRO Difference between

    Machine Learning and AI https://twitter.com/matvelloso
  23. 23 Build Software to Test Software exactpro.com EXACTPRO Terms Definition

    https://divis.io/en/2019/03/ai-for-laymen-part-2-symbolic-ai-neural- networks-and-deep-learning/#gref
  24. 24 Build Software to Test Software exactpro.com EXACTPRO Terms Definition

    https://divis.io/en/2019/03/ai-for-laymen-part-2-symbolic-ai-neural- networks-and-deep-learning/#gref
  25. 25 Build Software to Test Software exactpro.com EXACTPRO Symbolic AI

    https://youtu.be/GWy4OajL-Dc
  26. 26 Build Software to Test Software exactpro.com EXACTPRO AI Testing

    vs. Testing AI (ML Testing vs. Testing ML) Machine Learning Testing Machine Learning-based Testing Terms Definition http://ieeeaitests.com/ The Second IEEE International Conference On Artificial Intelligence Testing 3rd-6th August 2020 Keble College, Oxford University, UK
  27. 27 Build Software to Test Software exactpro.com EXACTPRO Machine Learning

    Testing Publication Stats
  28. 28 Build Software to Test Software exactpro.com EXACTPRO Machine Learning

    Testing Publication Stats
  29. 29 Build Software to Test Software exactpro.com EXACTPRO Research Distribution

    https://www.immuniweb.com/blog/machine-learning-AI-deep-learning-terms.html
  30. 30 Build Software to Test Software exactpro.com EXACTPRO Research Distribution

  31. 31 Build Software to Test Software exactpro.com EXACTPRO Structure of

    the Paper
  32. 32 Build Software to Test Software exactpro.com EXACTPRO Structure of

    the Paper
  33. 33 Build Software to Test Software exactpro.com EXACTPRO How to

    Test
  34. 34 Build Software to Test Software exactpro.com EXACTPRO How to

    Test
  35. 35 Build Software to Test Software exactpro.com 1 Larva 2

    Pupa 3 Adult Metamorphic Testing sin(x) = sin(x + 2) sin(x) = sin( - x)
  36. 36 Build Software to Test Software exactpro.com EXACTPRO Metamorphic Testing

    Donaldson, A. (2018). “Metamorphic Testing for Non-testable Systems.” https://www.testandverification.com/wp- content/uploads/2018/VF_2018/Alastair_DonaldsonGraphicsFuzz_ICL.pdf
  37. 37 Build Software to Test Software exactpro.com EXACTPRO Where to

    Test
  38. 38 Build Software to Test Software exactpro.com EXACTPRO What to

    Test
  39. 39 Build Software to Test Software exactpro.com EXACTPRO AI Systems

    Quality Characteristics - Ability to learn: The capacity of the system to learn from use for the system itself, or data and events it is exposed to. - Ability to generalize: The ability of the system to apply to different and previously unseen scenarios. - Trustworthiness: The degree to which the system is trusted by stakeholders, for example a health diagnostic A4Q AI and Software Testing Foundation Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html Testing the Intelligence of your AI
  40. 40 Build Software to Test Software exactpro.com EXACTPRO Comparison with

    Software Testing
  41. 41 Build Software to Test Software exactpro.com Build Software to

    Test Software
  42. 42 Build Software to Test Software exactpro.com EXACTPRO Comparison with

    Software Testing https://www.satisfice.com/blog/archives/4947
  43. 43 Build Software to Test Software exactpro.com EXACTPRO General Problems

    with Testing AI Systems Non-Deterministic Probabilistic Non-Testable Oracle Problem
  44. 44 Build Software to Test Software exactpro.com EXACTPRO Build Software

    to Test Software
  45. 45 Build Software to Test Software exactpro.com EXACTPRO Structure of

    the Paper
  46. 46 Build Software to Test Software exactpro.com Style-based GANs –

    Generating and Tuning Realistic Artificial Faces: https://www.lyrn.ai/2018/12/26/a-style- based-generator-architecture-for-generative- adversarialnetworks/ This Person Does Not Exist: https://thispersondoesnotexist.com Using AI for Test Data Generation
  47. 47 Build Software to Test Software exactpro.com Using AI for

    Test Data Generation
  48. 48 Build Software to Test Software exactpro.com EXACTPRO Autonomous Driving

    https://www.slideshare.net/taoxiease/intelligent-software-engineering-synergy-between-ai-and-software-engineering-125417110
  49. 49 Build Software to Test Software exactpro.com EXACTPRO Structure of

    the Paper
  50. 50 Build Software to Test Software exactpro.com No Testing for

    Conversational Assistants Chatbot
  51. 51 Build Software to Test Software exactpro.com Public Relations Disaster

    vs. Research Breakthrough
  52. 52 Build Software to Test Software exactpro.com Meena is Unavailable

    to Talk Quote: “Also, tackling safety and bias in the models is a key focus area for us, and given the challenges related to this, we are not currently releasing an external research demo”
  53. 53 Build Software to Test Software exactpro.com EXACTPRO Autonomous Driving

    https://blog.acolyer.org/2018/06/18/deeptest-automated-testing-of-deep-neural-network-driven-autonomous-cars/
  54. 54 Build Software to Test Software exactpro.com EXACTPRO Structure of

    the Paper
  55. 55 Build Software to Test Software exactpro.com EXACTPRO Data Sets

  56. 56 Build Software to Test Software exactpro.com EXACTPRO Data Sets

  57. 57 Build Software to Test Software exactpro.com EXACTPRO Data Sets

  58. 58 Build Software to Test Software exactpro.com EXACTPRO Data Sets

  59. 59 Build Software to Test Software exactpro.com EXACTPRO Open Source

    Tools
  60. 60 Build Software to Test Software exactpro.com EXACTPRO Open Source

    Tools https://medium.com/artificial-intelligence-ai-for-social-impact/how-might-data-science-students-consider-ethics-d45852e4a83a
  61. 61 Build Software to Test Software exactpro.com EXACTPRO Further Research

    • Test Input Generation • Test Assessment Criteria • The Oracle Problem • Testing Cost Reduction • Testing More Application Scenarios • Testing More ML Categories and Tasks • Testing Other Properties • Presenting More Testing Benchmarks • Mutating Investigation in Machine Learning System
  62. 62 Build Software to Test Software exactpro.com https://www.youtube.com/c/exactprosystems