Тестирование машинного обучения: обзор, пейзажи и горизонты

Иосиф Иткин
Генеральный директор, Exactpro

29 апреля 2020
Online QA Meetup

Иосиф работает над созданием инструментов и методов для тестирования высоконагруженных систем с 2006 года.
На встрече он поделится своими мыслями о недавно опубликованном масштабном исследовании «Тестирование машинного обучения: обзор, пейзажи и горизонты», проведенном учеными из Университетского колледжа Лондона (UCL): https://arxiv.org/pdf/1906.10742.pdf.

Видео: https://youtu.be/u826L6ijvUo

---
Подписывайтесь на Exactpro в социальных сетях:

LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Подписывайтесь на YouTube канал Exactpro https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software exactpro.com Iosif Itkin co-CEO and co-founder Exactpro Systems 29 April 2020 Machine Learning Testing: Survey, Landscapes and Horizons
2. 2 Build Software to Test Software exactpro.com The Subject of Today’s Presentation https://arxiv.org/pdf/1906.10742.pdf
3. 3 Build Software to Test Software exactpro.com The Source of Today’s Presentation https://www.eventbrite.com/e/online-talk-intelligent-testing-bringing-transparency-to-ml-in-automation-tickets-103719598110
4. 4 Build Software to Test Software exactpro.com The Source of Today’s Presentation https://www.slideshare.net/HeemengFoo/machine-learning-testing-survey-landscapes-and-horizons-the-cliff-notes
5. 5 Build Software to Test Software exactpro.com Previous Presentation 25 February 2020 https://youtu.be/KGVQVBdjvCU
6. 6 Build Software to Test Software exactpro.com Introduction • Started IT career 20 years ago as a software developer • Exactpro co-CEO and co-founder • Moved into performance and load testing in 2006 • Co-founded Exactpro on 9th September 2009 • EXTENT Software Testing and AI conference – 19 November 2020 • Working on th2.net next generation test tool • ISTQB FL 2018, A4Q AI and Software Testing • Build Software to Test Software
7. 7 Build Software to Test Software exactpro.com EXACTPRO A few quick facts about Exactpro • A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories and other market infrastructures. • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 580 specialists worldwide • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG in January 2018. Headquartered in the UK, with operations in the US, Georgia and Russia. • We provide software testing services for mission critical technology that underpins global financial markets. Our clients are regulated by FCA, Bank of England and their counterparts from other countries.
8. 8 Build Software to Test Software exactpro.com EXACTPRO Our clients are the leading exchange groups across the globe
9. 9 Build Software to Test Software exactpro.com Recent Press Releases with Clients and Partners
10. 10 Build Software to Test Software exactpro.com EXACTPRO Exactpro Partners and Cross Industry Work
11. 11 Build Software to Test Software exactpro.com 11 Build Software to Test Software exactpro.com • Platforms • People • Processes • Security
12. 12 Build Software to Test Software exactpro.com Challenging Non-Functional Requirements Daily capacity - 100+ mln transactions Peak rates - 40k+ transactions per second Average round-trip latency - <100 microseconds Availability - 100% 3000 trx 2.5 cm <1 mm
13. 13 Build Software to Test Software exactpro.com EXACTPRO Indirect Testing Methods Information extraction and Machine learning End-to-End Automated Test Library Whatever it takes! Test execution data and log analysis Passive Testing
14. 14 Build Software to Test Software exactpro.com TMPA-2013 October TMPA-2014 November TMPA-2015 November TMPA-2017 March TMPA-2019 | November TMPA 2019 program included the following topics: • Applications of Computational Topology to Artificial Intelligence; • Petri Nets and Their Extensions; • Static Taint Analysis for JavaScript Programs; • Machine Learning and Value Generation in Software Development; • Generation of Testing Metrics by Using Cluster Analysis of Bug Reports; • Partial Specifications of Libraries: Applications in Software Engineering; • Multi-perspective Process Mining with Embedding Configurations into DB- based Event Logs; • Pre-processing Network Messages of Trading Systems into Event Logs for Process Mining; • Passive Testing Techniques in Practice; • On DB-nets and Their Applications; • Relation between Test Coverage and Timed Automata Model Structure; • Chaotic Time Series Prediction: Run for the Horizon; • Development of Test Suite with Formally Verified FSM Coverage: A Case Study • Making Bounded Model Checking Interprocedural in (Static Analysis) Style TMPA-2021 |TBC
15. 15 Build Software to Test Software exactpro.com Exactpro IEEE publications: during testing, Exactpro uses knowledge and experience obtained over the years and reflected in a few papers published by IEEE: • Raising the Quality of Bug Reports by Predicting Software Defect Indicators https://ieeexplore.ieee.org/document/885946 • Development of Intelligent Virtual Assistant for Software Testing Team https://ieeexplore.ieee.org/document/8859438 • Poster: ClearTH Test Automation Framework: A Running Example of a DLT-Based Post-Trade System https://ieeexplore.ieee.org/abstract/document/8730204 • User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications https://ieeexplore.ieee.org/abstract/document/8718213 • Special features of testing tools applicable for use in trading systems production: http://ieeexplore.ieee.org/document/7163719/ • Usage of exchange simulators and test exchanges as tools for Ticker Plant systems testing http://ieeexplore.ieee.org/document/7163720/ • High performance load generator for automated trading systems testing http://ieeexplore.ieee.org/document/7163721/ Latest Exactpro Research and White papers: • https://exactpro.com/ideas/research-papers • https://exactpro.com/ideas/white-papers Testing is relentless learning - Exactpro research papers
16. 16 Build Software to Test Software exactpro.com EXACTPRO AI-based Systems in Finance Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years • Market Surveillance Systems • Conversational Assistants • Algo Trading Systems • Pricing Calculators • Machine Readable News • Insurance Claims https://www.bankofengland.co.uk/- /media/boe/files/report/2019/machine-learning-in-uk- financial-services.pdf
17. 17 Build Software to Test Software exactpro.com EXACTPRO AI-based Systems in Finance
18. 18 Build Software to Test Software exactpro.com EXACTPRO Research Paper https://arxiv.org/pdf/1906.10742.pdf
19. 19 Build Software to Test Software exactpro.com EXACTPRO List of References
20. 20 Build Software to Test Software exactpro.com EXACTPRO Terms Definition http://ieeeaitests.com/ The Second IEEE International Conference On Artificial Intelligence Testing 3rd-6th August 2020 Keble College, Oxford University, UK Artificial Intelligence vs. Machine Learning
21. 21 Build Software to Test Software exactpro.com EXACTPRO Difference between Machine Learning and AI https://twitter.com/matvelloso/status/1065778379612282885
22. 22 Build Software to Test Software exactpro.com EXACTPRO Difference between Machine Learning and AI https://twitter.com/matvelloso
23. 23 Build Software to Test Software exactpro.com EXACTPRO Terms Definition https://divis.io/en/2019/03/ai-for-laymen-part-2-symbolic-ai-neural- networks-and-deep-learning/#gref
24. 24 Build Software to Test Software exactpro.com EXACTPRO Terms Definition https://divis.io/en/2019/03/ai-for-laymen-part-2-symbolic-ai-neural- networks-and-deep-learning/#gref
25. 25 Build Software to Test Software exactpro.com EXACTPRO Symbolic AI https://youtu.be/GWy4OajL-Dc
26. 26 Build Software to Test Software exactpro.com EXACTPRO AI Testing vs. Testing AI (ML Testing vs. Testing ML) Machine Learning Testing Machine Learning-based Testing Terms Definition http://ieeeaitests.com/ The Second IEEE International Conference On Artificial Intelligence Testing 3rd-6th August 2020 Keble College, Oxford University, UK
27. 27 Build Software to Test Software exactpro.com EXACTPRO Machine Learning Testing Publication Stats
28. 28 Build Software to Test Software exactpro.com EXACTPRO Machine Learning Testing Publication Stats
29. 29 Build Software to Test Software exactpro.com EXACTPRO Research Distribution https://www.immuniweb.com/blog/machine-learning-AI-deep-learning-terms.html
30. 30 Build Software to Test Software exactpro.com EXACTPRO Research Distribution
31. 31 Build Software to Test Software exactpro.com EXACTPRO Structure of the Paper
32. 32 Build Software to Test Software exactpro.com EXACTPRO Structure of the Paper
33. 33 Build Software to Test Software exactpro.com EXACTPRO How to Test
34. 34 Build Software to Test Software exactpro.com EXACTPRO How to Test
35. 35 Build Software to Test Software exactpro.com 1 Larva 2 Pupa 3 Adult Metamorphic Testing sin(x) = sin(x + 2) sin(x) = sin( - x)
36. 36 Build Software to Test Software exactpro.com EXACTPRO Metamorphic Testing Donaldson, A. (2018). “Metamorphic Testing for Non-testable Systems.” https://www.testandverification.com/wp- content/uploads/2018/VF_2018/Alastair_DonaldsonGraphicsFuzz_ICL.pdf
37. 37 Build Software to Test Software exactpro.com EXACTPRO Where to Test
38. 38 Build Software to Test Software exactpro.com EXACTPRO What to Test
39. 39 Build Software to Test Software exactpro.com EXACTPRO AI Systems Quality Characteristics - Ability to learn: The capacity of the system to learn from use for the system itself, or data and events it is exposed to. - Ability to generalize: The ability of the system to apply to different and previously unseen scenarios. - Trustworthiness: The degree to which the system is trusted by stakeholders, for example a health diagnostic A4Q AI and Software Testing Foundation Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html Testing the Intelligence of your AI
40. 40 Build Software to Test Software exactpro.com EXACTPRO Comparison with Software Testing
41. 41 Build Software to Test Software exactpro.com Build Software to Test Software
42. 42 Build Software to Test Software exactpro.com EXACTPRO Comparison with Software Testing https://www.satisfice.com/blog/archives/4947
43. 43 Build Software to Test Software exactpro.com EXACTPRO General Problems with Testing AI Systems Non-Deterministic Probabilistic Non-Testable Oracle Problem
44. 44 Build Software to Test Software exactpro.com EXACTPRO Build Software to Test Software
45. 45 Build Software to Test Software exactpro.com EXACTPRO Structure of the Paper
46. 46 Build Software to Test Software exactpro.com Style-based GANs – Generating and Tuning Realistic Artificial Faces: https://www.lyrn.ai/2018/12/26/a-style- based-generator-architecture-for-generative- adversarialnetworks/ This Person Does Not Exist: https://thispersondoesnotexist.com Using AI for Test Data Generation
47. 47 Build Software to Test Software exactpro.com Using AI for Test Data Generation
48. 48 Build Software to Test Software exactpro.com EXACTPRO Autonomous Driving https://www.slideshare.net/taoxiease/intelligent-software-engineering-synergy-between-ai-and-software-engineering-125417110
49. 49 Build Software to Test Software exactpro.com EXACTPRO Structure of the Paper
50. 50 Build Software to Test Software exactpro.com No Testing for Conversational Assistants Chatbot
51. 51 Build Software to Test Software exactpro.com Public Relations Disaster vs. Research Breakthrough
52. 52 Build Software to Test Software exactpro.com Meena is Unavailable to Talk Quote: “Also, tackling safety and bias in the models is a key focus area for us, and given the challenges related to this, we are not currently releasing an external research demo”
53. 53 Build Software to Test Software exactpro.com EXACTPRO Autonomous Driving https://blog.acolyer.org/2018/06/18/deeptest-automated-testing-of-deep-neural-network-driven-autonomous-cars/
54. 54 Build Software to Test Software exactpro.com EXACTPRO Structure of the Paper
55. 55 Build Software to Test Software exactpro.com EXACTPRO Data Sets
56. 56 Build Software to Test Software exactpro.com EXACTPRO Data Sets
57. 57 Build Software to Test Software exactpro.com EXACTPRO Data Sets
58. 58 Build Software to Test Software exactpro.com EXACTPRO Data Sets
59. 59 Build Software to Test Software exactpro.com EXACTPRO Open Source Tools
60. 60 Build Software to Test Software exactpro.com EXACTPRO Open Source Tools https://medium.com/artificial-intelligence-ai-for-social-impact/how-might-data-science-students-consider-ethics-d45852e4a83a
61. 61 Build Software to Test Software exactpro.com EXACTPRO Further Research • Test Input Generation • Test Assessment Criteria • The Oracle Problem • Testing Cost Reduction • Testing More Application Scenarios • Testing More ML Categories and Tasks • Testing Other Properties • Presenting More Testing Benchmarks • Mutating Investigation in Machine Learning System
62. 62 Build Software to Test Software exactpro.com https://www.youtube.com/c/exactprosystems