Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A4Q AI and Software Testing

5206c19df417b8876825b5561344c1a0?s=47 Exactpro
PRO
February 26, 2020

A4Q AI and Software Testing

A4Q AI and Software Testing

Iosif Itkin
co-CEO & co-founder, Exactpro

QA Meetup
26 February 2020, Saratov

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro Vimeo channel https://vimeo.com/exactpro
Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

5206c19df417b8876825b5561344c1a0?s=128

Exactpro
PRO

February 26, 2020
Tweet

Transcript

  1. 1 Build Software to Test Software exactpro.com Iosif Itkin co-CEO

    and co-founder Exactpro Systems 25 February 2020
  2. 2 Build Software to Test Software exactpro.com Introduction • Started

    IT career 20 years ago as a software developer • Exactpro co-CEO and co-founder • Moved into performance and load testing in 2006 • Co-founded Exactpro on 9th September 2009 • EXTENT Software Testing and AI conference – 23 June 2020 • Working on th2.net next generation test tool • ISTQB FL 2018, A4Q AI and Software Testing • Build Software to Test Software
  3. 3 Build Software to Test Software exactpro.com EXACTPRO Contents •

    A few quick facts about Exactpro • A4Q AI and Software Testing • How is it different from the ISTQB Foundation Level? • What does an exam program consist of? • How difficult is it to prepare for it? • What training materials are there? • Which of these is applicable in practice?
  4. 4 Build Software to Test Software exactpro.com EXACTPRO A few

    quick facts about Exactpro • A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories and other market infrastructures. • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 570 specialists worldwide (130 in Saratov) • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG in January 2018. Headquartered in the UK, with operations in the US and Eastern Europe. • We provide software testing services for mission critical technology that underpins global financial markets. Our clients are regulated by FCA, Bank of England and their counterparts from other countries.
  5. 5 Build Software to Test Software exactpro.com EXACTPRO Our clients

    are the leading exchange groups across the globe
  6. 6 Build Software to Test Software exactpro.com EXACTPRO AI-based Systems

    in Finance Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years • Market Surveillance Systems • Conversational Assistants • Algo Trading Systems • Pricing Calculators • Machine Readable News • Insurance Claims https://www.bankofengland.co.uk/- /media/boe/files/report/2019/machine-learning-in-uk- financial-services.pdf
  7. 7 Build Software to Test Software exactpro.com EXACTPRO Exactpro Partners

    and Cross Industry Work
  8. 8 Build Software to Test Software exactpro.com Since 2012, 155

    specialists have passed the ISTQB® Certified Tester Foundation Level certification exam in Exactpro
  9. 9 Build Software to Test Software exactpro.com https://www.facebook.com/geostqb/ National ISTQB

    Board in Georgia
  10. 10 Build Software to Test Software exactpro.com EXACTPRO How this

    Syllabus is Organized There are three chapters with examinable content. The top-level heading for each chapter specifies the time for the chapter; timing is not provided below chapter level. For the A4Q AI and Software Testing Foundation training course, the syllabus requires a minimum of 17 hours, 10 minutes of instruction, distributed across the three chapters as follows: Chapter 1: AI and Software Testing Background 360 minutes Chapter 2: Testing Artificial Intelligence Systems 400 minutes Chapter 3: Using AI to Support Testing 270 minutes https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html
  11. 11 Build Software to Test Software exactpro.com EXACTPRO How this

    Syllabus is Organized There are six chapters with examinable content. The top-level heading for each chapter specifies the time for the chapter; timing is not provided below chapter level. For accredited training courses, the syllabus requires a minimum of 16.75 hours of instruction, distributed across the six chapters as follows: Chapter 1: 175 minutes Fundamentals of Testing Chapter 2: 100 minutes Testing Throughout the Software Development Lifecycle Chapter 3: 135 minutes Static Testing Chapter 4: 330 minutes Test Techniques Chapter 5: 225 minutes Test Management Chapter 6: 40 minutes Tool Support for Testing
  12. 12 Build Software to Test Software exactpro.com EXACTPRO Similarities •

    40 Multi-choice Questions • 26 Correct Answers to Pass • 60 Minutes (you can request 15 more minutes) • GASQ (160 EUR) or iSQI (Pearson VUE 175 EUR) in Russia • 40 Pages in AIST vs. 64 in ISTQB
  13. 13 Build Software to Test Software exactpro.com EXACTPRO Difference

  14. 14 Build Software to Test Software exactpro.com EXACTPRO Mock Exam

    https://www.gasq.org/files/content/gasq/downloads/certification/A4Q%20AI%20&%20Software%20Testing/A4Q_AI-and-SW-Testing_MockExam_v1.0.pdf
  15. 15 Build Software to Test Software exactpro.com EXACTPRO Syllabus https://www.gasq.org/files/content/gasq/downloads/certification/A4Q%20AI%20&%20Software%20Testing/AI_Software_Testing_Syllabus%20(1.0).pdf

  16. 16 Build Software to Test Software exactpro.com EXACTPRO Key Aspects

    of Artificial Intelligence 1.1 What are Human Intelligence and Artificial Intelligence? 1.2 History of AI 1.3 Symbolic AI 1.4 Sub-symbolic AI 1.5 Some ML Algorithms in More Detail • Bayesian Belief Networks • Naïve Bayes classifier • Support Vector Machine Algorithm • K-means Algorithm • Artificial Neural Networks: Perceptron Learning Algorithm 1.6 Applications and Limits of AI
  17. 17 Build Software to Test Software exactpro.com EXACTPRO 1.1 What

    are Human Intelligence and Artificial Intelligence? Types of Intelligence
  18. 18 Build Software to Test Software exactpro.com EXACTPRO 1.1 What

    are Human Intelligence and Artificial Intelligence? Turing Test https://www.pandorabots.com/mitsuku/
  19. 19 Build Software to Test Software exactpro.com EXACTPRO 1.2 History

    of AI
  20. 20 Build Software to Test Software exactpro.com EXACTPRO 1.3 Symbolic

    AI https://youtu.be/GWy4OajL-Dc
  21. 21 Build Software to Test Software exactpro.com EXACTPRO 1.3 Symbolic

    AI https://tmpaconf.org/ru/events/tmpa-school-2018/agenda-school-2018-ru
  22. 22 Build Software to Test Software exactpro.com EXACTPRO 1.3 Symbolic

    AI
  23. 23 Build Software to Test Software exactpro.com EXACTPRO 1.4 Sub-symbolic

    AI
  24. 24 Build Software to Test Software exactpro.com EXACTPRO 1.4 Sub-symbolic

    AI https://www.immuniweb.com/blog/machine-learning-AI-deep-learning-terms.html
  25. 25 Build Software to Test Software exactpro.com EXACTPRO

  26. 26 Build Software to Test Software exactpro.com EXACTPRO 1.5 Some

    ML Algorithms in More Detail https://www.coursera.org/learn/vvedenie-mashinnoe-obuchenie Краткая программа курса: Неделя 1. Введение. Примеры задач. Логические методы: решающие деревья и решающие леса. Неделя 2. Метрические методы классификации. Линейные методы, стохастический градиент. Неделя 3. Метод опорных векторов (SVM). Логистическая регрессия. Метрики качества классификации. Неделя 4. Линейная регрессия. Понижение размерности, метод главных компонент. Неделя 5. Композиции алгоритмов, градиентный бустинг. Нейронные сети. Неделя 6. Кластеризация и визуализация. Частичное обучение. Неделя 7. Прикладные задачи анализа данных: постановки и методы решения.
  27. 27 Build Software to Test Software exactpro.com EXACTPRO 1.5 Some

    ML Algorithms in More Detail Naïve Bayes classifier https://youtu.be/Q8l0Vip5YUw https://youtu.be/z5UQyCESW64 Andrew Ng:
  28. 28 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications

    and Limits of AI Activities of Machine Learning https://the-modeling-agency.com/crisp-dm.pdf
  29. 29 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications

    and Limits of AI Possible Biases in AI Systems https://schedule.sxsw.com/2019/events/PP83596 https://www.mckinsey.com/featured-insights/artificial-intelligence/tackling- bias-in-artificial-intelligence-and-in-humans
  30. 30 Build Software to Test Software exactpro.com https://youtu.be/ki2fsH7Il8E 29th January

    2020
  31. 31 Build Software to Test Software exactpro.com https://rbcs-us.com/site/assets/files/3793/a4q_ai_and_software_tester_foundation_course_outline_v0_9.pdf

  32. 32 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications

    and Limits of AI Possible Biases in AI Systems
  33. 33 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications

    and Limits of AI Ethical Issues in AI
  34. 34 Build Software to Test Software exactpro.com EXACTPRO 2.1 General

    Problems with Testing AI Systems Non-Deterministic Probabilistic Non-Testable Oracle Problem
  35. 35 Build Software to Test Software exactpro.com Challenging Non-Functional Requirements

    Daily capacity - 100+ mln transactions Peak rates - 40k+ transactions per second Average round-trip latency - <100 microseconds Availability - 100% 3000 trx 2.5 cm <1 mm
  36. 36 Build Software to Test Software exactpro.com EXACTPRO Build Software

    to Test Software Non-Deterministic Probabilistic Non-Testable Oracle Problem
  37. 37 Build Software to Test Software exactpro.com EXACTPRO Indirect Testing

    Methods Information extraction and Machine learning End-to-End Automated Test Library Whatever it takes! Test execution data and log analysis Passive Testing
  38. 38 Build Software to Test Software exactpro.com EXACTPRO Indirect Testing

    Methods Donaldson, A. (2018). “Metamorphic Testing for Non-testable Systems.” https://www.testandverification.com/wp- content/uploads/2018/VF_2018/Alastair_DonaldsonGraphicsFuzz_ICL.pdf
  39. 39 Build Software to Test Software exactpro.com EXACTPRO 2.2 Machine

    Learning Model Training and Testing https://www.deeplearning.ai/ • Training set — Which you run your learning algorithm on. • Development set — Which you use to tune parameters, select features, and make other decisions regarding the learning algorithm. Sometimes also called the hold-out cross validation set. • Test set — which you use to evaluate the performance of the algorithm, but not to make any decisions regarding what learning algorithm or parameters to use.
  40. 40 Build Software to Test Software exactpro.com EXACTPRO 2.3 AI

    Test Environments By Wbjimmyd - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?c urid=30859962 https://en.wikibooks.org/wiki/Artificial_Intelligence/AI_Agents_and_their_Environments
  41. 41 Build Software to Test Software exactpro.com EXACTPRO AI Systems

    Quality Characteristics - Ability to learn: The capacity of the system to learn from use for the system itself, or data and events it is exposed to. - Ability to generalize: The ability of the system to apply to different and previously unseen scenarios. - Trustworthiness: The degree to which the system is trusted by stakeholders, for example a health diagnostic A4Q AI and Software Testing Foundation Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html Testing the Intelligence of your AI
  42. 42 Build Software to Test Software exactpro.com EXACTPRO Ability to

    Generalize: Scope of End-to-End and Negative Testing
  43. 43 Build Software to Test Software exactpro.com EXACTPRO Using AI

    to Support Testing 3.1 AI in Testing 3.2 Applying AI to Testing Tasks and Quality Management 3.3 AI in Component Level Test Automation 3.4 AI in Integration Level or System Level Test Automation 3.5 AI-based Tool Support for Testing
  44. 44 Build Software to Test Software exactpro.com EXACTPRO 3.1 AI

    in Testing Oracle Problem
  45. 45 Build Software to Test Software exactpro.com EXACTPRO Tasks AI

    Can Be Applied To
  46. 46 Build Software to Test Software exactpro.com EXACTPRO Tasks AI

    Cannot Be Applied To
  47. 47 Build Software to Test Software exactpro.com EXACTPRO Tasks AI

    Cannot Be Applied To https://www.istqb.org/documents/ISTQB_The_Vision_on_the_Future_of_Software_Testing_Final.pdf
  48. 48 Build Software to Test Software exactpro.com Style-based GANs –

    Generating and Tuning Realistic Artificial Faces: https://www.lyrn.ai/2018/12/26/a-style- based-generator-architecture-for-generative- adversarialnetworks/ This Person Does Not Exist: https://thispersondoesnotexist.com Using AI for Test Data Generation
  49. 49 Build Software to Test Software exactpro.com Using AI for

    Test Data Generation
  50. 50 Build Software to Test Software exactpro.com 3.2 Applying AI

    to Testing Tasks and Quality Management BEFORE Extremely high memory consumption was observed in Prod system with release 1.2.3.4.5.6 installed during ITR and ITCH gateways crashed. AFTER Extremely high memory consumption was observed in Prod system with release 1.2.3.4.5.6 installed during ITR. All ITCH gateways consumed up to *** GB of RAM. They crashed, except for three pairs of ITCH gateways. Probably this issue can be related to issue #1234567. Backend logs, data files, core files, DB dumps are attached. Won’t Fix Pie Chart Won’t Fix Fix
  51. 51 Build Software to Test Software exactpro.com Supervised Learning. Sailfish

    Machine Learning Feature engineering, dimensionality reduction, training & evaluation of the model Predictions and tips Cradle Data Warehousing Fetching raw data from tools under test. Processing it into datasets with Data Mart & BI support Test Execution and Analysis Predictions and tips for failures in order to leverage the smoothness and accuracy of the testing outputs analysis QA Analyst Fewer feedback loops, less rework, higher quality, faster turnaround, better alignment of roles on a project
  52. 52 Build Software to Test Software exactpro.com https://unsplash.com/search/photos/san-francisco The First

    IEEE International Conference on Artificial Intelligence Testing (IEEE AITest 2019), 4-9 April 2019, San Francisco East Bay, CA, USA. User-Assisted Log Analysis for Quality Control of Distributed Fintech Systems Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav Yavorskiy, Evgenii Tsymbalov, Andrey Novikov and Kirill Rudakov. IEEE International Conference on Artificial Intelligence Testing San Francisco East Bay, USA http://ieeeaitests.com/
  53. 53 Build Software to Test Software exactpro.com 3.3 AI in

    Component Level Test Automation
  54. 54 Build Software to Test Software exactpro.com https://www.aitesting.org/ https://medium.com/@jarbon

  55. 55 Build Software to Test Software exactpro.com

  56. 56 Build Software to Test Software exactpro.com EXACTPRO 3.5 AI-based

    Tool Support for Testing • Configuration of the System • Return on Investment (ROI) • Effects on Existing Processes • Sensibility of Test Cases • Test Case Explosion • Maintainability • Severity of the Defects Found
  57. 57 Build Software to Test Software exactpro.com EXACTPRO https://isqi.org/lp/ai-software-testing-certification/

  58. 58 Build Software to Test Software exactpro.com 13th – 16th

    April 2020 in Keble College, Oxford University, UK
  59. 59 Build Software to Test Software exactpro.com https://www.youtube.com/c/exactprosystems