Использование машинного обучения при тестировании веб приложений

Использование машинного обучения при тестировании веб приложений

Иосиф Иткин, Exactpro

8 мая 2020

Онлайн-трансляция: https://youtu.be/AKDR1H6ORmQ.

--
Хотите знать больше об Exactpro?

Website: https://ru.exactpro.com/

Exactpro в социальных сетях:
VK - https://vk.com/exactpro_llc
Facebook - https://www.facebook.com/exactpro/
Twitter - https://twitter.com/exactpro
LinkedIn - https://www.linkedin.com/company/exactpro-systems-llc
Instagram - https://www.instagram.com/exactpro/
GitHub - https://github.com/Exactpro
Youtube - https://www.youtube.com/c/exactprosystems
Exactpro Vlog https://www.youtube.com/c/ExactproVlog

5206c19df417b8876825b5561344c1a0?s=128

Exactpro

May 08, 2020
Tweet

Transcript

  1. 1 Build Software to Test Software exactpro.com Click to edit

    the title text format
  2. 2 Build Software to Test Software exactpro.com Содержание • Вступительное

    слово • Полезные материалы • Веб приложение с точки зрения человека • Тестирование, автоматизация тестирования и машинное обучение • Человек с точки зрения веб приложения • Использование Selenium • Проблема оракула в тестировании • Применение машинного обучения в тестирование DOM • Применение машинного обучения в визуальном тестировании
  3. 3 Build Software to Test Software exactpro.com О себе •

    Начал работать разработчиком 20 лет назад • Перешел в нагрузочное тестирование в 2006 • Со-основал Exactpro 9 сентября 2009 • ISTQB FL 2018, A4Q AI and Software Testing • Build Software to Test Software
  4. 4 Build Software to Test Software exactpro.com Об университете -

    http://fast-ksu.ru/
  5. 5 Build Software to Test Software exactpro.com Более 100 выпускников

    КГУ успешно получили Сертификат ISTQB® Certified Tester Foundation Level
  6. 6 Build Software to Test Software exactpro.com TMPA-2013 October Kostroma

    TMPA-2014 November Kostroma TMPA-2015 November St. Petersburg TMPA-2017 March Moscow TMPA 2019 program included the following topics: • Applications of Computational Topology to Artificial Intelligence; • Petri Nets and Their Extensions; • Static Taint Analysis for JavaScript Programs; • Machine Learning and Value Generation in Software Development; • Generation of Testing Metrics by Using Cluster Analysis of Bug Reports; • Partial Specifications of Libraries: Applications in Software Engineering; • Multi-perspective Process Mining with Embedding Configurations into DB- based Event Logs; • Pre-processing Network Messages of Trading Systems into Event Logs for Process Mining; • Passive Testing Techniques in Practice; • On DB-nets and Their Applications; • Relation between Test Coverage and Timed Automata Model Structure; • Chaotic Time Series Prediction: Run for the Horizon; • Development of Test Suite with Formally Verified FSM Coverage: A Case Study • Making Bounded Model Checking Interprocedural in (Static Analysis) Style TMPA-2021 |TBC TMPA-2019 November Tbilisi
  7. 7 Build Software to Test Software exactpro.com A specialist firm

    focused on functional and non-functional testing of exchanges, clearing houses, depositories, trade repositories and other financial market infrastructures. О компании Headquartered in the UK with operations in the US and Eastern Europe. The clients are regulated by the FCA, Bank of England and their counterparts from other countries. Exactpro provides software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing and settlement platforms, market data systems, collateral management and risk management systems, central data warehouses, regulatory reporting, etc. Incorporated in 2009 with 10 people, Exactpro is now employing over 580 specialists. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG.
  8. 8 Build Software to Test Software exactpro.com Te st Li

    br ar y Высоконагруженные биржевые системы
  9. 9 Build Software to Test Software exactpro.com Недавние публикации

  10. 10 Build Software to Test Software exactpro.com Партнеры и объединения

  11. 11 Build Software to Test Software exactpro.com EXACTPRO Non-Deterministic Probabilistic

    Non-Testable Oracle Problem Высоконагруженные биржевые системы
  12. 12 Build Software to Test Software exactpro.com Daily capacity -

    100+ mln transactions Peak rates - 40k+ transactions per second Average round-trip latency - <100 microseconds Availability - 100% 3000 trx 2.5 cm <1 mm Высоконагруженные биржевые системы
  13. 13 Build Software to Test Software exactpro.com EXACTPRO Полезные материалы

    https://frontendmasters.com/books/front-end-handbook/2019/
  14. 14 Build Software to Test Software exactpro.com EXACTPRO Полезные материалы

    https://youtu.be/nvSqTlzivNM https://www.slideshare.net/roesslerj/how-to-apply-ai-to-testing
  15. 15 Build Software to Test Software exactpro.com Полезные материалы 25

    February 2020 https://youtu.be/KGVQVBdjvCU
  16. 16 Build Software to Test Software exactpro.com EXACTPRO Полезные материалы

    https://www.gasq.org/en/certification/selenium-tester.html
  17. 17 Build Software to Test Software exactpro.com EXACTPRO Полезные материалы

    https://www.youtube.com/watch?v=S9iytpte6OA https://www.testim.io/
  18. 18 Build Software to Test Software exactpro.com EXACTPRO Полезные материалы

    https://testautomationu.applitools.com/ai-for-element-selection-erasing-the-pain-of-fragile-test-scripts
  19. 19 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. HTTP(S) Request HTTP(S) Response 1 2 CSS HTML Javascript Internet Web Servers Static Content Browser DOM model
  20. 20 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. HTTP(S) Request HTTP(S) Response 1 2 CSS HTML Javascript Internet Application Servers Data Web Servers Static Content Browser DOM model
  21. 21 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. 1 Client-side scripts Run in the browser and process information without sending requests to the server 3 2 CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Internet
  22. 22 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. 1 Client-side scripts Run in the browser and process information without sending requests to the server 3 AJAX allows to create asynchronous web interactions without interfering with the display and behaviour of the existing page 4 2 CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Internet
  23. 23 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page WebSocket enables interaction between a web browser and a web server with lower overhead than half-duplex alternatives such as HTTP polling Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. 1 Client-side scripts Run in the browser and process information without sending requests to the server 3 AJAX allows to create asynchronous web interactions without interfering with the display and behaviour of the existing page 4 2 5 CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet
  24. 24 Build Software to Test Software exactpro.com EXACTPRO День Земли

    The Earth does not belong to us, we belong to the Earth.
  25. 25 Build Software to Test Software exactpro.com EXACTPRO День Земли

    The Earth does not belong to us, we belong to the Earth.
  26. 26 Build Software to Test Software exactpro.com EXACTPRO День Земли

    The Earth does not belong to us, we belong to the Earth.
  27. 27 Build Software to Test Software exactpro.com EXACTPRO День Земли

    The Earth does not belong to us, we belong to the Earth.
  28. 28 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page WebSocket enables interaction between a web browser and a web server with lower overhead than half-duplex alternatives such as HTTP polling Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. 1 Client-side scripts Run in the browser and process information without sending requests to the server 3 AJAX allows to create asynchronous web interactions without interfering with the display and behaviour of the existing page 4 2 5 CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet
  29. 29 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page WebSocket enables interaction between a web browser and a web server with lower overhead than half-duplex alternatives such as HTTP polling Responsive front-end design allows a site to adapt to user’s device. Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. 1 Client-side scripts Run in the browser and process information without sending requests to the server 3 AJAX allows to create asynchronous web interactions without interfering with the display and behaviour of the existing page 4 2 5 CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet
  30. 30 Build Software to Test Software exactpro.com EXACTPRO Структура веб

    приложения Browser send requests to the server, receive responses and visualize them as a page WebSocket enables interaction between a web browser and a web server with lower overhead than half-duplex alternatives such as HTTP polling Responsive front-end design allows a site to adapt to user’s device. Everything a user sees in the browser is a mix of HTML, CSS, and JavaScript. The back-end server-side scripts process the request, pull what they need from the database and application servers and then send it back. 1 Client-side scripts Run in the browser and process information without sending requests to the server 3 AJAX allows to create asynchronous web interactions without interfering with the display and behaviour of the existing page 4 2 5 CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet Web applications are always multi-user applications and many of them work under heavy load and are subject to concurrency access 6
  31. 31 Build Software to Test Software exactpro.com EXACTPRO Тестирование Качество

    - это соответствие характеристик ожиданиям Тестирование - это систематическое исследование ПО с целью нахождения дефектов Тестирование - это процесс и набор техник Ошибка, дефект, поломка
  32. 32 Build Software to Test Software exactpro.com EXACTPRO Уровни тестирования

    CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet A C D E F B
  33. 33 Build Software to Test Software exactpro.com EXACTPRO Автоматизация Автоматизация

    — одно из направлений научно-технического прогресса, использующее саморегулирующие технические средства и математические методы с целью освобождения человека от участия в процессах получения, преобразования, передачи и использования энергии, материалов или информации, либо существенного уменьшения степени этого участия или трудоёмкости выполняемых операций «Первое правило любой технологии, используемой в бизнесе, заключается в том, что автоматизация, применяемая к эффективному процессу, увеличивает эффективность. Второе правило, автоматизация, примененная к неэффективной процессу, увеличивает неэффективность.» Bill Gates
  34. 34 Build Software to Test Software exactpro.com EXACTPRO Модель человека

    для веб приложения
  35. 35 Build Software to Test Software exactpro.com EXACTPRO Модель человека

    для веб приложения
  36. 36 Build Software to Test Software exactpro.com EXACTPRO Модель человека

    для веб приложения
  37. 37 Build Software to Test Software exactpro.com EXACTPRO DOM модель

  38. 38 Build Software to Test Software exactpro.com EXACTPRO DOM модель

  39. 39 Build Software to Test Software exactpro.com EXACTPRO DOM модель

  40. 40 Build Software to Test Software exactpro.com EXACTPRO DOM модель

    Document <HTML> <head> <body> <h1> Root element Element “The Header” <h1> “the link” “href” “Sub Header” <h2> “The title” <title> Element Element Element Element Text Attribute Text Text Element Text The DOM defines: All HTML elements as objects • The properties of all HTML elements • The methods that can be used to access all HTML elements • The events that affect all HTML elements "The W3C Document Object Model (DOM) is a platform and language-neutral interface that allows programs and scripts to dynamically access and update the content, structure, and style of a document."
  41. 41 Build Software to Test Software exactpro.com EXACTPRO DOM модель

  42. 42 Build Software to Test Software exactpro.com EXACTPRO DOM модель

    улица машина машина переход человек светофор дорога дом дерево тротуар
  43. 43 Build Software to Test Software exactpro.com EXACTPRO DOM модель

    улица тротуар дерево переход машина дом дорога пешеход город машина машина листик
  44. 44 Build Software to Test Software exactpro.com EXACTPRO The Generic

    Test Automation Architecture https://github.com/YuriSilenok/SeleniumDemo
  45. 45 Build Software to Test Software exactpro.com EXACTPRO The Generic

    Test Automation Architecture Advantages: • You can find elements that don't have any unique attributes (id, class, name, etc.) • You can use XPath in generic locators, using the different "By" strategies (by id, by class, etc.) as needed Disadvantages: • Absolute XPath code is "brittle" and can break with the smallest change to the HTML structure • Relative XPath code can find the wrong node if the attribute or element that you look for is not unique on the page • Because XPath may be implemented differently between browsers, extra effort may be required to run WebDriver tests in those environments
  46. 46 Build Software to Test Software exactpro.com EXACTPRO DOM модель

  47. 47 Build Software to Test Software exactpro.com EXACTPRO HTML5 Canvas

  48. 48 Build Software to Test Software exactpro.com EXACTPRO Selenium WebDriver

    Tests
  49. 49 Build Software to Test Software exactpro.com EXACTPRO Selenium WebDriver

    Tests
  50. 50 Build Software to Test Software exactpro.com EXACTPRO Selenium WebDriver

    Test Execution
  51. 51 Build Software to Test Software exactpro.com EXACTPRO State of

    AI https://www.stateof.ai/ • 136 слайдов
  52. 52 Build Software to Test Software exactpro.com EXACTPRO Два основных

    фактора 1) Объем данных 2) Вычислительные мощности
  53. 53 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet
  54. 54 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet 1) в базе данных ☺
  55. 55 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet 1) в базе данных ☺ 2) логи серверов приложений
  56. 56 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet 1) в базе данных ☺ 2) логи серверов приложений 3) логи доступа
  57. 57 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet 1) в базе данных ☺ 2) логи серверов приложений 3) логи доступа 4) данные пользователей
  58. 58 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet 1) в базе данных ☺ 2) логи серверов приложений 3) логи доступа 4) данные пользователей 5) накликать
  59. 59 Build Software to Test Software exactpro.com EXACTPRO Где искать

    большие данные CSS HTML Javascript Application Servers Data Web Servers Static Content Browser DOM model HTTP(S) Request HTTP(S) Response Websocket Internet 1) в базе данных ☺ 2) логи серверов приложений 3) логи доступа 4) данные пользователей 5) накликать 6) другие приложения
  60. 60 Build Software to Test Software exactpro.com EXACTPRO Tasks AI

    Can Be Applied To
  61. 61 Build Software to Test Software exactpro.com Bug triaging BEFORE

    Extremely high memory consumption was observed in Prod system with release 1.2.3.4.5.6 installed during ITR and ITCH gateways crashed. AFTER Extremely high memory consumption was observed in Prod system with release 1.2.3.4.5.6 installed during ITR. All ITCH gateways consumed up to *** GB of RAM. They crashed, except for three pairs of ITCH gateways. Probably this issue can be related to issue #1234567. Backend logs, data files, core files, DB dumps are attached. Won’t Fix Pie Chart Won’t Fix Fix
  62. 62 Build Software to Test Software exactpro.com Bug triaging, e.g.

    of automated bug reports Machine Learning Feature engineering, dimensionality reduction, training & evaluation of the model Predictions and tips Cradle Data Warehousing Fetching raw data from tools under test. Processing it into datasets with Data Mart & BI support Test Execution and Analysis Predictions and tips for failures in order to leverage the smoothness and accuracy of the testing outputs analysis QA Analyst Fewer feedback loops, less rework, higher quality, faster turnaround, better alignment of roles on a project
  63. 63 Build Software to Test Software exactpro.com https://unsplash.com/search/photos/san-francisco The First

    IEEE International Conference on Artificial Intelligence Testing (IEEE AITest 2019), 4-9 April 2019, San Francisco East Bay, CA, USA. User-Assisted Log Analysis for Quality Control of Distributed Fintech Systems Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav Yavorskiy, Evgenii Tsymbalov, Andrey Novikov and Kirill Rudakov. Anomaly detection http://ieeeaitests.com/
  64. 64 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de
  65. 65 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de
  66. 66 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de
  67. 67 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de https://youtu.be/nvSqTlzivNM?t=1287
  68. 68 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de
  69. 69 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de
  70. 70 Build Software to Test Software exactpro.com EXACTPRO Oracle Problem

    In computing, software engineering, and software testing, a test oracle is a mechanism for determining whether a test has passed or failed The problem in applying AI to testing is the specification of the expected result that the outcome should be compared to. This is called the oracle problem, and it is more complex than it appears to be at first sight. In a broader sense, the oracle problem is the hardest problem of all: telling right from wrong. If this could be achieved in a general sense, all other theoretical problems could be reformulated as instances of the oracle problem and hence be solved. Note that this is not only true for mathematical or logical problems, but also for empirical and philosophical problems, and even for theological problems. For example, write a program that answers the question, “Does God exist?” and ask the test oracle whether the program is correct.
  71. 71 Build Software to Test Software exactpro.com EXACTPRO Tasks AI

    Cannot Be Applied To
  72. 72 Build Software to Test Software exactpro.com EXACTPRO Dr. Jeremias

    Rößler https://retest.de
  73. 73 Build Software to Test Software exactpro.com EXACTPRO The Generic

    Test Automation Architecture Advantages: • You can find elements that don't have any unique attributes (id, class, name, etc.) • You can use XPath in generic locators, using the different "By" strategies (by id, by class, etc.) as needed Disadvantages: • Absolute XPath code is "brittle" and can break with the smallest change to the HTML structure • Relative XPath code can find the wrong node if the attribute or element that you look for is not unique on the page • Because XPath may be implemented differently between browsers, extra effort may be required to run WebDriver tests in those environments
  74. 74 Build Software to Test Software exactpro.com EXACTPRO DOM модель

  75. 75 Build Software to Test Software exactpro.com EXACTPRO Is AI

    Taking Over FrontEnd Testing https://testim.io
  76. 76 Build Software to Test Software exactpro.com EXACTPRO Is AI

    Taking Over FrontEnd Testing https://testim.io https://youtu.be/S9iytpte6OA?t=1212
  77. 77 Build Software to Test Software exactpro.com EXACTPRO DOM модель

    улица тротуар дерево переход машина дом дорога пешеход город машина машина листик
  78. 78 Build Software to Test Software exactpro.com EXACTPRO Visual Testing

    https://applitools.com/
  79. 79 Build Software to Test Software exactpro.com EXACTPRO Visual Testing

    https://youtu.be/2CnOzf_DgP8?t=82
  80. 80 Build Software to Test Software exactpro.com EXACTPRO Visual Testing

  81. 81 Build Software to Test Software exactpro.com Click to edit

    the title text format Павел Браславский Мурад Мамедов Евгений Цымбалов Associate Professor, Higher School of Economics (@hsestpetersburg) and Researcher with @jetbrains AI Researcher (@exactpro) Data Scientist in Skoltech Meetup: Data Science Online 14 may Question answering: a short overview Вопросно-ответный поиск: краткий обзор. Failures’ Clustering: What a Machine Can Teach Us Кластеризация фейлов: Чему машина может научить нас. Active Learning for Machine Learning Активное обучение в машинном обучении.
  82. 82 Build Software to Test Software exactpro.com https://www.youtube.com/c/exactprosystems https://www.youtube.com/c/exactprovlog