Using Machine Learning to Test Web Applications

Elena Treshcheva
Researcher, Exactpro

San Diego State University, 12 June 2020

Video : https://youtu.be/hnBlsGs9LzE

In this presentation, Elena Treshcheva, Researcher, Business Development Manager at Exactpro, describes the fundamental concepts of machine learning and compares the capabilities of DOM-based testing to those of visual AI. The presentation was recorded at the open webinar held for San Diego State University students on 12 June 2020.

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 1
   Using Machine Learning
   to Test Web Applications
   Elena Treshcheva
   Researcher, Exactpro
   San Diego State University, 12 June 2020
   

   View Slide

2. 2 Build Software to Test Software
   exactpro.com
   Overview
   • References / Useful links
   • A web application from the point of view of a human being
   • Software testing, test automation and machine learning
   • A human being from the point of view of a web application
   • Testing with Selenium
   • The Oracle problem in software testing
   • ML applied to testing of DOM
   • ML applied to visual testing
   

   View Slide

3. 3 Build Software to Test Software
   exactpro.com
   A specialist firm focused on functional
   and non-functional testing of
   exchanges, clearing houses,
   depositories, trade repositories and
   other financial market infrastructures.
   About Exactpro
   Headquartered in the UK with
   operations in the US and Eastern
   Europe. The clients are regulated by
   the FCA, Bank of England and their
   counterparts from other countries.
   Exactpro provides software testing services for mission critical technology that underpins global financial
   markets. The firm is experienced with trading and clearing and settlement platforms, market data systems,
   collateral management and risk management systems, central data warehouses, regulatory reporting, etc.
   Incorporated in 2009 with 10 people,
   Exactpro is now employing over 590
   specialists.
   Part of the London Stock Exchange
   Group (LSEG) from May 2015 until
   January 2018, when the Exactpro
   management proceeded through the
   buyout of the company from LSEG.
   

   View Slide

4. 4 Build Software to Test Software
   exactpro.com
   EXACTPRO
   Non-Deterministic Probabilistic Non-Testable Oracle Problem
   Mission-critical Financial Markets Infrastructures
   

   View Slide

5. 5 Build Software to Test Software
   exactpro.com
   Typical requirements for an Exchange system
   Daily capacity - 100+ mln transactions
   Peak rates - 40k+ transactions per second
   Average round-trip latency - <100 microseconds
   Availability - 100%
   3000 trx 2.5 cm <1 mm
   

   View Slide

6. 6 Build Software to Test Software
   exactpro.com
   EXACTPRO
   Useful Links
   https://frontendmasters.com/books/front-end-handbook/2019/
   

   View Slide

7. 7 Build Software to Test Software
   exactpro.com
   EXACTPRO
   https://youtu.be/nvSqTlzivNM
   https://www.slideshare.net/roesslerj/how-to-apply-ai-to-testing
   Useful Links
   

   View Slide

8. 8 Build Software to Test Software
   exactpro.com
   25 February 2020
   https://youtu.be/KGVQVBdjvCU
   Useful Links
   

   View Slide

9. 9 Build Software to Test Software
   exactpro.com
   EXACTPRO
   https://www.gasq.org/en/certification/selenium-tester.html
   Useful Links
   

   View Slide

10. 10 Build Software to Test Software
    exactpro.com
    EXACTPRO
    https://www.youtube.com/watch?v=S9iytpte6OA
    https://www.testim.io/
    Useful Links
    

    View Slide

11. 11 Build Software to Test Software
    exactpro.com
    EXACTPRO
    https://testautomationu.applitools.com/ai-for-element-selection-erasing-the-pain-of-fragile-test-scripts
    Useful Links
    

    View Slide

12. 12 Build Software to Test Software
    exactpro.com
    EXACTPRO
    The structure of a Web Application
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    HTTP(S)
    Request
    HTTP(S)
    Response
    1
    2
    CSS
    HTML
    Javascript
    Internet
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    

    View Slide

13. 13 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    HTTP(S)
    Request
    HTTP(S)
    Response
    1
    2
    CSS
    HTML
    Javascript
    Internet
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    The structure of a Web Application
    

    View Slide

14. 14 Build Software to Test Software
    exactpro.com
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    1 Client-side scripts
    Run in the browser and
    process information
    without sending requests
    to the server
    3 2
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Internet
    The structure of a Web Application
    

    View Slide

15. 15 Build Software to Test Software
    exactpro.com
    EXACTPRO
    The structure of a Web Application
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    1 Client-side scripts
    Run in the browser and
    process information
    without sending requests
    to the server
    3 AJAX allows to create
    asynchronous web
    interactions without
    interfering with the
    display and behaviour of
    the existing page
    4 2
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Internet
    

    View Slide

16. 16 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    WebSocket enables interaction
    between a web browser and a
    web server with lower overhead
    than half-duplex alternatives
    such as HTTP polling
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    1 Client-side scripts
    Run in the browser and
    process information
    without sending requests
    to the server
    3 AJAX allows to create
    asynchronous web
    interactions without
    interfering with the
    display and behaviour of
    the existing page
    4 2
    5
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    The structure of a Web Application
    

    View Slide

17. 17 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    WebSocket enables interaction
    between a web browser and a
    web server with lower overhead
    than half-duplex alternatives
    such as HTTP polling
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    1 Client-side scripts
    Run in the browser and
    process information
    without sending requests
    to the server
    3 AJAX allows to create
    asynchronous web
    interactions without
    interfering with the
    display and behaviour of
    the existing page
    4 2
    5
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    The structure of a Web Application
    

    View Slide

18. 18 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    WebSocket enables interaction
    between a web browser and a
    web server with lower overhead
    than half-duplex alternatives
    such as HTTP polling
    Responsive front-end
    design allows a site to
    adapt to user’s device.
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    1 Client-side scripts
    Run in the browser and
    process information
    without sending requests
    to the server
    3 AJAX allows to create
    asynchronous web
    interactions without
    interfering with the
    display and behaviour of
    the existing page
    4 2
    5
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    The structure of a Web Application
    

    View Slide

19. 19 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Browser send requests
    to the server, receive
    responses and visualize
    them as a page
    WebSocket enables interaction
    between a web browser and a
    web server with lower overhead
    than half-duplex alternatives
    such as HTTP polling
    Responsive front-end
    design allows a site to
    adapt to user’s device.
    Everything a user
    sees in the browser
    is a mix of HTML,
    CSS, and
    JavaScript.
    The back-end server-side
    scripts process the request,
    pull what they need from the
    database and application
    servers and then send it back.
    1 Client-side scripts
    Run in the browser and
    process information
    without sending requests
    to the server
    3 AJAX allows to create
    asynchronous web
    interactions without
    interfering with the
    display and behaviour of
    the existing page
    4 2
    5
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    Web applications are always
    multi-user applications and
    many of them work under heavy
    load and are subject to
    concurrency access
    6
    The structure of a Web Application
    

    View Slide

20. 20 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Quality - The degree to which a component or system satisfies the stated and implied needs of its various
    stakeholders.
    Software Testing - The process consisting of all lifecycle activities, both static and dynamic, concerned with
    planning, preparation and evaluation of a component or system and related work products to determine that they
    satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.
    A person can make an error, which can lead to the introduction of a defect (fault or bug)
    in the software code. If a defect in the code is executed, this may cause a failure
    Software Testing
    

    View Slide

21. 21 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Testing Levels
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    A
    C
    D
    E
    F B
    

    View Slide

22. 22 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Automation
    Automation is the technology by which a process or procedure is performed with
    minimal human assistance.
    “The first rule of any technology used in a business is that
    automation applied to an efficient operation will magnify the
    efficiency. The second is that automation applied to an inefficient
    operation will magnify the inefficiency”
    Bill Gates
    

    View Slide

23. 23 Build Software to Test Software
    exactpro.com
    EXACTPRO
    A Model of the Human from a Web App’s Perspective
    

    View Slide

24. 24 Build Software to Test Software
    exactpro.com
    EXACTPRO
    A Model of the Human from a Web App’s Perspective
    

    View Slide

25. 25 Build Software to Test Software
    exactpro.com
    EXACTPRO
    A Model of the Human from a Web App’s Perspective
    

    View Slide

26. 26 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Document
    
    
    
    Root element
    Element
    “The Header”
    
    “the link”
    “href”
    “Sub Header”
    “The title”
    
    Element
    Element
    Element
    Element
    Text
    Attribute Text Text
    Element
    Text
    The DOM defines:
    All HTML elements as objects
    • The properties of all HTML elements
    • The methods that can be used to access all HTML elements
    • The events that affect all HTML elements
    "The W3C Document Object Model (DOM) is
    a platform and language-neutral interface
    that allows programs and scripts to
    dynamically access and update the content,
    structure, and style of a document."
    DOM (Document Object model)
    

    View Slide

27. 27 Build Software to Test Software
    exactpro.com
    EXACTPRO
    DOM модель
    

    View Slide

28. 28 Build Software to Test Software
    exactpro.com
    EXACTPRO
    DOM (Document Object model)
    

    View Slide

29. 29 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Document
    
    
    
    Root element
    Element
    “The Header”
    
    “the link”
    “href”
    “Sub Header”
    “The title”
    
    Element
    Element
    Element
    Element
    Text
    Attribute Text Text
    Element
    Text
    The DOM defines:
    All HTML elements as objects
    • The properties of all HTML elements
    • The methods that can be used to access all HTML elements
    • The events that affect all HTML elements
    "The W3C Document Object Model (DOM) is
    a platform and language-neutral interface
    that allows programs and scripts to
    dynamically access and update the content,
    structure, and style of a document."
    DOM (Document Object model)
    

    View Slide

30. 30 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Testing Levels
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    A
    C
    D
    E
    F B
    

    View Slide

31. 31 Build Software to Test Software
    exactpro.com
    EXACTPRO
    DOM (Document Object model)
    

    View Slide

32. 32 Build Software to Test Software
    exactpro.com
    DOM (Document Object model)
    street
    car
    car
    crosswalk
    person
    traffic light
    road
    building
    tree
    sidewalk
    

    View Slide

33. 33 Build Software to Test Software
    exactpro.com
    EXACTPRO
    street
    sidewalk
    tree crosswalk
    car
    building
    road
    pedestrian
    city
    car
    car
    branch
    DOM (Document Object model)
    street
    car
    car
    crosswalk
    person
    traffic light
    road
    building
    tree
    sidewalk
    

    View Slide

34. 34 Build Software to Test Software
    exactpro.com
    EXACTPRO
    The Generic Test Automation Architecture
    

    View Slide

35. 35 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Advantages:
    • You can find elements that don't have any unique
    attributes (id, class, name, etc.)
    • You can use XPath in generic locators, using the
    different "By" strategies (by id, by class, etc.) as
    needed
    Disadvantages:
    • Absolute XPath code is "brittle" and can break with
    the smallest change to the HTML structure
    • Relative XPath code can find the wrong node if the
    attribute or element that you look for is not unique
    on the page
    • Because XPath may be implemented differently
    between browsers, extra effort may be required to
    run WebDriver tests in those environments
    The Generic Test Automation Architecture
    

    View Slide

36. 36 Build Software to Test Software
    exactpro.com
    EXACTPRO
    State of AI
    https://www.stateof.ai/
    • 136 слайдов
    

    View Slide

37. 37 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Two main factors
    1) Data volume
    2) Computing power
    

    View Slide

38. 38 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    

    View Slide

39. 39 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    1) a database ☺
    

    View Slide

40. 40 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    2) web server logs
    1) a database ☺
    

    View Slide

41. 41 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    2) web server logs
    1) a database ☺
    3) access logs
    

    View Slide

42. 42 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    2) web server logs
    1) a database ☺
    3) access logs
    4) user data
    

    View Slide

43. 43 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    2) web server logs
    1) a database ☺
    3) access logs
    4) user data
    5) random clicking
    

    View Slide

44. 44 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Big Data: How to obtain it?
    CSS
    HTML
    Javascript
    Application
    Servers
    Data
    Web
    Servers
    Static
    Content
    Browser
    DOM model
    HTTP(S)
    Request
    HTTP(S)
    Response
    Websocket
    Internet
    2) web server logs
    1) a database ☺
    3) access logs
    4) user data
    5) random clicking
    6) other applications
    

    View Slide

45. 45 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Tasks AI Can Be Applied To
    

    View Slide

46. 46 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    https://youtu.be/nvSqTlzivNM?t=1287
    

    View Slide

47. 47 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    

    View Slide

48. 48 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    

    View Slide

49. 49 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    

    View Slide

50. 50 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    

    View Slide

51. 51 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    

    View Slide

52. 52 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Oracle Problem
    In computing, software engineering, and software testing, a test oracle is a mechanism for
    determining whether a test has passed or failed.
    The problem in applying AI to testing is the specification of the expected result that the outcome
    should be compared to. This is called the oracle problem, and it is more complex than it appears
    to be at first sight.
    In a broader sense, the oracle problem is the hardest problem of all: telling right from wrong. If
    this could be achieved in a general sense, all other theoretical problems could be reformulated as
    instances of the oracle problem and hence be solved. Note that this is not only true for
    mathematical or logical problems, but also for empirical and philosophical problems, and even for
    theological problems. For example, write a program that answers the question, “Does God exist?”
    and ask the test oracle whether the program is correct.
    

    View Slide

53. 53 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Humans vs. Machines
    “The classic aim of automation is to replace human manual control, planning and problem
    solving by automatic devices and computers. …the increased interest in human factors
    among engineers reflects the irony that the more advanced a control system is, so the more
    crucial may be the contribution of the human operator.”
    Lisanne Bainbridge, “Ironies of Automation” (1983)
    OR ?
    World Development Report 2019: The Changing Nature of Work
    

    View Slide

54. 54 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Tasks AI Cannot Be Applied To
    

    View Slide

55. 55 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Dr. Jeremias Rößler https://retest.de
    

    View Slide

56. 56 Build Software to Test Software
    exactpro.com
    EXACTPRO
    The Generic Test Automation Architecture
    Advantages:
    • You can find elements that don't have any unique
    attributes (id, class, name, etc.)
    • You can use XPath in generic locators, using the
    different "By" strategies (by id, by class, etc.) as
    needed
    Disadvantages:
    • Absolute XPath code is "brittle" and can break with
    the smallest change to the HTML structure
    • Relative XPath code can find the wrong node if the
    attribute or element that you look for is not unique
    on the page
    • Because XPath may be implemented differently
    between browsers, extra effort may be required to
    run WebDriver tests in those environments
    

    View Slide

57. 57 Build Software to Test Software
    exactpro.com
    EXACTPRO
    DOM (Document Object model)
    

    View Slide

58. 58 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Is AI Taking Over FrontEnd Testing
    https://testim.io
    

    View Slide

59. 59 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Is AI Taking Over FrontEnd Testing
    https://youtu.be/S9iytpte6OA?t=1212
    https://testim.io
    

    View Slide

60. 60 Build Software to Test Software
    exactpro.com
    EXACTPRO
    street
    sidewalk
    tree crosswalk
    car
    building
    road
    pedestrian
    city
    car
    car
    branch
    DOM (Document Object model)
    street
    car
    car
    crosswalk
    person
    traffic light
    road
    building
    tree
    sidewalk
    

    View Slide

61. 61 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Visual Testing
    https://applitools.com/
    

    View Slide

62. 62 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Visual Testing
    https://youtu.be/2CnOzf_DgP8?t=82
    

    View Slide

63. 63 Build Software to Test Software
    exactpro.com
    EXACTPRO
    https://testautomationu.applitools.com/ai-for-element-selection-erasing-the-pain-of-fragile-test-scripts
    Useful Links
    

    View Slide

64. 64 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Visual Testing
    

    View Slide

65. 65 Build Software to Test Software
    exactpro.com
    Click to edit the title text format
    - ტესტირება ეჯაილში - თინათინ ცირეკიძე, GeoSTQB-ის ვიცე-პრეზიდენტი და ლიბერთი ბანკის
    დეპარტამენტის ხარისხის მართვის ჯგუფის უფროსი
    - Future of Software Testing - Kari Kakkonen, Treasurer at FiSTB, Secretary at ISTQB, Director of Training
    at Knowit Solutions Oy and CEO and Author at Dragons Out Oy
    - გადასვლა ავტომატიზაციაზე - მიხეილ მაჩაბლიშვილი, Tulaco, ავტომატიზაციის უფროსი ინჟინერი
    - Testing Software Resiliency - Iosif Itkin, Exactpro, CEO and Co-Founder.
    არ გამოტოვოთ და მოინიშნეთ კალენდარში
    15 ივნისი საღამოს 6 საათი. გელოდებით!
    https://www.facebook.com/events/
    1640675652767046
    

    View Slide

66. 66 Build Software to Test Software
    exactpro.com
    Click to edit the title text format
    Register here
    

    View Slide

67. 67 Build Software to Test Software
    exactpro.com
    https://www.youtube.com/c/exactprosystems
    https://www.youtube.com/c/exactprovlog
    

    View Slide