Slide 1

Slide 1 text

Leveraging AI in Test Automation Products AI & Software Testing Night

Slide 2

Slide 2 text

Nozomi Ito ( 伊藤 望) Founder/CEO of MagicPod Inc. Organizer of SeConf Tokyo and SeleniumJP Author of several test automation books X: @nozomi_ito2 About me

Slide 3

Slide 3 text

Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.

Slide 4

Slide 4 text

Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.

Slide 5

Slide 5 text

No-code, cloud-based, end-to-end test automation platform for web & mobile apps

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.

Slide 8

Slide 8 text

AI features of MagicPod 2. AI Use in MagicPod ① Human-readable UI element captions ② Self-healing

Slide 9

Slide 9 text

UI element caption calculation by AI 2. AI Use in MagicPod - ①Human-readable UI element captions Human-readable caption ・Calculated by AI ・Not used by the test runner Selenium/Appium locator ・Calculated from UI tree ・Used by the test runner

Slide 10

Slide 10 text

Information used for caption calculation 2. AI Use in MagicPod - ①Human-readable UI element captions Information on the UI tree: attributes, texts, tag names, etc. Position relation between elements

Slide 11

Slide 11 text

Information used for caption calculation 2. AI Use in MagicPod - ①Human-readable UI element captions OCR (Recognize characters on the image) Object recognition by the deep learning engine

Slide 12

Slide 12 text

What is self-healing? 2. AI Use in MagicPod - ②Self-healing When a test fails to find a UI element due to a change in the application’s page structure, AI modifies the script according to the page change.

Slide 13

Slide 13 text

Click “register” Test script internal name updated! What happens if it is executed? Example of self-healing 2. AI Use in MagicPod - ②Self-healing name=register name=registerButton

Slide 14

Slide 14 text

Example of self-healing 2. AI Use in MagicPod - ②Self-healing The AI engine updates script and the test passes The user is notified of the update detail

Slide 15

Slide 15 text

2 types of self-healing 2. AI Use in MagicPod - ②Self-healing A. Multiple locators approach B. Single locator approach Adopted by MagicPod

Slide 16

Slide 16 text

A. Multiple locators approach 2. AI Use in MagicPod - ②Self-healing Suitable for record-and-playback test automation tools When recording a test: Record an element by multiple attributes Such as id, name, index When the test fails: When failing to identify the element by an attribute, try other recorded attributes

Slide 17

Slide 17 text

A. Multiple locators approach 2. AI Use in MagicPod - ②Self-healing ex.) Testim (testim.io) *based on our own research Update weight when changing the locator

Slide 18

Slide 18 text

B. Single locator approach 2. AI Use in MagicPod - ②Self-healing Adopted by MagicPod When creating a test: Store locators and the full UI tree (HTML) When the test fails: Compare the run-time UI tree and the recorded UI tree, and use a sufficiently similar element if it exists

Slide 19

Slide 19 text

B. Single locator approach 2. AI Use in MagicPod - ②Self-healing ex.) MagicPod Creation-time UI tree Run-time UI tree Creation-time locator New locator

Slide 20

Slide 20 text

B. Single locator approach 2. AI Use in MagicPod - ②Self-healing Factors for determining elements similarity Element attribute, position, parent-child relationship, etc. OCR result (MagicPod-only) Object recognition result by the deep learning engine (MagicPod-only) Advantages The same simple locator as Selenium/Appium

Slide 21

Slide 21 text

Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.

Slide 22

Slide 22 text

3. Use of Generative AI Very useful for code test automation tools such as Selenium, Playwright Not so useful for no-code test automation tools at this time Conversation with generative AI requires more time than just recording user operation Is AI useful for test scenario generation?

Slide 23

Slide 23 text

3. Use of Generative AI 3 ideas for generative AI collaboration ① Summarize test cases ② Generate change comments ③ (Detect random IDs)

Slide 24

Slide 24 text

3. Use of Generative AI - ①Summarize test cases Motivations: Understanding each test case requires large efforts for a new member It is not easy to keep human-written summary up-to-date Generate summary by generative AI

Slide 25

Slide 25 text

3. Use of Generative AI - ①Summarize test cases Experiment Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text The script "Check that signup works correctly" automates end-to-end testing for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. Test script Summary MagicPod public API Generative AI

Slide 26

Slide 26 text

3. Use of Generative AI - ①Summarize test cases Test script

Slide 27

Slide 27 text

3. Use of Generative AI - ①Summarize test cases Human-readable text Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Title: Check that singup works correctly

Slide 28

Slide 28 text

The following is a browser E2E test automation script. The script name is "[test script name]". Could you create a summary by which I can grasp what kind of test the script does? =============================================== [Human-readable test script ] 3. Use of Generative AI - ①Summarize test cases Prompt for generative AI

Slide 29

Slide 29 text

The script "Check that signup works correctly" automates end-to-end testing for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. 3. Use of Generative AI - ①Summarize test cases Generated summary

Slide 30

Slide 30 text

3. Use of Generative AI - ②Generate change comments Generate change comments by AI Motivations: Set better change comment than “Autosaved” when a change is saved automatically

Slide 31

Slide 31 text

Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Updated the script by modifying the user's name, address, and telephone number inputs to use new values for improved clarity and specificity in test data. 3. Use of Generative AI - ②Generate change comments Experiment Human-readable text (old) Test script (old) change comment Generative AI Test script (new) Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text (new)

Slide 32

Slide 32 text

3. Use of Generative AI - ②Generate change comments Change detail ......

Slide 33

Slide 33 text

3. Use of Generative AI - ②Generate change comments Prompt for generative AI The following 2 browser E2E test automation scripts represent the script before change and the script after the change when an engineer modified the script. Before Change ---------------- [Full human-readable script before the change] ---------------- After change ---------------- [Full human-readable script after the change] ---------------- Please create a change comment in one sentence.

Slide 34

Slide 34 text

Updated the script by modifying the user's name, address, and telephone number inputs to use new values for improved clarity and specificity in test data. 3. Use of Generative AI - ②Generate change comments Generated change comments Updated user profile information for signup test to reflect new user details and address. Variation 2 Variation 1

Slide 35

Slide 35 text

Motivations: We don’t want to use random IDs in locators since they are dynamic, cryptic, and vulnerable to change Random ID detection is not always easy 3. Use of Generative AI - ③Detect random IDs Smarter locators not using random IDs

Slide 36

Slide 36 text

3bb5fe17-285c-4aa7-828f-f54816ed1bdf user001 prefix_2bd5ff24-185c-4ab7-128f-f54616ededda second_section magicPod mantine-zbve6blpq test_combo gtm-noscript m-large 23_aKvs-b8bW2Vg3fwHozO 13LGdX8RMStbBE9w-t0gZ1 css-r6dqg1 prefix_3bb5fe17-285c-4aa7-828f-f54816ed1bdf code-snippet related-container data-v-0c1f62df privateStripeFrame456 aras-toolbar container hs-analytics aras-header aras-es-toolbar-input o3ofqfx9qfqo sc-ilJxFg bkOQwZ gh-header-actions mt-0 mb-3 mb-md-2 ml-1 flex-md-order-1 flex-shrink-0 d-flex flex-items-center gap-1 3. Use of Generative AI - ③Detect random IDs Experiment all ids & classes HTML Generative AI 3bb5fe17-285c-4aa7-828f-f54816ed1bdf prefix_2bd5ff24-185c-4ab7-128f-f54616ededda mantine-zbve6blpq 23_aKvs-b8bW2Vg3fwHozO 13LGdX8RMStbBE9w-t0gZ1 css-r6dqg1 prefix_3bb5fe17-285c-4aa7-828f-f54816ed1bdf data-v-0c1f62df o3ofqfx9qfqo sc-ilJxFg bkOQwZ random ids & classes Locator calculation logic

Slide 37

Slide 37 text

The followings are newline-separated list of id attributes in a HTML. Among them, which ones contain randomly generated texts? "Randomly generated" means, not decided by human, but generated by system and vulnerable to change. Please return the result as a newline-separated list. Note that we don't regard number sequences/prefixes/suffixes as "Randomly generated" texts. [New-line separated id list] 3. Use of Generative AI - ③Detect random IDs Prompt for generative AI

Slide 38

Slide 38 text

3. Use of Generative AI - ③Detect random IDs Result Generative AI can detect many random IDs. Detected ones are almost always correct But frequently it fails to detect some random IDs Besides, it takes 6 seconds to check just 30 IDs It is difficult to integrate the logic into the MagicPod product

Slide 39

Slide 39 text

Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.

Slide 40

Slide 40 text

4. Our Final Goal Final goal of MagicPod MagicPod can execute tests just from the test case for manual tester Creating a test case completely from scratch is further next goal Log in with user@example.com / pass001

Slide 41

Slide 41 text

Thank you!

Slide 42

Slide 42 text

No content