From code to no-code. How MagicPod and its AI evolve test automation with Selenium

The slide for https://conference.idstb.org/ held on Oct 10, 2024.

Transcript

1. How MagicPod and its AI evolve test automation with Selenium From code to no-code Nozomi Ito
2. CEO of MagicPod, a Japanese test automation tool company The organizer of the Japan Selenium User Community Author of a Japanese Selenium book About me Nozomi Ito
3. This is my second trip to Indonesia First trip Second trip
4. No-code, cloud-based test automation SaaS For browser and mobile app testing Simple UI but with powerful features Free plan and paid plan Trusted by more than 500 companies No-code test automation tools
5. Agenda 1. My journey from Selenium to MagicPod 2. How MagicPod leverages Selenium internally 3. Values MagicPod adds to Selenium 4. Utilization of AI 5. MagicPod’s goal for the future of no-code
6. Agenda 1. My journey from Selenium to MagicPod 2. How MagicPod leverages Selenium internally 3. Values MagicPod adds to Selenium 4. Utilization of AI 5. MagicPod’s goal for the future of no-code
7. Established Selenium company (2012) Worked on consulting/support for test automation with Selenium in Japan Selenium: Open source browser automation framework
8. Promoted Selenium in Japan Launched Japan Selenium User community (from PublicKey) Published "Selenium Practical introduction" Held periodical Selenium workshops Organized SeleniumConf Tokyo
9. Challenges during the promotion Many Japanese companies started test automation with Selenium But large amount of Japanese companies still prefer to manual test
10. Why they did not automate E2E tests Lack motivation and knowedge about Selenium Cost of maintaining E2E test was too high In Japan, most QAs don't have technical background. Lack skills to use Selenium. Even commercial tools (Katalon, Ranorex, etc.) were too difficult for them QAs Engineers
11. From Selenium to no-code (2017) I decided to create MagicPod to spread E2E test automation No-code tool using Selenium/Appium internally Cloud service with easy setup Self healing AI to reduce maintenance costs Concepts
12. No-code has been widely accepted in Japan (2024) Growth of number of MagicPod users
13. Many industrial experts joined MagicPod Authors of test automation books Translators of test automation books Appium/Selenium conf presenters Selenium/Appium contributors/committers
14. Agenda 1. My journey from Selenium to MagicPod 2. How MagicPod leverages Selenium internally 3. Values MagicPod adds to Selenium 4. Utilization of AI 5. MagicPod’s goal for the future of no-code
15. How MagicPod works
16. Internal mechanism - test creation UI scan logic Analyze Server Upload Detect UI image UI tree XML human-readable caption locator
17. Internal mechanism - test creation [ { "command": "input", "element": "\"Name\" input area", "value": "Demo user" }, ... ] Test script Database Drag & Drop Store A command is added depending on the dropped item Scan result
18. Internal mechanism - test execution driver.findElementbyName ('"Name" input area') .sendKeys("Demo user"); ... driver.findElementbyName ('"Name" input area') .sendKeys("Demo user"); ... driver.findElementbyName ('"Name" input area') .sendKeys("Demo user"); ... Server JSON Node.js × Mocha × Selenium(or Appium) script Download Convert Run Test result Upload Cloud device Database [ { "command": "input", "element": "\"Name\" input area", "value": "Demo user" }, ... ]
19. System structure of MagicPod Run Run MagicPod has 2 types of execution methods. Server Local Local Server Database Database Cloud Browser device Local Browser device Cloud test Local test Script Script MagicPod Desktop
20. Agenda 1. My journey from Selenium to MagicPod 2. How MagicPod leverages Selenium internally 3. Values MagicPod adds to Selenium 4. Utilization of AI 5. MagicPod’s goal for the future of no-code
21. Sophisticated locator calculation Automatically calculates and suggests the most robust locator Locator candidates in the recommended order
22. Sophisticated locator calculation Order of locator recommendation Unique id data-testid → id → name → text() → alt → value → title → aria-label → placeholder → class: id = userName tag name + unique id xpath = //button[@id='log in'] label xpath = //input[@id=//label[text()='User name']/@for]] tag name xpath = //input[2] child xpath = //div[@id='mainArea']/button[1] parent xpath = //span[text()='log in']/parent::button[1] sibling xpath = //div[text()='Name']/following-sibling::input[1] full path xpath xpath = /body[1]/div[1]/div[2]/div[3]/a
23. Sophisticated locator calculation Random unique identifiers have low priority Use statistical approach to judge "random" xpath=//button[text()='Sign up'] xpath=//button[2] xpath=//div[@id='mainForm']/div[1]/button[1] id=577656711eee44a
24. Automatic wait and retry fdffff Selenium Various explicit wait commands Automtically wait for elements to appear with implictWait driver.manage().timeouts() .implicitlyWait( Duration.ofSeconds(2));
25. Automatic wait and retry MagicPod Explicit wait commands like selenium More sophisticated implicit wait Wait for element position to be settled like Playwright Retry for stale element error fdffff WebElement e = driver.findElement( By.id("logIn")); e.click();
26. Powerful features without code Features for maintainability Shared step Conditionals While loop Variables coming soon
27. Powerful features without code Powerful commands resembling code-based testing Web api call 2FA code calculation Regular expression match Date calculation
28. Seamless frame and webview handling driver.switchTo().frame( driver.findElement( By.id("iframe1"))); driver.findElement( By.id("OK")).click(); Automatically calculate which context to switch Frame for web test and webview for native mobile app test Selenium MagicPod “OK” Button id=OK frame: iframe1
29. Built-in visual diff check feature Easily detect broken design issues. Detected the visual diff
30. Agenda 1. My journey from Selenium to MagicPod 2. How MagicPod leverages Selenium internally 3. Values MagicPod adds to Selenium 4. Utilization of AI 5. MagicPod’s goal for the future of no-code
31. AI uses in MagicPod Human-readable UI element captions 1. Self-healing 2. Summarization of test script 3.
32. 1. Human-readable UI element captions Human-readable caption ・Calculated by AI ・Not used by the test runner Selenium/Appium locator ・Calculated from UI tree ・Used by the test runner
33. 1. Human-readable UI element captions Information on the UI tree: attributes, texts, tag names, etc. Parent-child relation between elements
34. 1. Human-readable UI element captions OCR result (Recognize characters in the image) Object recognition by the deep learning engine
35. 2. Self-healing When a test fails to find a UI element due to a change in the application’s page structure, AI modifies the script according to the page change.
36. 2. Self-healing Click “register” Test script name attribute updated! What happens if it is executed? name=register name=registerButton
37. 2. Self-healing The AI engine updates script and the test passes The user is notified of the update detail
38. 3. Summarization of test script Summaries of test cases are automatically generated.
39. 3. Summarization of test script Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text The script "Check that signup works correctly" automates end-to-end testing for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. Test script Summary MagicPod public API Generative AI
40. 3. Summarization of test script Prompt Create a short summary (less than 300 characters) of the following browser testing user flow briefly in English: [Human-readable test script ]
41. Agenda 1. My journey from Selenium to MagicPod 2. How MagicPod leverages Selenium internally 3. Values MagicPod adds to Selenium 4. Utilization of AI 5. MagicPod’s goal for the future of no-code
42. Integration of code and no-code Encourage collaboration between engineers and non-engineers Short-term goal of MagicPod Mutually convertible LaunchApp() input(e('register', '"name" input area'), 'Yamada Taro') input(e('register', '"gender" input area'), 'Male') tap(e('register', '"Register" button'))
43. Execute tests directly from test cases designed for manual testing. Next major goal: creating test cases entirely from scratch. Log in with user@example.com / pass001 Long-term goal of MagicPod