Tweet
Tweet

Slide 1

Slide 1 text

UMNIUM AI CAMP @ FEB 13, 2025

Slide 2

Slide 2 text

ALEX RODIONOV - Quality Architect @ Toptal - Author @ Maccy - Ruby maintainer @ Selenium - Ruby maintainer @ Bazel

Slide 3

Slide 3 text

1. Writing tests: a. fi guring out the interactions and locators b. writing the code 2. Supporting tests: a. adapting to UI changes b. dealing with fl akiness c. surviving massive refactoring PAIN OF END-TO-END TESTING?

Slide 4

Slide 4 text

1. Autocomplete and chat (Copilot et al). 2. AI-powered test generators (LaVague). 3. AI-powered test runners (Shortest, Alumnium). AI-POWERED SOLUTIONS

Slide 5

Slide 5 text

2+2=4: WRITING A TEST FOR CALCULATOR

Slide 6

Slide 6 text

1. De fi ne tests in Cucumber format. 2. lavague-qa generates Selenium tests using AI. 3. Commit the tests to your repo. 4. Run on CI. 5. Re-generate whenever something changes. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 7

Slide 7 text

LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 8

Slide 8 text

LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 9

Slide 9 text

LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 10

Slide 10 text

LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 11

Slide 11 text

LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 12

Slide 12 text

LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 13

Slide 13 text

+ You can see what’s actually generated + Once generated, the tests are fast + Cheap - Cannot be integrated into existing tests - Have to use Gherkin language - No auto-regeneration (i.e. if the test failed) LAVAGUE (TEST GENERATOR) https://www.lavague.ai/

Slide 14

Slide 14 text

1. De fi ne tests in Shortest format. 2. Commit them to repo. 3. Run on CI. 4. Test uses Anthropic Computer Use API to interact with browser. SHORTEST (TEST RUNNER) https://shortest.com/

Slide 15

Slide 15 text

SHORTEST (TEST RUNNER) https://shortest.com/

Slide 16

Slide 16 text

SHORTEST (TEST RUNNER) https://shortest.com/

Slide 17

Slide 17 text

+ Easy to use + Adapts to changes in UI - Cannot be integrated into existing tests - Expensive - Slow https://shortest.com/ SHORTEST (TEST RUNNER)

Slide 18

Slide 18 text

1. High-level API for Selenium/Playwright: a. do something on the page; b. get something from the page; c. check something on the page. 2. Integrates into existing tests. 3. Provides gradual migration path. 4. Supports all major AI providers (Anthropic, Google, OpenAI). 5. Works on low-tier models (gpt-4o-mini, gemini-2.0- fl ash, claude-3-haiku) 6. Works with open models too (Llama 3.2). 7. Open source under MIT license. ALUMNIUM https://alumnium.ai/

Slide 19

Slide 19 text

ALUMNIUM https://alumnium.ai/

Slide 20

Slide 20 text

ALUMNIUM https://alumnium.ai/

Slide 21

Slide 21 text

DEMO

Slide 22

Slide 22 text

PERFORMANCE (1 TEST) 1. LaVague = 51s to generate, 1s to run 2. Shortest = 58s to run 3. Alumnium = 8s to run https://alumnium.ai/

Slide 23

Slide 23 text

COSTS (1 TEST) 1. LaVague = $0.08 (pay once) 2. Shortest = $0.15 3. Alumnium = $0.001 (80-150 times cheaper) https://alumnium.ai/

Slide 24

Slide 24 text

AL.DO(…) https://alumnium.ai/

Slide 25

Slide 25 text

AL.GET(…) https://alumnium.ai/

Slide 26

Slide 26 text

AL.CHECK(…) https://alumnium.ai/

Slide 27

Slide 27 text

1. LLM lack of determinism. 2. Support for di ff erent models. 3. Prompting . 4. Scaling to bigger a11y trees. 5. Improving performance. 6. Working with sites that has poor a11y. CHALLENGES https://alumnium.ai/

Slide 28

Slide 28 text

1. Other programming languages (JavaScript, Ruby, etc.) 2. Mobile applications testing. 3. Local models. 4. Evaluation benchmarks. ROADMAP https://alumnium.ai/

Slide 29

Slide 29 text

- https://alumnium.ai - https://github.com/alumnium-hq/alumnium - https://seleniumhq.slack.com/channels/alumnium - [email protected] THANK YOU!