What I did last summer to cure our Testing

My experience with fixing and refactoring of Test Automation.
Nowadays we all accumulated quite a lot of Automated Testing solutions and have issues managing it.

Transcript

1. What I did last summer to cure our Testing Vjacheslav Lukashevich Evolution Gaming
2. ABOUT ME v I am a Tester v Studied Test Automation extensively v Worked on a number of great projects with Test Automation v Coaching and Teaching v Co-organiser of Riga Test Automation Club 1
3. MY STORY v Joined Evolution Gaming in Spring 2018 v QA Lead v QA for one of the product Teams v During summer worked to gain confidence in Test Automation of my Team 2
4. v World leader in live dealer gaming v Serves more top-tier gaming brands than any other provider v Business is actively growing v 3 Development Centres with ~400 engineers v Riga v Tallinn v Amsterdam Evolution Gaming 3
5. TECHNOLOGIES 4 v Lots of Integrations with v Video control devices and software v Various other hardware #scala #akka #cqrs #kaffka #typescript #react #redux #babylonjs #webpack #jest #postcss #aws #kubernetes #selenium #appium #gattling #webdriver.io #pupeteer #allure #java
6. HIGH DEMAND FOR TESTING v Products highly customisable and feature-rich v Lots of changes v New features/products v Modification of old features v Constantly releasing to various environments 5
7. LOTS OF TEST AUTOMATION! v Tests for all products v Bots for performance testing v Infrastructure to run it v Cloud and Physical environments v CI/CD pipelines v Monitoring, alerting and analytics 6
8. TEST AUTOMATION in My Team v Test framework started in 2011 v Plain Java scenarios with Page Objects v Lots of different builds v On application code PUSH to GIT v Scheduled v Runs for Flash, HTML5, Mobile platforms 7
9. 8 Now is the time when a LOT of Test Automation is already in place!
10. 9 ..and we have issues managing this! (my main motivation for this speech)
11. 10 v Builds often are RED v Application status unclear v Tests are difficult to understand v Testers reporting never-ending ‘fixing tests’ v People find issues not found by automation The problem Automation state
12. As a RESULT vCatching stuff late vStress! vMore effort than it should be… vLacking a sense of control vBad reputation of tests… 11 Urgent need to take some actions!
13. Sounds familiar, right?... 12
14. Why is it so? 13
15. It just happened… v Knowledge gaps v Not enough technical supervision v Push - progress quick! v People taking shortcuts v Quick and dirty… v Copy-Paste v Skip discussions -implement ‘own wheel’ v Lost Vision and Goals 14
16. This is what it is called! 15 https://xtend-it.com/blog/consider-technical-debt-real-debt/
17. Is everything so bad? 16
18. On the positive side 17 v Still catching bugs v Some defence line is in place! v Contains recorded knowledge of our features v Reused across several Teams v Testing of different solutions v Various environments Does a lot of stuff we can’t accomplish manually!
19. It is all about Efficiency! Efficiency is the (often measurable) ability to avoid wasting materials, energy, effort, money, and time in doing something or in producing a desired result. https://en.wikipedia.org/wiki/Efficiency 18
20. WILL TALK ABOUT v Making small steps to fix test automation v Constantly asked myself “What is the right way?” 19 19
21. I’ve just joined my Team Setting up my workplace and tools 20
22. Should I implement everything “Correctly” from scratch?.. Automation problem 21
23. Will they wait for me? v Is there a guarantee, that a new auto- test solution will be better? v I am still at the exploration phase v Many great things in place v Infrastructure, pipelines, builds v Preconditions for tests v Lots of ready tests 22
24. I will take steps to get more control over the existing auto-test solution! My Actions 23
25. STEP 1 Reconnaissance 24
26. I took some tests and refactored for knowledge! Step 1 25
27. My Plan v I’ve picked GREEN ones from SMOKE TESTS and started adjusting them v Not tried to implement ideal architecture v Focused on better test cases v Using the AAA pattern 26 26
28. v AAA pattern defines regular Test Case structure v Arrange Act Assert v Arrange Assert Act Assert 27 Patterns help!
29. Questionable Structures.. v Arrange Act Assert Act v Act Act Act! v Act Assert Act Assert Act Assert … v Act Act Act Act Act Assert 28
30. Tests Plain Java + Page Objects 2 9 No clear Test Steps 29
31. Split test code into parts using AAA pattern Extracted functions into libraries and helpers! 30 https://upload.wikimedia.org/wikipedia/commons/3/32/National_Library_of_Greece_in_Athens.jpg Libraries - sign of civilisation!
32. List of refactored items 1. Moved key Test Inputs into final variables 2. Extracted code into @Step and helper functions 3. Reduced some duplication 4. Moved last ACT step to the @Before method 5. Various Code Style fixes 31
33. Code before After 32
34. Result of Step 1 (refactor for knowledge) v Verified and fixed several tests v Created a few common step libraries v Have knowledge about “what does this test do?” v Spent 3 days onboarding in the project 33
35. Regain confidence in Smoke Tests 34 Step 2
36. Smoke Tests build – not most of the time v Development Culture breaker v Time Killer v Produces a lot of inefficiency v Have to look all the time! 35
37. Let me do something with it! 1. Fix tests! 2. Remove unneeded 3. Place in quarantine! v Broken or unclear 36
38. Quarantine for Broken tests (1) 1) Marked tests to exclude them from builds! 37
39. 2) Added separate build for these tests! 3) Set process for reviewing and fixing quarantine tests 38 Quarantine for Broken tests (2)
40. v 10 GREEN smoke test builds v Muuuuch more confidence! v Moved ~10 tests to Quarantine v Took me 2-3 hours a day over 2 weeks 39 Result of Step 2 (Fix smoke tests) People getting used to GREEN!
41. Establish good maintenance practices Step 3 40
42. 1st Practice Constant Review and Improvement of failure reporting v Test should shout loudly! v What has failed? v Test state? v App state? v If not clear v Wasting time on analysis! 41
43. Review builds during morning coffee 42 v ~1 hour each morning v Which builds Failed? v Was the error clear to me? v Did a few minor fixes to make failure more clear next time
44. Is this error reporting effective? Stacktraceanalysis 43
45. 44 The stack trace was produced by helper function that used Awaitility with some generic failure message Fixing failure reporting
46. Add custom error message 45 2) Exception message handling 1) Custom exception message text
47. Allure report – as good as your @Steps are 46 Is this report good?
48. Actions for report improvement 1. Analyse failure chain 2. Add @Steps on its way 3. Add more descriptive @Step names 47
49. 48 Results (failure reporting improvement) 1. Descriptive error 2. Clear point of failure
50. 49 2nd practice My favourite! 49
51. RESULT of Step 3 (Maintenance practices) v Analysis time decreased v Stability of tests improved v Started catching bugs earlier! 50
52. Go deeper! Refactoring Step 4 51
53. “So he entered the swamp…” v Many people are afraid of going into Legacy Code swamp v And start refactoring! 52 https://blog.intracto.com/paying-technical-debt-how-to-rescue-legacy-code-through-refactoring
54. Spike Agile practice v Perform a time boxed task to understand v The problem v How long it can take to fix 53 https://www.scaledagileframework.com/spikes/
55. Refactoring plan 54 v Unification of logic v Reduction code duplication v Focus on stability v Tests often rely on similar logic to synchronise state v Further improvement of reporting
56. Results of Step 4 (Refactoring) “This code is mine” v Regression is much more stable v I know the sicknesses my tests still have v I did 2 deep dives for 2 sprints v Each took 3 days 55
57. Result of summer 1. Reviewed and fixed number of tests 2. Recovered Smoke Tests 3. Established good maintenance practices 4. Performed deep dive refactoring 5. Holiday! 56
58. It is possible to fix automated tests! What about fixing knowledge gaps? 5 7
59. Raise Code Review culture v Show code to colleagues v Share and defend own values v Ask for feedback v Give feedback on colleagues code v Help to fix issues v Fix yourself sometimes 58
60. QA guild Set up Education and alignment v Topics discussed v How do we define Good Test Automation? v Effective analysis of automated test results v Test Automation Architecture v Tests in the pipeline, CI/CD v Performance testing bots v … v Internal hackathon to play with technologies 59
61. Sharing is caring v Talk! v Discussed the ideas, solutions and concerns. v Developers, PO’s, Scrum Masters, Testers v Go Cross Functional! v Knowledge sharing meetings to understand the development side 60
62. What’s next? 61
63. v Big separated Test Framework is out of date approach v Need to be closer to development! v Good tests are easier to migrate v + you are covered while migrating 62 Long way to go!
64. Don’t become a slave to Test Automation! Find the courage to fight your problems! 63 Final words