Ideas learned from TOSCA for Open Source Test Automation

In a past I had worked few years on test automation using TOSCA and passed multiple certifications with this tool. This was one of most valuable periods from automation experience in my career, because I learned a lot about complex test automation solutions and framework architecture.
TOSCA has extensive collection of really nice features and ideas, that you can learn from and reuse in your own projects and tools.

In this presentation I’m covering:
* cool features and how they can be applied in open source projects
* test automation expertise areas and knowledge in general.

Transcript

1. Vjacheslav Lukashevich Test Automation Meetup, December 2017
2. My Test Automation experience 2 Commercial TOSCA QTP Ranorex Open Source TestComplete FIT, Fitnesse Selenium Cucumber SAHI REST Allure JAVA C# Javascript
3. Projects I’ve worked Enterprise Insurance / Re-insurance Banking Globally, regionally used Lots of fields, rules, forms, workflows, integrations, migrations, data combinations 3
4. What I know about Tricentis and TOSCA https://www.tricentis.com/ Based in Austria, Vienna TOSCA widely used in Austria, Germany, Switzerland companies Heard about usages (from recruiters ) in Australia Ireland 4
5. My story with TRICENTIS TOSCA 5
6. My experience with TOSCA Courses and certifications Certified User Foundation Level Certified User Advanced Level Certified Quality Designer Technical Training Certified Tricentis Professional 6
7. My experience with TOSCA >4 years Test Automation for several projects Consulting projects Integration with other tools and writing extensions 7
8. 8 Who needs all this commercial tools features? Excuse me... I have a question!
9. 9 Maslow pyramid
10. 10 Gojko Adzic – Quality Pyramid https://gojko.net/2012/05/08/redefining-software-quality/
11. 11 Physiological needs Ability to steer system, repeatability, portability “Maslow pyramid” for Test Automation Safety needs Reuse patterns, test suites, locator strategies Love/ Belonging CI, test coverage under control Esteem needs CD, project decision driven by tests Self actualization Control over coverage, change impact analysis
12. 12 CONTEXT DRIVES NEEDS FULLFILMENT
15. 15 Ok. Let’s talk on the features! Use case How to do it in TOSCA! Doing the same without it?
16. 16 DYNAMIC TEST CASE DATA ENRICHING YOUR DSL
17. Scenario in TOSCA 17
18. Example – car registration app 18 Enter/Edit Car Data Audi 30/12/2017 Vjacheslav Lukashevich User1 Make: Registration date: Owner: Registered by: JX-1300 License plate No: Cancel Save Car JX-1300 registered
19. Example – car registration app 19 Search Car Data - - Make: Owner: License plate No: Search JX-1234 Make Reg. date Owner Registrar License plate Audi 30/12/2017 Vjacheslav Lukashevich User 1 JX-1234
21. Scenario in TOSCA 21
22. TOSCA - Dynamic Data 22
23. TOSCA - Dynamic Data {RND[<Number of Digits>]} {RANDOMTEXT[<Number of characters>]} {DATE}, {MONTH}, {MONTHLAST} {Date+1M+1D}, {Minute} {CALC[5+5]} 23
24. Buffers 24
25. Java – Cucumber implementing data exchange between steps Forked JVM Shared context object With Map<> for storing data 25 Class TestDataContext { private Map<String,String> testDataMap ; … Class CucumberSteps1 { private TestDataContext testData = TesDataContext.getInstance(); .. testData.put(.. testData.get(.. Class CucumberSteps2 { private TestDataContext testData = TesDataContext.getInstance(); .. testData.put(.. testData.get(..
26. Java – Cucumber implementing data exchange between steps 26
27. Java – Cucumber implementing data exchange between steps Store data in Java Map<> in test context object Get data from Map 27
29. 29 TDM TEST DATA MANAGEMENT EXTERNAL TEST DATA SOURCE TO HELP YOUR TESTS
30. Example use case Testing in production like environment 30 Register email, Send SMS with One Time Password Provide personal information Processing of new user in external system Set of features for processed user Registration External process Internal process 1 Internal process 2 • requires real phone numbers • relies on 3rd party Can fail in the middle Requires 1 day to complete Can fail Requires all previos steps to be DONE 3-45 minutes 5 minutes ~1 day -
31. Problem Can’t mock/stub external dependcies Difficult to run in a single test case Whole flow takes too long Likely to fail in the middle Long feedback about farest features in the flow 31
32. Solution Use data source to manage test data Database, Web service CRUD operations 32
33. Test data logistics 33 Pre-register accounts DB with user data (Email, Password, State, Modified Date) Test with new account Test with account processed by external system Get 1 account with state “REGISTERED” Get 1 account with state “PERSONAL_INFO_READY” AND Date Updated >1 day
34. TOSCA – TDM support Oracle, MS SQL, DB2, SqLite Designer for test data structure Viewer for test data Set of TOSCA modules to support TDM manipulations in test case 34
35. Test Design TDM 35
37. TOSCA Generic TDM modules 37
38. TOSCA Generic TDM modules 38
40. Java implementation Database with simple table User, Password, State, Modified Date JDBCTemplate to query this DB Generic Cucumber Steps Get Record with specific state Get Record with specific state and date Set State for the Record 40
41. Cucumber 41
42. Test Data initial state 42
43. Test Data after few test execution rounds 43
44. Result Test can be distributed over time Execution in “waves” 1st set of tests to prepare data 2nd set of tests to move it forward Can be distributed across multiple machines Concurrency handling required Test Data table can support other activities Manual testing Issue analysis 44
45. 45 TEMPLATES AND TEST CASE INSTANTIATION
46. Test Case Data Decision Table technique 46
47. Parametrized test case VS Instantiation 47 Data for Multiple Tests Parametrized Test Case Test Template with parameters Test Case instance 1 Data for Multiple Tests Test Case instance 2 N Merge with Generate tests Run Run
48. When is better to instantiate Lots of Forks in the test case logic Scenarios Steps Similarities between scenarios Still some difference DSL’s is not supporting forks Cucumber FIT 48
49. Systems that are good for the generating test Oriented graph, tree 49
50. Test Data from one of my projects 50 Excel sheet with 500 Scenarios 394 Columns (attributes, step description) 1111 Rows
51. Analysis of test data 1st part of all scenarios is very similar! Many scenarios have groups of similar logic! 51
52. Java https://freemarker.apache.org Generate text output (HTML, e-mails, source code, etc.) Based on Templates (FTL) and changing data. 52
53. 53 Results of the project 100 small FTL templates VS 500 big individual tests Automated all before the plan Resources spent are in line with budget Acceptance testing done automatically Catched lots of problems Regression test suite is ready and was used for >5 years
54. Adding includes to Cucumber Using FreeMarker 54
55. TOSCA - Template Test Data Source Test Design Excel Template Test Case Re-instantiate Drag and Drop to map 55
56. Test Data Sheet for Template 56
57. TOSCA Template Map fields to Test Design attributes 57
58. TOSCA – Generate Tests “Instantiate” Get first set of test cases “Re-Instantiate” When template changes When data changes 58
59. 59 SELECTING TEST DATA COMBINATIONS
60. 60 I need to design test cases to test the requirements!
61. 61 I would like to design my test cases in the way that I get: •Good coverage •Traceability with requirements •Minimal amount of test cases to save execution time •Good readability and documentation •Easy maintenance
62. Have efficient tests 62 In other words..
63. 63 Test er How can I make it?...
64. 64 Use the theory of the test case design Testing techniques are already developed by the test professionals and academic people to help in the testing tasks
65. Input Age Gender Output Price 65 Example system Attributes
66. Age • <18 • 18-23 • >23 - <60 • >=60 Gender • Male • Female 66 Equivalence classes  Price • Normal price • +20% • -10% discount • -5% discount • -15% discount • + 15%
67. Age • <18 • 18-23 • >23 - <60 • >=60 Calculation of combinations number 4x2 = 8 combinations for complete coverage If new attributes is added (for example - Car Model) with 2 values 4x2x2 = 16 combinations for complete coverage 67 All combinations  Gender • Male • Female
68. Each attribute is represented Age • <18 • 18-23 • >23 - <60 • >=60 Calculation of combinations number 4 combination to cover all values If new attributes is added (for example - Car Model) with 2 values It will be still 4 combination 68 Orthogonal approach  Gender • Male • Female
69. All combinations Orthogonal approach Combinatio n count Big "Combinatorial explosion" Smallest possible Test Coverage Maximal possible Relatively small Execution time Maximal possible Smallest possible 69 All combinations VS orthogonal approach
71. 71 Algorithm for creating Test Data 1) Find /Define attributes (Inputs/outputs) Make, Volume, Age, Message 2) Draft equivalence classes Audi, 5L, 2L, >18, <18, No message, Error message 3) Create necessary combinations Happy path, alternative scenarios, negative scenarios
72. TOSCA Test Design Support Separate module (Test Design) Support of combinatorial strategies 72
73. Generating test cases Linear expansion 73 Straight-through Negative Positive, alternative flow
75. TOSCA more features for test design Filter all test cases with specific value Highlight the values that are not used 75
76. Benefits of use of combinatorial strategy Simple algorithm for create combinations Creation of combinations is manageable Max amount of test is known in advance 76
77. 77 Test Design in Excel Draft Attributes as small tables (colors for happy path, neg.) Create tests manually by rotating around happy path
78. 78 Some final notes TOSCA has a lot of great features To improve your productivity To build advanced Testing Solutions Test Automation knowledge is not tool specific Same approaches can be applied with other tools
79. 79 Use all opportunities to learn and improve your testing! Thank you! Questions?
80. Materials used in presentation TOSCA Help and User Guide Collection of own screenshots from different TOSCA versions That’s why they look a bit differently  80