Ideas learned from TOSCA for Open Source Test Automation

Slide 1

Slide 1 text

Vjacheslav Lukashevich Test Automation Meetup, December 2017

Slide 2

Slide 2 text

My Test Automation experience 2 Commercial TOSCA QTP Ranorex Open Source TestComplete FIT, Fitnesse Selenium Cucumber SAHI REST Allure JAVA C# Javascript

Slide 3

Slide 3 text

Projects I’ve worked Enterprise Insurance / Re-insurance Banking Globally, regionally used Lots of fields, rules, forms, workflows, integrations, migrations, data combinations 3

Slide 4

Slide 4 text

What I know about Tricentis and TOSCA https://www.tricentis.com/ Based in Austria, Vienna TOSCA widely used in Austria, Germany, Switzerland companies Heard about usages (from recruiters ) in Australia Ireland 4

Slide 5

Slide 5 text

My story with TRICENTIS TOSCA 5

Slide 6

Slide 6 text

My experience with TOSCA Courses and certifications Certified User Foundation Level Certified User Advanced Level Certified Quality Designer Technical Training Certified Tricentis Professional 6

Slide 7

Slide 7 text

My experience with TOSCA >4 years Test Automation for several projects Consulting projects Integration with other tools and writing extensions 7

Slide 8

Slide 8 text

8 Who needs all this commercial tools features? Excuse me... I have a question!

Slide 9

Slide 9 text

9 Maslow pyramid

Slide 10

Slide 10 text

10 Gojko Adzic – Quality Pyramid https://gojko.net/2012/05/08/redefining-software-quality/

Slide 11

Slide 11 text

11 Physiological needs Ability to steer system, repeatability, portability “Maslow pyramid” for Test Automation Safety needs Reuse patterns, test suites, locator strategies Love/ Belonging CI, test coverage under control Esteem needs CD, project decision driven by tests Self actualization Control over coverage, change impact analysis

Slide 12

Slide 12 text

12 CONTEXT DRIVES NEEDS FULLFILMENT

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

15 Ok. Let’s talk on the features! Use case How to do it in TOSCA! Doing the same without it?

Slide 16

Slide 16 text

16 DYNAMIC TEST CASE DATA ENRICHING YOUR DSL

Slide 17

Slide 17 text

Scenario in TOSCA 17

Slide 18

Slide 18 text

Example – car registration app 18 Enter/Edit Car Data Audi 30/12/2017 Vjacheslav Lukashevich User1 Make: Registration date: Owner: Registered by: JX-1300 License plate No: Cancel Save Car JX-1300 registered

Slide 19

Slide 19 text

Example – car registration app 19 Search Car Data - - Make: Owner: License plate No: Search JX-1234 Make Reg. date Owner Registrar License plate Audi 30/12/2017 Vjacheslav Lukashevich User 1 JX-1234

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Scenario in TOSCA 21

Slide 22

Slide 22 text

TOSCA - Dynamic Data 22

Slide 23

Slide 23 text

TOSCA - Dynamic Data {RND[]} {RANDOMTEXT[]} {DATE}, {MONTH}, {MONTHLAST} {Date+1M+1D}, {Minute} {CALC[5+5]} 23

Slide 24

Slide 24 text

Buffers 24

Slide 25

Slide 25 text

Java – Cucumber implementing data exchange between steps Forked JVM Shared context object With Map<> for storing data 25 Class TestDataContext { private Map testDataMap ; … Class CucumberSteps1 { private TestDataContext testData = TesDataContext.getInstance(); .. testData.put(.. testData.get(.. Class CucumberSteps2 { private TestDataContext testData = TesDataContext.getInstance(); .. testData.put(.. testData.get(..

Slide 26

Slide 26 text

Java – Cucumber implementing data exchange between steps 26

Slide 27

Slide 27 text

Java – Cucumber implementing data exchange between steps Store data in Java Map<> in test context object Get data from Map 27

Slide 28

Slide 28 text

Slide 29

Slide 29 text

29 TDM TEST DATA MANAGEMENT EXTERNAL TEST DATA SOURCE TO HELP YOUR TESTS

Slide 30

Slide 30 text

Example use case Testing in production like environment 30 Register email, Send SMS with One Time Password Provide personal information Processing of new user in external system Set of features for processed user Registration External process Internal process 1 Internal process 2 • requires real phone numbers • relies on 3rd party Can fail in the middle Requires 1 day to complete Can fail Requires all previos steps to be DONE 3-45 minutes 5 minutes ~1 day -

Slide 31

Slide 31 text

Problem Can’t mock/stub external dependcies Difficult to run in a single test case Whole flow takes too long Likely to fail in the middle Long feedback about farest features in the flow 31

Slide 32

Slide 32 text

Solution Use data source to manage test data Database, Web service CRUD operations 32

Slide 33

Slide 33 text

Test data logistics 33 Pre-register accounts DB with user data (Email, Password, State, Modified Date) Test with new account Test with account processed by external system Get 1 account with state “REGISTERED” Get 1 account with state “PERSONAL_INFO_READY” AND Date Updated >1 day

Slide 34

Slide 34 text

TOSCA – TDM support Oracle, MS SQL, DB2, SqLite Designer for test data structure Viewer for test data Set of TOSCA modules to support TDM manipulations in test case 34

Slide 35

Slide 35 text

Test Design TDM 35

Slide 36

Slide 36 text

Slide 37

Slide 37 text

TOSCA Generic TDM modules 37

Slide 38

Slide 38 text

TOSCA Generic TDM modules 38

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Java implementation Database with simple table User, Password, State, Modified Date JDBCTemplate to query this DB Generic Cucumber Steps Get Record with specific state Get Record with specific state and date Set State for the Record 40

Slide 41

Slide 41 text

Cucumber 41

Slide 42

Slide 42 text

Test Data initial state 42

Slide 43

Slide 43 text

Test Data after few test execution rounds 43

Slide 44

Slide 44 text

Result Test can be distributed over time Execution in “waves” 1st set of tests to prepare data 2nd set of tests to move it forward Can be distributed across multiple machines Concurrency handling required Test Data table can support other activities Manual testing Issue analysis 44

Slide 45

Slide 45 text

45 TEMPLATES AND TEST CASE INSTANTIATION

Slide 46

Slide 46 text

Test Case Data Decision Table technique 46

Slide 47

Slide 47 text

Parametrized test case VS Instantiation 47 Data for Multiple Tests Parametrized Test Case Test Template with parameters Test Case instance 1 Data for Multiple Tests Test Case instance 2 N Merge with Generate tests Run Run

Slide 48

Slide 48 text

When is better to instantiate Lots of Forks in the test case logic Scenarios Steps Similarities between scenarios Still some difference DSL’s is not supporting forks Cucumber FIT 48

Slide 49

Slide 49 text

Systems that are good for the generating test Oriented graph, tree 49

Slide 50

Slide 50 text

Test Data from one of my projects 50 Excel sheet with 500 Scenarios 394 Columns (attributes, step description) 1111 Rows

Slide 51

Slide 51 text

Analysis of test data 1st part of all scenarios is very similar! Many scenarios have groups of similar logic! 51

Slide 52

Slide 52 text

Java https://freemarker.apache.org Generate text output (HTML, e-mails, source code, etc.) Based on Templates (FTL) and changing data. 52

Slide 53

Slide 53 text

53 Results of the project 100 small FTL templates VS 500 big individual tests Automated all before the plan Resources spent are in line with budget Acceptance testing done automatically Catched lots of problems Regression test suite is ready and was used for >5 years

Slide 54

Slide 54 text

Adding includes to Cucumber Using FreeMarker 54

Slide 55

Slide 55 text

TOSCA - Template Test Data Source Test Design Excel Template Test Case Re-instantiate Drag and Drop to map 55

Slide 56

Slide 56 text

Test Data Sheet for Template 56

Slide 57

Slide 57 text

TOSCA Template Map fields to Test Design attributes 57

Slide 58

Slide 58 text

TOSCA – Generate Tests “Instantiate” Get first set of test cases “Re-Instantiate” When template changes When data changes 58

Slide 59

Slide 59 text

59 SELECTING TEST DATA COMBINATIONS

Slide 60

Slide 60 text

60 I need to design test cases to test the requirements!

Slide 61

Slide 61 text

61 I would like to design my test cases in the way that I get: •Good coverage •Traceability with requirements •Minimal amount of test cases to save execution time •Good readability and documentation •Easy maintenance

Slide 62

Slide 62 text

Have efficient tests 62 In other words..

Slide 63

Slide 63 text

63 Test er How can I make it?...

Slide 64

Slide 64 text

64 Use the theory of the test case design Testing techniques are already developed by the test professionals and academic people to help in the testing tasks

Slide 65

Slide 65 text

Input Age Gender Output Price 65 Example system Attributes

Slide 66

Slide 66 text

Age • <18 • 18-23 • >23 - <60 • >=60 Gender • Male • Female 66 Equivalence classes  Price • Normal price • +20% • -10% discount • -5% discount • -15% discount • + 15%

Slide 67

Slide 67 text

Age • <18 • 18-23 • >23 - <60 • >=60 Calculation of combinations number 4x2 = 8 combinations for complete coverage If new attributes is added (for example - Car Model) with 2 values 4x2x2 = 16 combinations for complete coverage 67 All combinations  Gender • Male • Female

Slide 68

Slide 68 text

Each attribute is represented Age • <18 • 18-23 • >23 - <60 • >=60 Calculation of combinations number 4 combination to cover all values If new attributes is added (for example - Car Model) with 2 values It will be still 4 combination 68 Orthogonal approach  Gender • Male • Female

Slide 69

Slide 69 text

All combinations Orthogonal approach Combinatio n count Big "Combinatorial explosion" Smallest possible Test Coverage Maximal possible Relatively small Execution time Maximal possible Smallest possible 69 All combinations VS orthogonal approach

Slide 70

Slide 70 text

Slide 71

Slide 71 text

71 Algorithm for creating Test Data 1) Find /Define attributes (Inputs/outputs) Make, Volume, Age, Message 2) Draft equivalence classes Audi, 5L, 2L, >18, <18, No message, Error message 3) Create necessary combinations Happy path, alternative scenarios, negative scenarios

Slide 72

Slide 72 text

TOSCA Test Design Support Separate module (Test Design) Support of combinatorial strategies 72

Slide 73

Slide 73 text

Generating test cases Linear expansion 73 Straight-through Negative Positive, alternative flow

Slide 74

Slide 74 text

Slide 75

Slide 75 text

TOSCA more features for test design Filter all test cases with specific value Highlight the values that are not used 75

Slide 76

Slide 76 text

Benefits of use of combinatorial strategy Simple algorithm for create combinations Creation of combinations is manageable Max amount of test is known in advance 76

Slide 77

Slide 77 text

77 Test Design in Excel Draft Attributes as small tables (colors for happy path, neg.) Create tests manually by rotating around happy path

Slide 78

Slide 78 text

78 Some final notes TOSCA has a lot of great features To improve your productivity To build advanced Testing Solutions Test Automation knowledge is not tool specific Same approaches can be applied with other tools

Slide 79

Slide 79 text

79 Use all opportunities to learn and improve your testing! Thank you! Questions?

Slide 80

Slide 80 text

Materials used in presentation TOSCA Help and User Guide Collection of own screenshots from different TOSCA versions That’s why they look a bit differently  80