Experience of generating Automated Tests for integration testing

Slide 1

Slide 1 text

Experience of generating Automated Tests for integration testing Vjacheslav Lukashevich TAPOST 2015

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Story Project and testing mission Testing and test automation challenge Solution – Automated Tests Generation Automation sub-project Project results Lessons I have learned 3

Slide 4

Slide 4 text

PROJECT AND TESTING MISSION 4

Slide 5

Slide 5 text

Contract Management system project 1st Release Application details Global Reinsurance business Intranet Web application Data capture and management Replacing and extending part of existing system Project setup Agile development, 2 weeks iterations Technology Web frontend, Java Server Side EJB integrations Test Automation tools Framework for integrated testing (FIT) 5

Slide 6

Slide 6 text

IT landscape transformation project 6

Slide 7

Slide 7 text

The reinsurance contracts are BIG Each contract can have a lot of data There are a lots of data variations >100 business rules for data 7

Slide 8

Slide 8 text

Mission – Test Integration between systems Whole contract object is sent via integration interface 8

Slide 9

Slide 9 text

Integration testing plan 9

Slide 10

Slide 10 text

10 Lesson - Legacy system gives lot of opportunities Different test oracles exist! Working system Documentation Testing artifacts

Slide 11

Slide 11 text

11 Approach for integration testing

Slide 12

Slide 12 text

Contract Management regression test suite received from existing team Denormalized data, 1 scenario = one to many rows 12

Slide 13

Slide 13 text

PROBLEMS AND CHALLENGES 13

Slide 14

Slide 14 text

Scenarios have LOTS of data.. 500 Scenarios 394 Columns 1111 Rows 14

Slide 15

Slide 15 text

Problem#1 Each scenario is BIG Some are VERY BIG…. We need to execute lots of steps to create and fill contract screens 15

Slide 16

Slide 16 text

16 A little bit about FIT tool

Slide 17

Slide 17 text

Framework for Integrated Test (FIT) 17 Design time Execution results

Slide 18

Slide 18 text

FIT Scripts are business scenarios No programming logic inside script (IF, loops) Top down execution Values are mostly hardcoded inside FIT script Reusable parts can be parametrised 18

Slide 19

Slide 19 text

FIT scripts development Test engineer compose FIT script using any suitable HTML editor 19

Slide 20

Slide 20 text

20 Too many Scenarios Steps needed 300-1500 steps per scenario Data variations Parametrisation required Problem#2 Scripts volume is too BIG

Slide 21

Slide 21 text

Problem#3 No control over automation We can’t support this amount of tests Tests FAIL constantly and there is no confidence in our testing 21

Slide 22

Slide 22 text

22 How test automation status looked like on our daily standup 22

Slide 23

Slide 23 text

SOLUTION 23

Slide 24

Slide 24 text

It could be difficult to step out when you are deeply in process 24

Slide 25

Slide 25 text

VS Maze from different viewpoints 25

Slide 26

Slide 26 text

Analysis of test data 1st part of all scenarios is very similar! Many scenarios have groups of similar logic! 26

Slide 27

Slide 27 text

HTML generation from templates is common approach in industry FITs are HTML! 27

Slide 28

Slide 28 text

Let’s automate the test automation creation! 28

Slide 29

Slide 29 text

29 Developer Tester Team assigned to prototype FIT generation

Slide 30

Slide 30 text

Template engine – FMPP FMPP - FreeMarker-based text file PreProcessor General-purpose text file preprocessor tool. Java Open source page 30 30

Slide 31

Slide 31 text

Test Data related tasks Analyze scenarios and define test generation rules Align test data for generation Align column names Align field values Add additional fields to support generation logic Scenario group ID Object ID in the contract graph Action flags 31

Slide 32

Slide 32 text

Generation rules # IF THEN 1 First row of a scenario Fill Treaty Header, Additional details, Non-Contractual details screens with field values from this row 2 Field value = "-" Consider this as empty value 3 Last row of the scenario Re-save Claims and Segments for all segments/riders ever used in the scenario as "Completed" or "Draft“ based on the values in the ClaimsCompleted for Claims or AdminCompleted for Admin 4 BenefitType has list of values separated by comma (,) Consider this as a multiple Benefit values for the Additional Benefit multi value list .. …. … 32

Slide 33

Slide 33 text

Aligned data 33

Slide 34

Slide 34 text

Generator implementation tasks Create templates for screens Define composition/chains of templates for scenario group Implement test generation rules 34

Slide 35

Slide 35 text

FMPP example template_group_11.ftl createNewTreatyHeader.ftl 35

Slide 36

Slide 36 text

Test Generation 36

Slide 37

Slide 37 text

Test automation approach page 37 MODEL BASED TESTING + DATA DRIVEN TESTING HTML TEMPLATES + EXCEL TEST DATA 37

Slide 38

Slide 38 text

AUTOMATION SUB-PROJECT 38

Slide 39

Slide 39 text

Dedicated team Test lead Tester Developer Development process Issue tracking Scope management Testing process Execution on local and integrated environment Automation sub project 39

Slide 40

Slide 40 text

40 Tester and developer role

Slide 41

Slide 41 text

Automation Sub Project 41

Slide 42

Slide 42 text

42 Executing of tests Local environment Debugging of automated tests Runs fast Integrated environment Goal to find problems with integration After problem fixes we were re-executing failed scenarios

Slide 43

Slide 43 text

Findings Defects in application Errors on the interface side Errors in the automation code Errors in the specifications Errors in legacy application Functionality Updates in CM (Maintenance) 43

Slide 44

Slide 44 text

PROJECT RESULTS 44

Slide 45

Slide 45 text

45 Automation is manageable Scope known Implementation controlled Maintenance controlled Tests are up to date to specific version Constant and intensive feedback from testing

Slide 46

Slide 46 text

46 Result for 1st Release 100 small templates VS 500 big individual tests Automated all before the plan Resources spent are in line with budget Acceptance testing done automatically Catched lots of problems Regression test suite is ready

Slide 47

Slide 47 text

47 Results after several releases Tests are still working Maintenance done with reasonable resources Suite was slightly extended with new tests Approach was adopted in other projects

Slide 48

Slide 48 text

WHAT I HAVE LEARNED 48

Slide 49

Slide 49 text

Lesson #1 Test Generation is VERY effective for applications that can be represented as a workflow graph 49

Slide 50

Slide 50 text

Scenario graph sample from our project 50

Slide 51

Slide 51 text

Lesson #2 Cooperation of developer and tester for Test Automation implementation is VERY productive! 51

Slide 52

Slide 52 text

Lesson #3 Managing automation as sub-project is VERY GOOD model for dealing with complex automation tasks People assigned Planning, scope, progress managed 52

Slide 53

Slide 53 text

Lesson #4 Managing test data in excel format is VERY convenient! Find tests that use the specific data for problem analysis Easy maintenance of test data Add new columns Bulk updates 53

Slide 54

Slide 54 text

Lesson #5 Keep it simple and documented! Keep business logic on the test data Keep test data in simple form Keep test generation logic as simple as possible Document the solution and approach 54

Slide 55

Slide 55 text

Lesson #6 Service/API based automation allows to execute VERY BIG volume of testing 55

Slide 56

Slide 56 text

…AND MANY OTHER LESSONS.. 56 Sometimes later…

Slide 57

Slide 57 text

57 Questions? Use ALL OPPORTUNITIES to make your testing better and GOOD LUCK with your TEST AUTOMATION