Ex Machina: The framework that knows its bugs

Transcript

1. Ex Machina: The framework that knows its bugs 1
2. Aneta Petkova Automation QA Lead @Honeywell Aerospace 2
3. Intro Lets talk performance! 3
4. Intro x x 4
6. Intro 0 2 4 6 8 10 0 50 100 250 500 1000 Person-hours # of tests Monitoring time per tests count Problem: Daily test results monitoring takes very long 6
7. Person Framework Test-issue link Solution: Current state analysis For every failure, do: NO: Log new defect YES: Check status Is issue reported? Check status Log defect 7
8. Build test case – issue link Get defect for test Check defect state Decision Update ticket (optional) Solution: The concept 8
9. Solution: Honeywell tech setup Java framework RestAssured TestNG Atlassian JIRA Rest API 9
10. Solution: Honeywell tech setup Java framework RestAssured TestNG Atlassian JIRA Rest API 10
11. Implementation TestNG Groups: Set defect id as TestNG group for each test where its applicable. Example: @Test(groups = “DEV-3456”) public void newAnalysisTest(){}; 11 Build test case – issue link
12. Implementation TestNG IAnnotationTransformer: Use TestNG IAnnotationTransformer interface to read each defect id. Example: String[] groups = iTestAnnotation.getGroups(); Get defect for test Check defect state Decision Update ticket (optional) 12
13. Implementation TestNG collects all available groups and decides which tests to run for the entire suite, not on a test-by-test basis Get defect for test Check defect state Decision Update ticket (optional) 13
14. Implementation JIRA REST API: Method used: u GET /rest/api/2/issue/{issueIdOrKey} Authentication: u Basic u Have a separate user created for your framework Get defect for test Check defect state Decision Update ticket (optional) 14
15. Implementation RestAssured: Make the API call Example: Response response = given() .auth().preemptive().basic(username, password) .accept(ContentType.JSON) .get(ISSUE_BASE_URL + jiraKey); Get defect for test Check defect state Decision Update ticket (optional) 15
16. Implementation Task lifecycle: Use your bug tracker Get defect for test Check defect state Decision Update ticket (optional) 16
17. Implementation: JIRA workflow Open, Reopened, For implementation New Merged in HEAD HEAD Merged in RC RC Resolved, Closed Done 17
18. Implementation Step 2: Map the defect statuses to your tests state Jira Status Open/ For implementation Fixed in HEAD Fixed in RC/ Resolved/Closed Origin Branch HEAD RC HEAD RC HEAD RC Test Branch HEAD RC HEAD RC HEAD RC HEAD RC HEAD RC HEAD RC Execute ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ Get defect for test Check defect state Decision Update ticket (optional) 18
19. Implementation 19
20. Implementation TestNG: Override some of the TestNG methods that allow actions once a test execution is finished. A few options are available: u afterInvocation() from IInvokedMethodListener interface u onTestFailure()/onTestSuccess() from TestListenerAdapter Get defect for test Check defect state Decision Update ticket (optional) 20
21. Implementation JIRA: Use JIRA’s API endpoints to: u Leave a comment with execution status u POST /rest/api/2/issue/{issueIdOrKey}/comment u Update the issue status u POST /rest/api/2/issue/{issueIdOrKey}/transitions Get defect for test Check defect state Decision Update ticket (optional) 21
22. DEMO TEST DISABLING 22
25. 25 (groups = "DEV-48112")
37. 37 TRUE if branch under test is affected by defect
38. 38 TRUE if branch under test is affected by defect true
39. 39 TRUE if client under test is affected by defect
40. 40 TRUE if client under test is affected by defect true
45. DEMO DEFECT VERIFICATION 45
51. 51 {"body":"Automation Fail: Re-check"}
54. Potential Hazards 54 Overrun your bug tracker with comments Open/close defects Commit for every new defect
55. Actions 55 Overrun your bug tracker with comments Open/close defects Commit for every new defect Check for pre- existing comments Implement strict monitoring process “To go forward you must go back”
56. Alternative Solution One-time only Add custom field in bug tracker On Defect Logging Put test case names for each bug Before test execution 1. Query tickets 2. Get tests from result 3. Exclude tests from execution 56
57. Conclusion 51.0 36.0 18.5 27.0 0 10 20 30 40 50 60 Monitoring Time Jira Duplicates Pass-through defects Time to next delivery Decrease in % after 1st year 57
58. 51.0 36.0 18.5 27.0 0 10 20 30 40 50 60 Monitoring Time Jira Duplicates Pass-through defects Time to next delivery Decrease in % after 1st year Conclusion 58
59. You will not: • Cut your monitoring time by 90% • Start deploying to production ten times a day • Become famous like Elon Musk Conclusion 59 You will: • Reduce test execution time • Make your Jenkins builds great again • Get back valuable time while pretending you’re monitoring tests
60. Thank you! Q&A linkedin.com/in/apetkova aneta.petkova@honeywell.com All product names, logos, and images are property of their respective owners Slides 5,15, 16 – images by Plamena Petkova Slide 62 – image by https://mary147.deviantart.com/ 60 @anetavp (due to popular demand J)