Ex Machina: The framework that knows its bugs

E3076b26dd2cd384939694fa5a00948e?s=47 apetkova
October 16, 2017

Ex Machina: The framework that knows its bugs

E3076b26dd2cd384939694fa5a00948e?s=128

apetkova

October 16, 2017
Tweet

Transcript

  1. Ex Machina: The framework that knows its bugs 1

  2. Aneta Petkova Automation QA Lead @Honeywell Aerospace 2

  3. Intro Lets talk performance! 3

  4. Intro x x 4

  5. 5

  6. Intro 0 2 4 6 8 10 0 50 100

    250 500 1000 Person-hours # of tests Monitoring time per tests count Problem: Daily test results monitoring takes very long 6
  7. Person Framework Test-issue link Solution: Current state analysis For every

    failure, do: NO: Log new defect YES: Check status Is issue reported? Check status Log defect 7
  8. Build test case – issue link Get defect for test

    Check defect state Decision Update ticket (optional) Solution: The concept 8
  9. Solution: Honeywell tech setup Java framework RestAssured TestNG Atlassian JIRA

    Rest API 9
  10. Solution: Honeywell tech setup Java framework RestAssured TestNG Atlassian JIRA

    Rest API 10
  11. Implementation TestNG Groups: Set defect id as TestNG group for

    each test where its applicable. Example: @Test(groups = “DEV-3456”) public void newAnalysisTest(){}; 11 Build test case – issue link
  12. Implementation TestNG IAnnotationTransformer: Use TestNG IAnnotationTransformer interface to read each

    defect id. Example: String[] groups = iTestAnnotation.getGroups(); Get defect for test Check defect state Decision Update ticket (optional) 12
  13. Implementation TestNG collects all available groups and decides which tests

    to run for the entire suite, not on a test-by-test basis Get defect for test Check defect state Decision Update ticket (optional) 13
  14. Implementation JIRA REST API: Method used: u GET /rest/api/2/issue/{issueIdOrKey} Authentication:

    u Basic u Have a separate user created for your framework Get defect for test Check defect state Decision Update ticket (optional) 14
  15. Implementation RestAssured: Make the API call Example: Response response =

    given() .auth().preemptive().basic(username, password) .accept(ContentType.JSON) .get(ISSUE_BASE_URL + jiraKey); Get defect for test Check defect state Decision Update ticket (optional) 15
  16. Implementation Task lifecycle: Use your bug tracker Get defect for

    test Check defect state Decision Update ticket (optional) 16
  17. Implementation: JIRA workflow Open, Reopened, For implementation New Merged in

    HEAD HEAD Merged in RC RC Resolved, Closed Done 17
  18. Implementation Step 2: Map the defect statuses to your tests

    state Jira Status Open/ For implementation Fixed in HEAD Fixed in RC/ Resolved/Closed Origin Branch HEAD RC HEAD RC HEAD RC Test Branch HEAD RC HEAD RC HEAD RC HEAD RC HEAD RC HEAD RC Execute ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ Get defect for test Check defect state Decision Update ticket (optional) 18
  19. Implementation 19

  20. Implementation TestNG: Override some of the TestNG methods that allow

    actions once a test execution is finished. A few options are available: u afterInvocation() from IInvokedMethodListener interface u onTestFailure()/onTestSuccess() from TestListenerAdapter Get defect for test Check defect state Decision Update ticket (optional) 20
  21. Implementation JIRA: Use JIRA’s API endpoints to: u Leave a

    comment with execution status u POST /rest/api/2/issue/{issueIdOrKey}/comment u Update the issue status u POST /rest/api/2/issue/{issueIdOrKey}/transitions Get defect for test Check defect state Decision Update ticket (optional) 21
  22. DEMO TEST DISABLING 22

  23. 23

  24. 24

  25. 25 (groups = "DEV-48112")

  26. 26

  27. 27

  28. 28

  29. 29

  30. 30

  31. 31

  32. 32

  33. 33

  34. 34

  35. 35

  36. 36

  37. 37 TRUE if branch under test is affected by defect

  38. 38 TRUE if branch under test is affected by defect

    true
  39. 39 TRUE if client under test is affected by defect

  40. 40 TRUE if client under test is affected by defect

    true
  41. 41

  42. 42

  43. 43

  44. 44

  45. DEMO DEFECT VERIFICATION 45

  46. 46

  47. 47

  48. 48

  49. 49

  50. 50

  51. 51 {"body":"Automation Fail: Re-check"}

  52. 52

  53. 53

  54. Potential Hazards 54 Overrun your bug tracker with comments Open/close

    defects Commit for every new defect
  55. Actions 55 Overrun your bug tracker with comments Open/close defects

    Commit for every new defect Check for pre- existing comments Implement strict monitoring process “To go forward you must go back”
  56. Alternative Solution One-time only Add custom field in bug tracker

    On Defect Logging Put test case names for each bug Before test execution 1. Query tickets 2. Get tests from result 3. Exclude tests from execution 56
  57. Conclusion 51.0 36.0 18.5 27.0 0 10 20 30 40

    50 60 Monitoring Time Jira Duplicates Pass-through defects Time to next delivery Decrease in % after 1st year 57
  58. 51.0 36.0 18.5 27.0 0 10 20 30 40 50

    60 Monitoring Time Jira Duplicates Pass-through defects Time to next delivery Decrease in % after 1st year Conclusion 58
  59. You will not: • Cut your monitoring time by 90%

    • Start deploying to production ten times a day • Become famous like Elon Musk Conclusion 59 You will: • Reduce test execution time • Make your Jenkins builds great again • Get back valuable time while pretending you’re monitoring tests
  60. Thank you! Q&A linkedin.com/in/apetkova aneta.petkova@honeywell.com All product names, logos, and

    images are property of their respective owners Slides 5,15, 16 – images by Plamena Petkova Slide 62 – image by https://mary147.deviantart.com/ 60 @anetavp (due to popular demand J)