Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ex Machina: The framework that knows its bugs

apetkova
October 16, 2017

Ex Machina: The framework that knows its bugs

apetkova

October 16, 2017
Tweet

More Decks by apetkova

Other Decks in Technology

Transcript

  1. Ex Machina:
    The framework that knows its bugs
    1

    View full-size slide

  2. Aneta Petkova
    Automation QA Lead
    @Honeywell Aerospace
    2

    View full-size slide

  3. Intro
    Lets talk
    performance!
    3

    View full-size slide

  4. Intro
    0
    2
    4
    6
    8
    10
    0 50 100 250 500 1000
    Person-hours
    # of tests
    Monitoring time per tests count
    Problem:
    Daily test results
    monitoring
    takes very long
    6

    View full-size slide

  5. Person Framework
    Test-issue link
    Solution: Current state analysis
    For every failure, do:
    NO: Log new defect
    YES: Check status
    Is issue reported?
    Check status
    Log defect
    7

    View full-size slide

  6. Build test case – issue link
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    Solution: The concept 8

    View full-size slide

  7. Solution: Honeywell tech setup
    Java framework
    RestAssured
    TestNG
    Atlassian JIRA
    Rest API
    9

    View full-size slide

  8. Solution: Honeywell tech setup
    Java framework
    RestAssured
    TestNG
    Atlassian JIRA
    Rest API
    10

    View full-size slide

  9. Implementation
    TestNG Groups: Set defect id as TestNG group for each test where its
    applicable.
    Example:
    @Test(groups = “DEV-3456”)
    public void newAnalysisTest(){};
    11
    Build test case – issue link

    View full-size slide

  10. Implementation
    TestNG IAnnotationTransformer: Use TestNG
    IAnnotationTransformer interface to read each defect id.
    Example:
    String[] groups = iTestAnnotation.getGroups();
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    12

    View full-size slide

  11. Implementation
    TestNG collects all available groups and
    decides which tests to run for the entire
    suite, not on a test-by-test basis
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    13

    View full-size slide

  12. Implementation
    JIRA REST API: Method used:
    u GET /rest/api/2/issue/{issueIdOrKey}
    Authentication:
    u Basic
    u Have a separate user created for your framework
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    14

    View full-size slide

  13. Implementation
    RestAssured: Make the API call
    Example:
    Response response = given()
    .auth().preemptive().basic(username,
    password)
    .accept(ContentType.JSON)
    .get(ISSUE_BASE_URL + jiraKey);
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    15

    View full-size slide

  14. Implementation
    Task lifecycle: Use your bug tracker
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    16

    View full-size slide

  15. Implementation: JIRA workflow
    Open, Reopened, For implementation
    New
    Merged in HEAD
    HEAD
    Merged in RC
    RC
    Resolved, Closed
    Done
    17

    View full-size slide

  16. Implementation
    Step 2: Map the defect statuses to your tests state
    Jira
    Status
    Open/
    For implementation
    Fixed in HEAD Fixed in RC/
    Resolved/Closed
    Origin
    Branch
    HEAD RC HEAD RC HEAD RC
    Test
    Branch
    HEAD RC HEAD RC HEAD RC HEAD RC HEAD RC HEAD RC
    Execute ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    18

    View full-size slide

  17. Implementation 19

    View full-size slide

  18. Implementation
    TestNG: Override some of the TestNG methods that allow
    actions once a test execution is finished. A few options are
    available:
    u afterInvocation() from IInvokedMethodListener interface
    u onTestFailure()/onTestSuccess() from TestListenerAdapter
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    20

    View full-size slide

  19. Implementation
    JIRA: Use JIRA’s API endpoints to:
    u Leave a comment with execution status
    u POST /rest/api/2/issue/{issueIdOrKey}/comment
    u Update the issue status
    u POST /rest/api/2/issue/{issueIdOrKey}/transitions
    Get
    defect for
    test
    Check
    defect
    state
    Decision
    Update
    ticket
    (optional)
    21

    View full-size slide

  20. DEMO
    TEST DISABLING
    22

    View full-size slide

  21. 25
    (groups = "DEV-48112")

    View full-size slide

  22. 37
    TRUE if branch under test
    is affected by defect

    View full-size slide

  23. 38
    TRUE if branch under test
    is affected by defect
    true

    View full-size slide

  24. 39
    TRUE if client under test is
    affected by defect

    View full-size slide

  25. 40
    TRUE if client under test is
    affected by defect
    true

    View full-size slide

  26. DEMO
    DEFECT VERIFICATION
    45

    View full-size slide

  27. 51
    {"body":"Automation Fail: Re-check"}

    View full-size slide

  28. Potential Hazards 54
    Overrun your bug
    tracker with
    comments
    Open/close
    defects
    Commit for every
    new defect

    View full-size slide

  29. Actions 55
    Overrun your bug
    tracker with
    comments
    Open/close
    defects
    Commit for every
    new defect
    Check for pre-
    existing comments
    Implement strict
    monitoring process
    “To go forward you
    must go back”

    View full-size slide

  30. Alternative Solution
    One-time only
    Add custom field
    in bug tracker
    On Defect
    Logging
    Put test case
    names for each
    bug
    Before test
    execution
    1. Query tickets
    2. Get tests from
    result
    3. Exclude tests
    from execution
    56

    View full-size slide

  31. Conclusion
    51.0
    36.0
    18.5
    27.0
    0
    10
    20
    30
    40
    50
    60
    Monitoring Time Jira Duplicates Pass-through
    defects
    Time to next delivery
    Decrease in % after 1st year
    57

    View full-size slide

  32. 51.0
    36.0
    18.5
    27.0
    0
    10
    20
    30
    40
    50
    60
    Monitoring Time Jira Duplicates Pass-through
    defects
    Time to next delivery
    Decrease in % after 1st year
    Conclusion 58

    View full-size slide

  33. You will not:
    • Cut your monitoring time by 90%
    • Start deploying to production ten times a day
    • Become famous like Elon Musk
    Conclusion 59
    You will:
    • Reduce test execution time
    • Make your Jenkins builds great again
    • Get back valuable time while pretending you’re
    monitoring tests

    View full-size slide

  34. Thank you!
    Q&A
    linkedin.com/in/apetkova
    [email protected]
    All product names, logos, and images are property of their respective owners
    Slides 5,15, 16 – images by Plamena Petkova
    Slide 62 – image by https://mary147.deviantart.com/
    60
    @anetavp (due to popular demand J)

    View full-size slide