Deep Testing

Transcript

1. FILIPE FREIRE – JUNE 2019
   ADIDAS TECH SUMMIT
   DEEP TESTING
   

   View Slide

2. ABOUT ME
   JUN-19
   DEEP TESTING 2
   Filipe Freire – Software Tester.
   Developer. Husband. Learner. Open-
   source software lover and contributor.
   Adidas projects: worked in .com’s
   Embellishment & Customization, Mobile
   App Backend APIs and currently Hype 2.0
   Visitmypersonal blog: filfreire.com
   

   View Slide

3. “WE ARE IN THE BUSINESS
   OF WINNING WARS (…)”
   KELSEY GRAMMER AS “GEN. PARTRIDGE” -“THE PENTAGON WARS” 1998
   

   View Slide

4. Testing a laser-guided bomb:
   place BOMB above the target
   

   View Slide

5. Testing heat seeking missile:
   GLUE tons of hotplates the vehicle
   

   View Slide

6. THE PENTAGON WARS
   JUN-19
   DEEP TESTING 6
   “(I wanted the Army to run realistic tests) to see what
   would happen to it (the Bradley) and the people inside
   when it was hit by weapons we could expect on the
   battlefield. The army did not want to run these tests.”
   – James Burton
   Shallow testing
   example
   

   View Slide

7. DEEPSEA CHALLENGER
   JUN-19
   DEEP TESTING 7
   Ron Allum:“Our solutions draw upon
   thousands of hours of practical
   experience at sea and at depth.”
   Deep testing
   example
   

   View Slide

8. “DEEP SEA” ATTITUDE
   JUN-19
   DEEP TESTING 8
   deep testing: whatever it is I’m testing I
   expect to find every single important bug
   that is out there to find
   

   View Slide

9. JUN-19
   DEEP TESTING 9
   “If you don't pursue the questions
   beneath the surface, you generally
   won't obtain your money's worth out of
   your investment in testing.”
   - Gerald M. Weinberg
   

   View Slide

10. GOING DEEP
    JUN-19
    DEEP TESTING 10
    IDENTITY
    POSTURE
    OWNERSHIP
    NO
    BULL$%1t
    

    View Slide

11. IDENTITY
    JUN-19
    DEEP TESTING 11
    We trust doctors, mechanics,
    plumbers…
    We don’t tell them what/how
    to do.
    As a tester…
    do I own my identity?
    or I’m owned by it?
    

    View Slide

12. PRECONCEIVED IDENTITY
    JUN-19
    DEEP TESTING 12
    Imposed/preconceived
    - Gatekeeper
    - I create tons of test
    cases or “Automate all
    the testing and test
    cases”
    - I decide and “give the
    go” for the
    manager/owner
    Self-imposed/self-owned
    - Trustworthy
    - I hunt for meaningful
    risks and bugs, using any
    means necessary
    - I inform the
    manager/owner who is
    paid to decide “go”
    

    View Slide

13. POSTURE
    JUN-19
    DEEP TESTING 13
    Am I ready to test…
    Anything?
    Anytime?
    Right now?
    Complex stuff?
    Using any tools?
    

    View Slide

14. OWNERSHIP
    JUN-19
    DEEP TESTING 14
    “I am the go-to tester to talk
    about testing done in the project
    I’m with.”
    Do I own: the tools? Envs? Code
    reviews? Bug backlog? …
    Example heuristic:
    “I get logs to help Devs with
    debug info?”
    

    View Slide

15. NO BULL$%1T
    JUN-19
    DEEP TESTING 15
    Do I use: the process, the bureaucracy,
    the test cases, the time (or lack of it), the
    environment…
    As an excuse to take zero action?
    Example heuristic:
    Can I prepare, without previous notice, a
    load test for my entire API in under 3 hours?
    

    View Slide

16. “WHAT BLINDS MY SIGHT?”
    JUN-19
    DEEP TESTING 16
    - I don’t have test cases! Is that bad?
    - I don’t have all the information! So?
    - I don’t have time! Why?
    - …
    I’m blocked until… I 1) understand and 2) do
    something about it!
    

    View Slide

17. JUN-19
    DEEP TESTING 17
    “Sound decisions cannot be made in an
    environment where truth is hard to
    find.”
    - James G. Burton
    

    View Slide

18. EATING MY DOG‘S FOOD
    JUN-19
    DEEP TESTING 18
    Q1 (& Q2) 2019
    . Hardcore project and
    timelines.
    . Complex architecture &
    business logic (tons of
    malicious checks).
    . 25 business weeks.
    . Every second counts.
    test strategy: mindmap
    test plan: 5 paragraphs
    0 test cases
    >100 meaningful bugs reported
    >400 avg http req/day
    2-4 merges to “master“ / day
    7 diff api services to test
    Release to prod (few hours)= $ >6 figures
    

    View Slide

19. NUMBER OF BUGS?
    JUN-19
    DEEP TESTING 19
    “Counting beans = pointless”
    Every single bug tells a story
    through time. “Bugs are not beans”
    Our prio (testing perspective): find
    and report the most important
    ones in the time given…
    How can I direct my testing to do
    this?
    Eating my
    Dog‘s food
    

    View Slide

20. JUN-19
    DEEP TESTING 20
    “(Be careful about) falling victim of
    equating information quality with data
    quantity (bug counts, pages of test
    reports, number of test cases run, and
    so on)”
    - Gerald M. Weinberg
    

    View Slide

21. CONTEXT MATTERS!
    JUN-19
    DEEP TESTING 21
    - More critical exploit paths than
    regular purchase flow;
    - Persistence tech changed halfway;
    - Architecture changed halfway;
    - Lost documentation at one point;
    - Constant changes and
    improvements;
    Eating my
    Dog‘s food
    Underlying need to be adaptable.
    (Tools: postman, curl, wget, kubectl, aws cli, jenkins, docker, bash, charles, sublime, git…
    … and our “minds”)
    

    View Slide

22. Meaningful
    Testing
    WRAPPING UP
    JUN-19
    DEEP TESTING 22
    NO BULL$%1T
    IDENTITY
    POSTURE
    OWNERSHIP
    Structured
    approach to testing
    Context-adapted
    approach to testing
    

    View Slide

23. THANK YOU!
    QUESTIONS?
    

    View Slide