Deep Testing

Transcript

1. FILIPE FREIRE – JUNE 2019 ADIDAS TECH SUMMIT DEEP TESTING
2. ABOUT ME JUN-19 DEEP TESTING 2 Filipe Freire – Software Tester. Developer. Husband. Learner. Open- source software lover and contributor. Adidas projects: worked in .com’s Embellishment & Customization, Mobile App Backend APIs and currently Hype 2.0 Visitmypersonal blog: filfreire.com
3. “WE ARE IN THE BUSINESS OF WINNING WARS (…)” KELSEY GRAMMER AS “GEN. PARTRIDGE” -“THE PENTAGON WARS” 1998
4. Testing a laser-guided bomb: place BOMB above the target
5. Testing heat seeking missile: GLUE tons of hotplates the vehicle
6. THE PENTAGON WARS JUN-19 DEEP TESTING 6 “(I wanted the Army to run realistic tests) to see what would happen to it (the Bradley) and the people inside when it was hit by weapons we could expect on the battlefield. The army did not want to run these tests.” – James Burton Shallow testing example
7. DEEPSEA CHALLENGER JUN-19 DEEP TESTING 7 Ron Allum:“Our solutions draw upon thousands of hours of practical experience at sea and at depth.” Deep testing example
8. “DEEP SEA” ATTITUDE JUN-19 DEEP TESTING 8 deep testing: whatever it is I’m testing I expect to find every single important bug that is out there to find
9. JUN-19 DEEP TESTING 9 “If you don't pursue the questions beneath the surface, you generally won't obtain your money's worth out of your investment in testing.” - Gerald M. Weinberg
10. GOING DEEP JUN-19 DEEP TESTING 10 IDENTITY POSTURE OWNERSHIP NO BULL$%1t
11. IDENTITY JUN-19 DEEP TESTING 11 We trust doctors, mechanics, plumbers… We don’t tell them what/how to do. As a tester… do I own my identity? or I’m owned by it?
12. PRECONCEIVED IDENTITY JUN-19 DEEP TESTING 12 Imposed/preconceived - Gatekeeper - I create tons of test cases or “Automate all the testing and test cases” - I decide and “give the go” for the manager/owner Self-imposed/self-owned - Trustworthy - I hunt for meaningful risks and bugs, using any means necessary - I inform the manager/owner who is paid to decide “go”
13. POSTURE JUN-19 DEEP TESTING 13 Am I ready to test… Anything? Anytime? Right now? Complex stuff? Using any tools?
14. OWNERSHIP JUN-19 DEEP TESTING 14 “I am the go-to tester to talk about testing done in the project I’m with.” Do I own: the tools? Envs? Code reviews? Bug backlog? … Example heuristic: “I get logs to help Devs with debug info?”
15. NO BULL$%1T JUN-19 DEEP TESTING 15 Do I use: the process, the bureaucracy, the test cases, the time (or lack of it), the environment… As an excuse to take zero action? Example heuristic: Can I prepare, without previous notice, a load test for my entire API in under 3 hours?
16. “WHAT BLINDS MY SIGHT?” JUN-19 DEEP TESTING 16 - I don’t have test cases! Is that bad? - I don’t have all the information! So? - I don’t have time! Why? - … I’m blocked until… I 1) understand and 2) do something about it!
17. JUN-19 DEEP TESTING 17 “Sound decisions cannot be made in an environment where truth is hard to find.” - James G. Burton
18. EATING MY DOG‘S FOOD JUN-19 DEEP TESTING 18 Q1 (& Q2) 2019 . Hardcore project and timelines. . Complex architecture & business logic (tons of malicious checks). . 25 business weeks. . Every second counts. test strategy: mindmap test plan: 5 paragraphs 0 test cases >100 meaningful bugs reported >400 avg http req/day 2-4 merges to “master“ / day 7 diff api services to test Release to prod (few hours)= $ >6 figures
19. NUMBER OF BUGS? JUN-19 DEEP TESTING 19 “Counting beans = pointless” Every single bug tells a story through time. “Bugs are not beans” Our prio (testing perspective): find and report the most important ones in the time given… How can I direct my testing to do this? Eating my Dog‘s food
20. JUN-19 DEEP TESTING 20 “(Be careful about) falling victim of equating information quality with data quantity (bug counts, pages of test reports, number of test cases run, and so on)” - Gerald M. Weinberg
21. CONTEXT MATTERS! JUN-19 DEEP TESTING 21 - More critical exploit paths than regular purchase flow; - Persistence tech changed halfway; - Architecture changed halfway; - Lost documentation at one point; - Constant changes and improvements; Eating my Dog‘s food Underlying need to be adaptable. (Tools: postman, curl, wget, kubectl, aws cli, jenkins, docker, bash, charles, sublime, git… … and our “minds”)
22. Meaningful Testing WRAPPING UP JUN-19 DEEP TESTING 22 NO BULL$%1T IDENTITY POSTURE OWNERSHIP Structured approach to testing Context-adapted approach to testing
23. THANK YOU! QUESTIONS?