Software Testing For Business Leaders

Covers:
- Risk
- Landscape of Software Quality Today
- Poor quality affects release times and users
- How testing affects your team culture
- Brutal truths about testing

Transcript

1. S O F T WA R E T E S T I N G F O R B U S I N E S S L E A D E R S S D L M E E T U P
2. L O U I E P E N A F L O R @ R E S T L E S S T H I N K E R W H O A M I ?
3. O V E R V I E W • Today’s software landscape • Risk • Team culture • Tests • Brutal truths • What you can do
4. T O D AY ’ S S O F T WA R E L A N D S C A P E • Extremely low barrier to entry • Many niche software solutions • OSS accelerates development (3rd party libraries) • Talent is hard to find • UX is king
5. Q U A L I T Y
6. Q U A L I T Y • Standards are high • UX is a product differentiator • Focus is difficult • Next generation is quick to change
7. D E F E C T S
8. D E F E C T S • Written by humans • Decrease trust from the user • Create poor user experience • Affects team dynamics
9. R I S K
10. D E - R I S K I N G • Anti-virus • Spam and Phishing • Firewalls and IDS • Back-ups and Disaster Recovery • PCI SOC2/SAS70
11. “Are we de-risking our software projects?”
12. A P P L E S S L / T L S 2 0 1 4 • Nicknamed “goto-fail” • “hidden in plain sight” • Affected millions
13. A P P L E S S L / T L S 2 0 1 4 . . . hashOut.data = hashes + SSL_MD5_DIGEST_LEN; hashOut.length = SSL_SHA1_DIGEST_LEN; if ((err = SSLFreeBuffer(&hashCtx)) != 0) goto fail; if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0) goto fail; goto fail; /* MISTAKE! THIS LINE SHOULD NOT BE HERE */ if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0) goto fail; err = sslRawVerify(...); . . . https://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/
14. P R E V E N T I O N • Stronger coding standards - lint verification • Human code review • Unit Test
15. P R E V E N T E D I N 7 L I N E S O F C O D E /* Verify that a bogus signature does not validate */ - (void) testVerifyRSASignature { SSLBuffer signedParams; SSLAllocBuffer(&signedParams, 32); uint8_t badSignature[128]; memset(badSignature, 0, sizeof(badSignature)); OSStatus err; err = SSLVerifySignedServerKeyExchange(&_ctx, true, signedParams, badSignature, sizeof(badSignature)); XCTAssertNotEqual(err, 0, @"SSLVerifySignedServerKeyExchange() returned success on a completely bogus signature"); } https://github.com/landonf/Testability-CVE-2014-1266/blob/master/TestableSecurityTests/TestableSecurityTests.m#L34
16. – W O R K I N G E F F E C T I V E LY W I T H L E G A C Y C O D E “There is only two ways to write code. Edit and pray or test and verify.”
17. D E - R I S K I N G U P S I D E
18. U P S I D E • Move faster - ship product faster, be more productive • Higher quality software • Happier devs/employees • Happier customers
19. D E F E C T P R E V E N T I O N • Cost of fixing a bug is 8 times greater than writing a test • Lower defect counts lead to faster release schedules • Many companies have defect levels that give them longer schedules than necessary • Poor quality is one of the most common reason for schedule overruns
 
 Software Quality at top speed - http://www.stevemcconnell.com/articles/art04.htm
20. C U LT U R E D E V T E A M
21. C U LT U R E • Engineers are humans first • Writing code is a craft • Autonomy
22. C U LT U R E • Attract & retain top talent • Engineers want to work on cool stuff, not fight fires • No crazy long hours where you make mistakes • More momentum on Sprints in Agile
23. T E S T S
24. – “ U N C L E ” B O B M A R T I N C L E A N C O D E “Legacy code is code without tests.”
25. “Are you writing legacy code in real-time?”
26. T Y P E S O F T E S T S UNIT SERVICE/API UI Number of tests INCREASE IN: - Cost to develop and maintain - Execution time - Possibility of false positives BUT ALSO IN: - Coverage - End user (business) relevance
27. P O S T D E P L O Y M E N T T E S T S • Staging environments are difficult to replicate • Verify deploy • Verify specific data
28. T E S T I N G B A S I C S • Teams should agree on test rules • Fast to run • Barrier to entry in writing a test should be low • Continuous Integration System (Build system) • Watch defect count over time
29. T E S T F R A M E W O R K S • JUnit • PHPUnit • XCTest • MochaJS • Capybara • Cucumber
30. C O D E C O V E R A G E T O O L S • JAVA - EMMA, SONAR • JS - JSCover, Blanket.js • Ruby - SimpleConv • PHP -PHPUnit
31. T E A M Q U A L I T Y A S S U R A N C E
33. Q A T E A M • Different mindset than developers • Think more like a user • Can become great developers
34. Q A T E A M • Developers think they don’t need tests if they have QA team • Can cause developers to be less thorough • Easy for bad habit to be accepted in the culture
35. Q A + T E S T S • Unit tests can catch a lot of defects before it gets to QA • Tests can help developers understand failure scenarios better • QA can be more efficient
36. “What if QA spent less time finding defects?”
37. Q A • Determine if users get true value from your product • Are we meeting the business demands and value props?
38. B R U TA L T R U T H S
39. B R U TA L T R U T H S A B O U T T E S T I N G • Writing a test can be 2x the time it takes to complete a task • Learning how to write code that is easily testable can take time • SLOC for tests can exceed actual product codebase • Tests require maintenance
40. S Q L I T E • SQLite library consists of approximately 112.8 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) • By comparison, the project has 811 times as much test code and test scripts - 91555.1 KSLOC
41. S T R AT E G Y M O V I N G F O R WA R D W H A T Y O U C A N D O
42. I N S I G H T I N T O C U LT U R E “What’s your current code coverage?”
43. S T R AT E G Y M O V I N G F O R WA R D • What do I have right now? • What are our problems? • What is customer facing? • What’s the highest risk? • Understand you are not alone
44. G E T T I N G S TA R T E D • Test runners and harnesses • Continuous Integration • Low barrier for tests • Accept testing into your culture
45. D O N ’ T • Have a month long sprint dedicated to getting tests in your project • Ignore failed tests in the beginning • Give up due to smaller feature releases
46. T I P S • Writing tests is a great way to get any developer familiar with a codebase • Rotate a developer on a team to only writing tests in a sprint • Have fun watching your test count go up • Testing culture scales really well if done right
47. S U M M A RY
48. “How do I convince the rest of the exec team?”
49. C R E AT I N G K P I ' S • Look at existing data • Are defects creating longer release cycles or sprints? • How often are interruptions or added issues are happening each sprint? • Check issue tracker for bugs created vs completed • How long do updates take to pass QA? • Could NPS score be improved?
50. I N V E S T I G AT I N G C U LT U R E • Ask developers • Ask QA
51. “Quality software makes happy users and developers”
52. “Having strong code coverage can help you ship faster”
53. “Having tests helps de-risk the business”
54. L O U I E P E N A F L O R @ R E S T L E S S T H I N K E R Q U E S T I O N S ?