Software Testing For Business Leaders

Software Testing For Business Leaders

- Risk
- Landscape of Software Quality Today
- Poor quality affects release times and users
- How testing affects your team culture
- Brutal truths about testing



January 15, 2016


  1. S O F T WA R E T E S

    T I N G F O R B U S I N E S S L E A D E R S S D L M E E T U P
  2. L O U I E P E N A F

    L O R @ R E S T L E S S T H I N K E R W H O A M I ?
  3. O V E R V I E W • Today’s

    software landscape • Risk • Team culture • Tests • Brutal truths • What you can do
  4. T O D AY ’ S S O F T

    WA R E L A N D S C A P E • Extremely low barrier to entry • Many niche software solutions • OSS accelerates development (3rd party libraries) • Talent is hard to find • UX is king
  5. Q U A L I T Y

  6. Q U A L I T Y • Standards are

    high • UX is a product differentiator • Focus is difficult • Next generation is quick to change
  7. D E F E C T S

  8. D E F E C T S • Written by

    humans • Decrease trust from the user • Create poor user experience • Affects team dynamics
  9. R I S K

  10. D E - R I S K I N G

    • Anti-virus • Spam and Phishing • Firewalls and IDS • Back-ups and Disaster Recovery • PCI SOC2/SAS70
  11. “Are we de-risking our software projects?”

  12. A P P L E S S L / T

    L S 2 0 1 4 • Nicknamed “goto-fail” • “hidden in plain sight” • Affected millions
  13. A P P L E S S L / T

    L S 2 0 1 4 . . . = hashes + SSL_MD5_DIGEST_LEN; hashOut.length = SSL_SHA1_DIGEST_LEN; if ((err = SSLFreeBuffer(&hashCtx)) != 0) goto fail; if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0) goto fail; goto fail; /* MISTAKE! THIS LINE SHOULD NOT BE HERE */ if ((err =, &hashOut)) != 0) goto fail; err = sslRawVerify(...); . . .
  14. P R E V E N T I O N

    • Stronger coding standards - lint verification • Human code review • Unit Test
  15. P R E V E N T E D I

    N 7 L I N E S O F C O D E /* Verify that a bogus signature does not validate */ - (void) testVerifyRSASignature { SSLBuffer signedParams; SSLAllocBuffer(&signedParams, 32); uint8_t badSignature[128]; memset(badSignature, 0, sizeof(badSignature)); OSStatus err; err = SSLVerifySignedServerKeyExchange(&_ctx, true, signedParams, badSignature, sizeof(badSignature)); XCTAssertNotEqual(err, 0, @"SSLVerifySignedServerKeyExchange() returned success on a completely bogus signature"); }
  16. – W O R K I N G E F

    F E C T I V E LY W I T H L E G A C Y C O D E “There is only two ways to write code. Edit and pray or test and verify.”
  17. D E - R I S K I N G

    U P S I D E
  18. U P S I D E • Move faster -

    ship product faster, be more productive • Higher quality software • Happier devs/employees • Happier customers
  19. D E F E C T P R E V

    E N T I O N • Cost of fixing a bug is 8 times greater than writing a test • Lower defect counts lead to faster release schedules • Many companies have defect levels that give them longer schedules than necessary • Poor quality is one of the most common reason for schedule overruns
 Software Quality at top speed -
  20. C U LT U R E D E V T

    E A M
  21. C U LT U R E • Engineers are humans

    first • Writing code is a craft • Autonomy
  22. C U LT U R E • Attract & retain

    top talent • Engineers want to work on cool stuff, not fight fires • No crazy long hours where you make mistakes • More momentum on Sprints in Agile
  23. T E S T S

  24. – “ U N C L E ” B O

    B M A R T I N C L E A N C O D E “Legacy code is code without tests.”
  25. “Are you writing legacy code in real-time?”

  26. T Y P E S O F T E S

    T S UNIT SERVICE/API UI Number of tests INCREASE IN: - Cost to develop and maintain - Execution time - Possibility of false positives BUT ALSO IN: - Coverage - End user (business) relevance
  27. P O S T D E P L O Y

    M E N T T E S T S • Staging environments are difficult to replicate • Verify deploy • Verify specific data
  28. T E S T I N G B A S

    I C S • Teams should agree on test rules • Fast to run • Barrier to entry in writing a test should be low • Continuous Integration System (Build system) • Watch defect count over time
  29. T E S T F R A M E W

    O R K S • JUnit • PHPUnit • XCTest • MochaJS • Capybara • Cucumber
  30. C O D E C O V E R A

    G E T O O L S • JAVA - EMMA, SONAR • JS - JSCover, Blanket.js • Ruby - SimpleConv • PHP -PHPUnit
  31. T E A M Q U A L I T

    Y A S S U R A N C E
  32. None
  33. Q A T E A M • Different mindset than

    developers • Think more like a user • Can become great developers
  34. Q A T E A M • Developers think they

    don’t need tests if they have QA team • Can cause developers to be less thorough • Easy for bad habit to be accepted in the culture
  35. Q A + T E S T S • Unit

    tests can catch a lot of defects before it gets to QA • Tests can help developers understand failure scenarios better • QA can be more efficient
  36. “What if QA spent less time finding defects?”

  37. Q A • Determine if users get true value from

    your product • Are we meeting the business demands and value props?
  38. B R U TA L T R U T H

  39. B R U TA L T R U T H

    S A B O U T T E S T I N G • Writing a test can be 2x the time it takes to complete a task • Learning how to write code that is easily testable can take time • SLOC for tests can exceed actual product codebase • Tests require maintenance
  40. S Q L I T E • SQLite library consists

    of approximately 112.8 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) • By comparison, the project has 811 times as much test code and test scripts - 91555.1 KSLOC
  41. S T R AT E G Y M O V

    I N G F O R WA R D W H A T Y O U C A N D O
  42. I N S I G H T I N T

    O C U LT U R E “What’s your current code coverage?”
  43. S T R AT E G Y M O V

    I N G F O R WA R D • What do I have right now? • What are our problems? • What is customer facing? • What’s the highest risk? • Understand you are not alone
  44. G E T T I N G S TA R

    T E D • Test runners and harnesses • Continuous Integration • Low barrier for tests • Accept testing into your culture
  45. D O N ’ T • Have a month long

    sprint dedicated to getting tests in your project • Ignore failed tests in the beginning • Give up due to smaller feature releases
  46. T I P S • Writing tests is a great

    way to get any developer familiar with a codebase • Rotate a developer on a team to only writing tests in a sprint • Have fun watching your test count go up • Testing culture scales really well if done right
  47. S U M M A RY

  48. “How do I convince the rest of the exec team?”

  49. C R E AT I N G K P I

    ' S • Look at existing data • Are defects creating longer release cycles or sprints? • How often are interruptions or added issues are happening each sprint? • Check issue tracker for bugs created vs completed • How long do updates take to pass QA? • Could NPS score be improved?
  50. I N V E S T I G AT I

    N G C U LT U R E • Ask developers • Ask QA
  51. “Quality software makes happy users and developers”

  52. “Having strong code coverage can help you ship faster”

  53. “Having tests helps de-risk the business”

  54. L O U I E P E N A F

    L O R @ R E S T L E S S T H I N K E R Q U E S T I O N S ?