Software Testing For Business Leaders

Covers:
- Risk
- Landscape of Software Quality Today
- Poor quality affects release times and users
- How testing affects your team culture
- Brutal truths about testing

Transcript

1. S O F T WA R E T E S T I N G
   F O R B U S I N E S S L E A D E R S
   S D L M E E T U P
   

   View Slide

2. L O U I E
   P E N A F L O R
   @ R E S T L E S S T H I N K E R
   W H O A M I ?
   

   View Slide

3. O V E R V I E W
   • Today’s software landscape
   • Risk
   • Team culture
   • Tests
   • Brutal truths
   • What you can do
   

   View Slide

4. T O D AY ’ S S O F T WA R E L A N D S C A P E
   • Extremely low barrier to entry
   • Many niche software solutions
   • OSS accelerates development (3rd party libraries)
   • Talent is hard to find
   • UX is king
   

   View Slide

5. Q U A L I T Y
   

   View Slide

6. Q U A L I T Y
   • Standards are high
   • UX is a product differentiator
   • Focus is difficult
   • Next generation is quick to change
   

   View Slide

7. D E F E C T S
   

   View Slide

8. D E F E C T S
   • Written by humans
   • Decrease trust from the user
   • Create poor user experience
   • Affects team dynamics
   

   View Slide

9. R I S K
   

   View Slide

10. D E - R I S K I N G
    • Anti-virus
    • Spam and Phishing
    • Firewalls and IDS
    • Back-ups and Disaster Recovery
    • PCI SOC2/SAS70
    

    View Slide

11. “Are we de-risking our software projects?”
    

    View Slide

12. A P P L E
    S S L / T L S 2 0 1 4
    • Nicknamed “goto-fail”
    • “hidden in plain sight”
    • Affected millions
    

    View Slide

13. A P P L E S S L / T L S 2 0 1 4
    . . .
    hashOut.data = hashes + SSL_MD5_DIGEST_LEN;
    hashOut.length = SSL_SHA1_DIGEST_LEN;
    if ((err = SSLFreeBuffer(&hashCtx)) != 0)
    goto fail;
    if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)
    goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)
    goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
    goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
    goto fail;
    goto fail; /* MISTAKE! THIS LINE SHOULD NOT BE HERE */
    if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
    goto fail;
    err = sslRawVerify(...);
    . . .
    https://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/
    

    View Slide

14. P R E V E N T I O N
    • Stronger coding standards - lint verification
    • Human code review
    • Unit Test
    

    View Slide

15. P R E V E N T E D I N 7 L I N E S O F C O D E
    /* Verify that a bogus signature does not validate */
    - (void) testVerifyRSASignature {
    SSLBuffer signedParams;
    SSLAllocBuffer(&signedParams, 32);
    uint8_t badSignature[128];
    memset(badSignature, 0, sizeof(badSignature));
    OSStatus err;
    err = SSLVerifySignedServerKeyExchange(&_ctx, true, signedParams, badSignature, sizeof(badSignature));
    XCTAssertNotEqual(err, 0, @"SSLVerifySignedServerKeyExchange() returned success on a completely bogus
    signature");
    }
    https://github.com/landonf/Testability-CVE-2014-1266/blob/master/TestableSecurityTests/TestableSecurityTests.m#L34
    

    View Slide

16. – W O R K I N G E F F E C T I V E LY W I T H L E G A C Y C O D E
    “There is only two ways to write code. Edit and
    pray or test and verify.”
    

    View Slide

17. D E - R I S K I N G
    U P S I D E
    

    View Slide

18. U P S I D E
    • Move faster - ship product faster, be more productive
    • Higher quality software
    • Happier devs/employees
    • Happier customers
    

    View Slide

19. D E F E C T P R E V E N T I O N
    • Cost of fixing a bug is 8 times greater than writing a test
    • Lower defect counts lead to faster release schedules
    • Many companies have defect levels that give them
    longer schedules than necessary
    • Poor quality is one of the most common reason for
    schedule overruns

    

    Software Quality at top speed - http://www.stevemcconnell.com/articles/art04.htm
    

    View Slide

20. C U LT U R E
    D E V T E A M
    

    View Slide

21. C U LT U R E
    • Engineers are humans first
    • Writing code is a craft
    • Autonomy
    

    View Slide

22. C U LT U R E
    • Attract & retain top talent
    • Engineers want to work on cool stuff, not fight fires
    • No crazy long hours where you make mistakes
    • More momentum on Sprints in Agile
    

    View Slide

23. T E S T S
    

    View Slide

24. – “ U N C L E ” B O B M A R T I N
    C L E A N C O D E
    “Legacy code is code without tests.”
    

    View Slide

25. “Are you writing legacy code in real-time?”
    

    View Slide

26. T Y P E S O F T E S T S
    UNIT
    SERVICE/API
    UI
    Number of tests
    INCREASE IN:
    - Cost to develop and maintain
    - Execution time
    - Possibility of false positives
    BUT ALSO IN:
    - Coverage
    - End user (business) relevance
    

    View Slide

27. P O S T D E P L O Y M E N T T E S T S
    • Staging environments are difficult to replicate
    • Verify deploy
    • Verify specific data
    

    View Slide

28. T E S T I N G B A S I C S
    • Teams should agree on test rules
    • Fast to run
    • Barrier to entry in writing a test should be low
    • Continuous Integration System (Build system)
    • Watch defect count over time
    

    View Slide

29. T E S T F R A M E W O R K S
    • JUnit
    • PHPUnit
    • XCTest
    • MochaJS
    • Capybara
    • Cucumber
    

    View Slide

30. C O D E C O V E R A G E T O O L S
    • JAVA - EMMA, SONAR
    • JS - JSCover, Blanket.js
    • Ruby - SimpleConv
    • PHP -PHPUnit
    

    View Slide

31. T E A M
    Q U A L I T Y A S S U R A N C E
    

    View Slide

32. View Slide

33. Q A T E A M
    • Different mindset than developers
    • Think more like a user
    • Can become great developers
    

    View Slide

34. Q A T E A M
    • Developers think they don’t need tests if they have QA
    team
    • Can cause developers to be less thorough
    • Easy for bad habit to be accepted in the culture
    

    View Slide

35. Q A + T E S T S
    • Unit tests can catch a lot of defects before it gets to
    QA
    • Tests can help developers understand failure scenarios
    better
    • QA can be more efficient
    

    View Slide

36. “What if QA spent less time finding defects?”
    

    View Slide

37. Q A
    • Determine if users get true value from your product
    • Are we meeting the business demands and value
    props?
    

    View Slide

38. B R U TA L T R U T H S
    

    View Slide

39. B R U TA L T R U T H S A B O U T T E S T I N G
    • Writing a test can be 2x the time it takes to complete a
    task
    • Learning how to write code that is easily testable can
    take time
    • SLOC for tests can exceed actual product codebase
    • Tests require maintenance
    

    View Slide

40. S Q L I T E
    • SQLite library consists of approximately 112.8 KSLOC
    of C code. (KSLOC means thousands of "Source Lines
    Of Code" or, in other words, lines of code excluding
    blank lines and comments.)
    • By comparison, the project has 811 times as much test
    code and test scripts - 91555.1 KSLOC
    

    View Slide

41. S T R AT E G Y M O V I N G
    F O R WA R D
    W H A T Y O U C A N D O
    

    View Slide

42. I N S I G H T I N T O C U LT U R E
    “What’s your current code coverage?”
    

    View Slide

43. S T R AT E G Y M O V I N G F O R WA R D
    • What do I have right now?
    • What are our problems?
    • What is customer facing?
    • What’s the highest risk?
    • Understand you are not alone
    

    View Slide

44. G E T T I N G S TA R T E D
    • Test runners and harnesses
    • Continuous Integration
    • Low barrier for tests
    • Accept testing into your culture
    

    View Slide

45. D O N ’ T
    • Have a month long sprint dedicated to getting tests in
    your project
    • Ignore failed tests in the beginning
    • Give up due to smaller feature releases
    

    View Slide

46. T I P S
    • Writing tests is a great way to get any developer
    familiar with a codebase
    • Rotate a developer on a team to only writing tests in a
    sprint
    • Have fun watching your test count go up
    • Testing culture scales really well if done right
    

    View Slide

47. S U M M A RY
    

    View Slide

48. “How do I convince the rest of the exec team?”
    

    View Slide

49. C R E AT I N G K P I ' S
    • Look at existing data
    • Are defects creating longer release cycles or sprints?
    • How often are interruptions or added issues are
    happening each sprint?
    • Check issue tracker for bugs created vs completed
    • How long do updates take to pass QA?
    • Could NPS score be improved?
    

    View Slide

50. I N V E S T I G AT I N G C U LT U R E
    • Ask developers
    • Ask QA
    

    View Slide

51. “Quality software makes happy users and
    developers”
    

    View Slide

52. “Having strong code coverage can help you ship
    faster”
    

    View Slide

53. “Having tests helps de-risk the business”
    

    View Slide

54. L O U I E
    P E N A F L O R
    @ R E S T L E S S T H I N K E R
    Q U E S T I O N S ?
    

    View Slide