$30 off During Our Annual Pro Sale. View Details »

Software Testing For Business Leaders

Software Testing For Business Leaders

Covers:
- Risk
- Landscape of Software Quality Today
- Poor quality affects release times and users
- How testing affects your team culture
- Brutal truths about testing

RestlessThinker

January 15, 2016
Tweet

More Decks by RestlessThinker

Other Decks in Technology

Transcript

  1. S O F T WA R E T E S T I N G
    F O R B U S I N E S S L E A D E R S
    S D L M E E T U P

    View Slide

  2. L O U I E
    P E N A F L O R
    @ R E S T L E S S T H I N K E R
    W H O A M I ?

    View Slide

  3. O V E R V I E W
    • Today’s software landscape
    • Risk
    • Team culture
    • Tests
    • Brutal truths
    • What you can do

    View Slide

  4. T O D AY ’ S S O F T WA R E L A N D S C A P E
    • Extremely low barrier to entry
    • Many niche software solutions
    • OSS accelerates development (3rd party libraries)
    • Talent is hard to find
    • UX is king

    View Slide

  5. Q U A L I T Y

    View Slide

  6. Q U A L I T Y
    • Standards are high
    • UX is a product differentiator
    • Focus is difficult
    • Next generation is quick to change

    View Slide

  7. D E F E C T S

    View Slide

  8. D E F E C T S
    • Written by humans
    • Decrease trust from the user
    • Create poor user experience
    • Affects team dynamics

    View Slide

  9. R I S K

    View Slide

  10. D E - R I S K I N G
    • Anti-virus
    • Spam and Phishing
    • Firewalls and IDS
    • Back-ups and Disaster Recovery
    • PCI SOC2/SAS70

    View Slide

  11. “Are we de-risking our software projects?”

    View Slide

  12. A P P L E
    S S L / T L S 2 0 1 4
    • Nicknamed “goto-fail”
    • “hidden in plain sight”
    • Affected millions

    View Slide

  13. A P P L E S S L / T L S 2 0 1 4
    . . .
    hashOut.data = hashes + SSL_MD5_DIGEST_LEN;
    hashOut.length = SSL_SHA1_DIGEST_LEN;
    if ((err = SSLFreeBuffer(&hashCtx)) != 0)
    goto fail;
    if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)
    goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)
    goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
    goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
    goto fail;
    goto fail; /* MISTAKE! THIS LINE SHOULD NOT BE HERE */
    if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
    goto fail;
    err = sslRawVerify(...);
    . . .
    https://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/

    View Slide

  14. P R E V E N T I O N
    • Stronger coding standards - lint verification
    • Human code review
    • Unit Test

    View Slide

  15. P R E V E N T E D I N 7 L I N E S O F C O D E
    /* Verify that a bogus signature does not validate */
    - (void) testVerifyRSASignature {
    SSLBuffer signedParams;
    SSLAllocBuffer(&signedParams, 32);
    uint8_t badSignature[128];
    memset(badSignature, 0, sizeof(badSignature));
    OSStatus err;
    err = SSLVerifySignedServerKeyExchange(&_ctx, true, signedParams, badSignature, sizeof(badSignature));
    XCTAssertNotEqual(err, 0, @"SSLVerifySignedServerKeyExchange() returned success on a completely bogus
    signature");
    }
    https://github.com/landonf/Testability-CVE-2014-1266/blob/master/TestableSecurityTests/TestableSecurityTests.m#L34

    View Slide

  16. – W O R K I N G E F F E C T I V E LY W I T H L E G A C Y C O D E
    “There is only two ways to write code. Edit and
    pray or test and verify.”

    View Slide

  17. D E - R I S K I N G
    U P S I D E

    View Slide

  18. U P S I D E
    • Move faster - ship product faster, be more productive
    • Higher quality software
    • Happier devs/employees
    • Happier customers

    View Slide

  19. D E F E C T P R E V E N T I O N
    • Cost of fixing a bug is 8 times greater than writing a test
    • Lower defect counts lead to faster release schedules
    • Many companies have defect levels that give them
    longer schedules than necessary
    • Poor quality is one of the most common reason for
    schedule overruns


    Software Quality at top speed - http://www.stevemcconnell.com/articles/art04.htm

    View Slide

  20. C U LT U R E
    D E V T E A M

    View Slide

  21. C U LT U R E
    • Engineers are humans first
    • Writing code is a craft
    • Autonomy

    View Slide

  22. C U LT U R E
    • Attract & retain top talent
    • Engineers want to work on cool stuff, not fight fires
    • No crazy long hours where you make mistakes
    • More momentum on Sprints in Agile

    View Slide

  23. T E S T S

    View Slide

  24. – “ U N C L E ” B O B M A R T I N
    C L E A N C O D E
    “Legacy code is code without tests.”

    View Slide

  25. “Are you writing legacy code in real-time?”

    View Slide

  26. T Y P E S O F T E S T S
    UNIT
    SERVICE/API
    UI
    Number of tests
    INCREASE IN:
    - Cost to develop and maintain
    - Execution time
    - Possibility of false positives
    BUT ALSO IN:
    - Coverage
    - End user (business) relevance

    View Slide

  27. P O S T D E P L O Y M E N T T E S T S
    • Staging environments are difficult to replicate
    • Verify deploy
    • Verify specific data

    View Slide

  28. T E S T I N G B A S I C S
    • Teams should agree on test rules
    • Fast to run
    • Barrier to entry in writing a test should be low
    • Continuous Integration System (Build system)
    • Watch defect count over time

    View Slide

  29. T E S T F R A M E W O R K S
    • JUnit
    • PHPUnit
    • XCTest
    • MochaJS
    • Capybara
    • Cucumber

    View Slide

  30. C O D E C O V E R A G E T O O L S
    • JAVA - EMMA, SONAR
    • JS - JSCover, Blanket.js
    • Ruby - SimpleConv
    • PHP -PHPUnit

    View Slide

  31. T E A M
    Q U A L I T Y A S S U R A N C E

    View Slide

  32. View Slide

  33. Q A T E A M
    • Different mindset than developers
    • Think more like a user
    • Can become great developers

    View Slide

  34. Q A T E A M
    • Developers think they don’t need tests if they have QA
    team
    • Can cause developers to be less thorough
    • Easy for bad habit to be accepted in the culture

    View Slide

  35. Q A + T E S T S
    • Unit tests can catch a lot of defects before it gets to
    QA
    • Tests can help developers understand failure scenarios
    better
    • QA can be more efficient

    View Slide

  36. “What if QA spent less time finding defects?”

    View Slide

  37. Q A
    • Determine if users get true value from your product
    • Are we meeting the business demands and value
    props?

    View Slide

  38. B R U TA L T R U T H S

    View Slide

  39. B R U TA L T R U T H S A B O U T T E S T I N G
    • Writing a test can be 2x the time it takes to complete a
    task
    • Learning how to write code that is easily testable can
    take time
    • SLOC for tests can exceed actual product codebase
    • Tests require maintenance

    View Slide

  40. S Q L I T E
    • SQLite library consists of approximately 112.8 KSLOC
    of C code. (KSLOC means thousands of "Source Lines
    Of Code" or, in other words, lines of code excluding
    blank lines and comments.)
    • By comparison, the project has 811 times as much test
    code and test scripts - 91555.1 KSLOC

    View Slide

  41. S T R AT E G Y M O V I N G
    F O R WA R D
    W H A T Y O U C A N D O

    View Slide

  42. I N S I G H T I N T O C U LT U R E
    “What’s your current code coverage?”

    View Slide

  43. S T R AT E G Y M O V I N G F O R WA R D
    • What do I have right now?
    • What are our problems?
    • What is customer facing?
    • What’s the highest risk?
    • Understand you are not alone

    View Slide

  44. G E T T I N G S TA R T E D
    • Test runners and harnesses
    • Continuous Integration
    • Low barrier for tests
    • Accept testing into your culture

    View Slide

  45. D O N ’ T
    • Have a month long sprint dedicated to getting tests in
    your project
    • Ignore failed tests in the beginning
    • Give up due to smaller feature releases

    View Slide

  46. T I P S
    • Writing tests is a great way to get any developer
    familiar with a codebase
    • Rotate a developer on a team to only writing tests in a
    sprint
    • Have fun watching your test count go up
    • Testing culture scales really well if done right

    View Slide

  47. S U M M A RY

    View Slide

  48. “How do I convince the rest of the exec team?”

    View Slide

  49. C R E AT I N G K P I ' S
    • Look at existing data
    • Are defects creating longer release cycles or sprints?
    • How often are interruptions or added issues are
    happening each sprint?
    • Check issue tracker for bugs created vs completed
    • How long do updates take to pass QA?
    • Could NPS score be improved?

    View Slide

  50. I N V E S T I G AT I N G C U LT U R E
    • Ask developers
    • Ask QA

    View Slide

  51. “Quality software makes happy users and
    developers”

    View Slide

  52. “Having strong code coverage can help you ship
    faster”

    View Slide

  53. “Having tests helps de-risk the business”

    View Slide

  54. L O U I E
    P E N A F L O R
    @ R E S T L E S S T H I N K E R
    Q U E S T I O N S ?

    View Slide