Covers:
- Risk
- Landscape of Software Quality Today
- Poor quality affects release times and users
- How testing affects your team culture
- Brutal truths about testing
T O D AY ’ S S O F T WA R E L A N D S C A P E • Extremely low barrier to entry • Many niche software solutions • OSS accelerates development (3rd party libraries) • Talent is hard to find • UX is king
A P P L E S S L / T L S 2 0 1 4 . . . hashOut.data = hashes + SSL_MD5_DIGEST_LEN; hashOut.length = SSL_SHA1_DIGEST_LEN; if ((err = SSLFreeBuffer(&hashCtx)) != 0) goto fail; if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0) goto fail; goto fail; /* MISTAKE! THIS LINE SHOULD NOT BE HERE */ if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0) goto fail; err = sslRawVerify(...); . . . https://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/
P R E V E N T E D I N 7 L I N E S O F C O D E /* Verify that a bogus signature does not validate */ - (void) testVerifyRSASignature { SSLBuffer signedParams; SSLAllocBuffer(&signedParams, 32); uint8_t badSignature[128]; memset(badSignature, 0, sizeof(badSignature)); OSStatus err; err = SSLVerifySignedServerKeyExchange(&_ctx, true, signedParams, badSignature, sizeof(badSignature)); XCTAssertNotEqual(err, 0, @"SSLVerifySignedServerKeyExchange() returned success on a completely bogus signature"); } https://github.com/landonf/Testability-CVE-2014-1266/blob/master/TestableSecurityTests/TestableSecurityTests.m#L34
D E F E C T P R E V E N T I O N • Cost of fixing a bug is 8 times greater than writing a test • Lower defect counts lead to faster release schedules • Many companies have defect levels that give them longer schedules than necessary • Poor quality is one of the most common reason for schedule overruns
Software Quality at top speed - http://www.stevemcconnell.com/articles/art04.htm
C U LT U R E • Attract & retain top talent • Engineers want to work on cool stuff, not fight fires • No crazy long hours where you make mistakes • More momentum on Sprints in Agile
T Y P E S O F T E S T S UNIT SERVICE/API UI Number of tests INCREASE IN: - Cost to develop and maintain - Execution time - Possibility of false positives BUT ALSO IN: - Coverage - End user (business) relevance
T E S T I N G B A S I C S • Teams should agree on test rules • Fast to run • Barrier to entry in writing a test should be low • Continuous Integration System (Build system) • Watch defect count over time
Q A T E A M • Developers think they don’t need tests if they have QA team • Can cause developers to be less thorough • Easy for bad habit to be accepted in the culture
Q A + T E S T S • Unit tests can catch a lot of defects before it gets to QA • Tests can help developers understand failure scenarios better • QA can be more efficient
B R U TA L T R U T H S A B O U T T E S T I N G • Writing a test can be 2x the time it takes to complete a task • Learning how to write code that is easily testable can take time • SLOC for tests can exceed actual product codebase • Tests require maintenance
S Q L I T E • SQLite library consists of approximately 112.8 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) • By comparison, the project has 811 times as much test code and test scripts - 91555.1 KSLOC
S T R AT E G Y M O V I N G F O R WA R D • What do I have right now? • What are our problems? • What is customer facing? • What’s the highest risk? • Understand you are not alone
D O N ’ T • Have a month long sprint dedicated to getting tests in your project • Ignore failed tests in the beginning • Give up due to smaller feature releases
T I P S • Writing tests is a great way to get any developer familiar with a codebase • Rotate a developer on a team to only writing tests in a sprint • Have fun watching your test count go up • Testing culture scales really well if done right
C R E AT I N G K P I ' S • Look at existing data • Are defects creating longer release cycles or sprints? • How often are interruptions or added issues are happening each sprint? • Check issue tracker for bugs created vs completed • How long do updates take to pass QA? • Could NPS score be improved?