100% Code Coverage via automated tests in Symfony applications

If you create software that is planned for continuous enhancements and maintenance over several years then you'll need a sustainable strategy for code quality.
Code coverage by automated tests is one important metric, and its value should equal 100% at all times.
This talk will show why this is so and how it can be achieved in PHP / Symfony based applications.

Transcript

1. 100% Code Coverage via automated tests in Symfony applications Symfony User Group Cologne 2017-04-26
2. Wir entwickeln weiter
3. Once upon a time *** *** Shipment Tracking  Only a few deployments slots per year  Deployment -> $$$  Redeployment?  wait for next slot  pay more $$$  -> Quality or die!
4. Automated Testing & Metrics
5. Why automated testing? “I can‘t believe we‘re still talking about this“ – Sebastian Bergmann @ SymfonyLive 2017 Cf. Sebastians talk @ Symfony Live 2017 Some selected topics:  Confidence by repeatable cause & effect  Best case: Agility… or, at least: maintainability  Testability can lead to better code design…  … bad design yields code that is hard to test  Reducing complexity to merely binary information: Red -> Green cycle
6. Reduce complexity to merely binary information ”Complexity” by Mark Skipper, License: CC BY 2.0
7. © mensemedia Gesellschaft für Neue Medien mbH 7 www.mensemedia.net
8. © mensemedia Gesellschaft für Neue Medien mbH 8 www.mensemedia.net
9. © mensemedia Gesellschaft für Neue Medien mbH 9 www.mensemedia.net Code Coverage
10. Why Code Coverage? ”So, ummmm, how many undiscovered ruins are there?“ – Dean Leffingwell & Don Widrig We want to find out what we don‘t know.  Blind spots  Dead/unused code  Design flaws Is my code hard to test because of its design?  Produces kind of a checklist: What must still be done before we ship?  Example:  Airplane pilot is happy he started 1 engine…  …but plane has 3 more “Managing Software Requirements”, page 83, Dean Leffingwell, Don Widrig, Addison-Wesley 2000
11. Systems Engineering: Software Lifecycle ”Be nice to future-you“ – unknown How many people will have worked on software after…  … 2 years?  … 5 years?  … 10 years?  -> There will be more people new to the project than initial team members  Future “generations“ of developers must be prepared for…  … changes, enhancements etc  … patches, updates, upgrades, replacements (AngularJS 1, anyone?) etc  -> Team needs to know about cause and effect, needs confidence in the code  -> Tests help a lot! 1 2 5 10
12. But why the fudge 100%???
13. Why 100% Code Coverage?  -> Tests help a lot!  … for managing the code that the tests cover  … for code they don‘t cover there‘s no help available  So, why not 100% to help future generations?  What makes the uncovered code special that it deserves to be excluded?  Will the future team still know the reason?  After the usual and inevitable drifts in the code base…  …will the reason still be valid in the future? ”U Turn“ by André-Pierre du Plessis, License: CC BY 2.0
14. Why 100% Code Coverage?  Problem distribution:  Bug density is often not distributed homogeneously  “90% of problems are caused by 10% of the code“  What if these 10% are in your uncovered code?  No? R u sure? Really?  How can you be sure when you don‘t have tests?  … after all, tests make you durably confident  Or, which code will likely cause more problems:  highly tested code?  untested, potentially even untestable code? ”Detective with pipe and magnifzing glass“ by Juhele, https://openclipart.org/detail/254342/detective-with-pipe-and-magnifying-glass ”Confidence is ignorance. If you're feeling cocky, it's because there's something you don't know“ – Eoin Colfer, Artemis Fowl
15. Why 100% Code Coverage? ”It wasn’t me. It was that way already.“ – my children Prevent decay by the “Broken Windows“ excuse:  If some code allows for coding less well then it will soon accrete more dirty code  … and developers can blame it on the darned legacy code  Instead, you should embrace a “No code left behind“ strategy Cf. also https://blog.codinghorror.com/the-broken-window-theory/ “Broken windows, Northampton State Hospital” (https://commons.wikimedia.org/w/index.php?curid=22625973 ) by Karan Jain, License: CC BY-SA 2.0
16. Why 100% Code Coverage? Use the power of metrics  It‘s rules that everyone needs to stick to  … objectively measured by a 3rd party  -> No more discussions about the relevance of some code  … that only waste time and energy  Exceptions to the rules will add even MORE complexity  Instead…  … let the metric “reward“ the team when at 100% and boost it with extra motivation  … use the metric to reduce complexity to… … Red and Green “Complexity kills. It sucks the life out of developers […]“ – Ray Ozzie Source: Geek & Poke http://geek-and-poke.com/geekandpoke/2017/2/18/50-shades-part-2
17. © mensemedia Gesellschaft für Neue Medien mbH 17 www.mensemedia.net
18. © mensemedia Gesellschaft für Neue Medien mbH 18 www.mensemedia.net
19. Let‘s talk about accountability
20. © mensemedia Gesellschaft für Neue Medien mbH 21 www.mensemedia.net Uncovered Code can have impact on quality, maintenance, training and onboarding cost, eventually leading to technical debt.
21. © mensemedia Gesellschaft für Neue Medien mbH 22 www.mensemedia.net Quality, cost and task planning are BUSINESS issues, and only the PO can do a trade off.
22. © mensemedia Gesellschaft für Neue Medien mbH 23 www.mensemedia.net For professional developers*, automated testing is an obligation. Uncovered Code does not comply to that obligation. * especially in interpreted or dynamically typed langages
23. But wait, that‘s not feasible! !
24. Or is it? ?
25. Advice #1: Start @ 100%, Stay @ 100%
26. TDD – Test Driven/First Development Start @ 100%, Stay @ 100% “The three rules of TDD“ by Uncle Bob Martin: http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd Image: http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2014/ch1b_28_cg by user “Jgu7“ ”The three rules of TDD“:  1: No production code unless to make a test pass  2: Only so much test code to make the test fail  3: Only so much production code to make the failing test pass  What should be the expected Code Coverage?  -> 100%!  Then, if needed: refactor (test or production code) -> can still yield 100% Coverage  -> run the tests with Code Coverage to find it out and “fill the gaps” as part of the refactoring
27. TDD DON‘Ts Start @ 100%, Stay @ 100% “Hexagonal Architecture” a.k.a. “Ports and Adapters” by Alistair Cockburn, http://alistair.cockburn.us/Hexagonal+architecture For this to work you should not start on the “outside“, i.e. do NOT write any of these first:  Controller  Command  View  Data Access Object / Query  File System Accessor  …  Symfony supports this approach:  There is no need to write the Controller first
28. © mensemedia Gesellschaft für Neue Medien mbH 29 www.mensemedia.net Also, it‘s a psych thing If you start on the outside you will likely experience:  the need for non-Unit tests (i.e. slow tests)  the need to mock stuff (mocks are evil)  slow progress  a Code Coverage below 100%  less motivation because you‘re always “in debt“, i.e. < 100% https://de.wikipedia.org/wiki/Datei:Sigmund_Freud_LIFE.jpg
29. © mensemedia Gesellschaft für Neue Medien mbH 30 www.mensemedia.net Also, it‘s a psych thing If you start on the outside you will likely experience:  a tendency to do the wrong things…  … a.k.a.“Coder‘s Stockholm-Syndrome“  the urge to put more into the class, e.g. the Controller, than you should…  … to kind of make the effort spent on the Controller “pay off“  … potentially leading to higher coupling and less cohesion https://de.wikipedia.org/wiki/Datei:Sigmund_Freud_LIFE.jpg
30. TDD – DOs Start @ 100%, Stay @ 100% “The Clean Architecture” by Uncle Bob Martin, https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html Instead start at the core: The DOMAIN  Understand your domain (it‘s there, trust me)  Create the DOMAIN using TDD  … while ignoring the outside (DB, Controller etc)  Code against interfaces (=ports) and stubs  Avoid the Stockholm-Syndrome: Integrate “real” outside layers later  Let the Code Coverage metric show you where to add missing tests to fill the gaps -> Symfony‘s good design supports this approach (usually) pretty well “source code dependencies can only point inwards”
31. There is no reason why you SHOULDN‘T WANT 100% Code Coverage. (There may be reasons why you CAN‘T get to 100%, though…)
32. Code Coverage is a fallacy just a tool. lie lie fallacy It depends on what you use it for.
33. How to trick Code Coverage “Ohh, I‘m a liar. Yeah.“ – Henry Rollins  When you run a Test Case it will always cover some code  For example, calling assertNotNull(new MyClass()) for each class should yield 5-20% coverage without any real value (@covers might help…)  However, you can also fake tests, e.g. no/fake asserts, redundant checks, (some) mocks, …  -> So, we shouldn‘t write tests anymore?  … and you can fake code as well, e.g. if you require “no if/else, switch/case etc” and developers get creative  -> So, we shouldn‘t write code anymore? while ($status === "done") { echo "Status: Done"; break; }
34. Just because Code Coverage can be faked does not mean it is irrelevant.
35. Code Coverage != Test Coverage Code Coverage means…  … your existing code is covered by tests  … but you still can have – pretty obvious – bugs Code Coverage does NOT mean…  … that all possible scenarios are covered  … that would be Test Coverage … public static function divide($a, $b) { return $a / $b; } public function test_divide() { $actual = MyCalculator::divide(6, 2); $this->assertEquals(3, $actual); } // no type check, no range validation, // division by zero is possible! // ...
36. Where Code Coverage cannot be measured “I don't think anybody tests enough of anything.“ – James Gosling  Views  Twig and others that allow for markup + conditional code  There is no tool (that I know of) to measure Code Coverage in Twig  Configuration  How can we spot we tested ALL configuration settings?  How can we spot obsolete configuration? “Game over” by porpom, CC BY 2.0, https://www.flickr.com/photos/poirpom/4894850703/ {% if temp > 18 and temp < 27 %} <p>Let‘s take a walk.</p> {% endif %}
37. Where Code Coverage breaks: Aliens, Trade Offs & Lack of Knowledge
38. Alien Code Generally: full coverage can be impossible if you cannot change the production code  3rd party code  -> avoid mixing “alien“ code with your precious own code  -> consider moving it to a separate source dir, e.g. via composer  -> beware of alien code that looks like yours (by scaffolding etc)  Own legacy code…  … static calls, global calls, global state, tight coupling etc …  Symfony supports you nicely…  … no static calls needed, interfaces available, Dependency Injection container https://www.amazon.de/dp/0131177052/
39. Conscious Trade Offs Proofs of Concept (PoC)  … basically: “I don‘t really know…  … what I want yet.“  … how to do it yet.“  -> TDD and Code Coverage don‘t make much sense when you don‘t know what to assert  However, professionals should create PoCs only as throw-away code
40. Lack of Knowledge  Things you don‘t know how to test (yet)  Sometimes you need to master the technology first  It may take some time to find solutions for testing  -> Don‘t write testing off for good, keep looking for solutions
41. Lack of Knowledge  Insufficiently designed code that…  … you haven‘t realized to be poorly designed yet  … you don‘t know how to fix yet  … don‘t blame the test for not covering badly designed code  … don‘t declare badly designed code as a “special case“ that may be excluded from the rule  -> trust the test
42. Advice #2: When in doubt, trust the test
43. Trust the test ”Tests are for logic. My code does not contain logic “  You might not see any logic…  …but, usually, it‘s there  If your Controller or Data Access Object are larger than your Service then, usually, logic is there but you put it in the wrong place BIG Controller BIG DAO tiny Service “Weight Training Crossfit Fitness Models” by ThoroughlyReviewed, https://thoroughlyreviewed.com/, found at https://www.flickr.com/photos/poirpom/4894850703/, License: CC BY 2.0
44. Trust the test ”Tests are for logic. My code does not contain logic“ If the code has NO conditional statements and you think of it as “configuration as code“…  … why don‘t you turn it into real “configuration as configuration“? Speaking of configuration… hey SensioLabs…  …why do I need to create a “*Bundle“ class? Can‘t that just be configuration or convention? <?php namespace AppBundle; use Symfony\Component\HttpKernel\Bundle\Bundle; class AppBundle extends Bundle { }
45. © mensemedia Gesellschaft für Neue Medien mbH 46 www.mensemedia.net Trust the test: If (your own) code is hard to test then it likely has a design problem.
46. Reality Check  Our experience shows: it is possible to achieve 100% Code Coverage in Symfony Applications  … so don‘t blame it on Symfony ;-)  However, you might need to change the way you code and design  … but those changes will eventually lead you to Clean Code and Clean Architecture, cf.  Clean Code Developer (German)  Clean Code Principles Wiki (English)  Clean Code book  Clean Architecture book (2017) “Big_Data_Higgs” by KamiPhuc, License: CC BY 2.0
47. I know… … it‘s a controversial topic. Can we still be friends?
48. THANKS! Questions? Feedback? -> Please contact us! mensemedia Gesellschaft für Neue Medien mbH Neumannstraße 10 D- 40235 Düsseldorf mail@mensemedia.net www.mensemedia.net Andreas Czakaj Managing Director IT / CTO Tel +49.211.55 04 78-25 andreas.czakaj@mensemedia.net Twitter: AndreasCzakaj