Case Studies in Terrible Testing

Projects fail because they don’t test. Some fail because they test the wrong things. Others fail because they test too much. In this session, an enterprise consultant turned startup entrepreneur will share project case studies in testing atrocities and what can be learned from them. You’ll come away questioning your own testing. Check your dogma and let’s build better software.

Check out TrackJS JavaScript Error Monitoring
https://trackjs.com/javascript?utm_source=slides&utm_medium=speakerdeck&utm_term=testing

Transcript

1. TERRIBLE TESTING case studies in
2. are you testing the right things?
3. I have no idea
4. PROJECTS SOFTWARE TESTING and what they taught me about 5
5. toddhgardner Todd H Gardner !
6. 1 Big Todd’s Awesome CRM
7. 1 Deliver a Customer Relationship Management system to replace ad-hoc processes for internal sales teams
9. I have no idea what I’m doing
10. Consulting!
11. They have no idea what they’re doing
12. 70% Code Coverage REQUIRED
13. Assertion-Free Testing public void Test_MyObj_For_Coverage() { MyObj obj = new MyObj(); try { obj.method1(); } catch (Exception e) {} try { obj.method2(); } catch (Exception e) {} try { obj.method3(); } catch (Exception e) {} // 100% Coverage, W00t! }
14. Cool Tests Bro
17. 1 Big Todd’s Awesome CRM The Aftermath
19. They all had their rules… 1. Thou shall test everything 2. Thou shalt not ship code 3. Thou shall have more sys 4. Thou shall abstract test f 5. Thou shalt not duplicate
20. goodness testiness coverage test code ratio
21. It’s unprofessional to ship untested code “
22. 2 Big Corp’s Video Academy
23. 2 Create a online Video Training Platform for crypto-zoologists
24. unit code coverage 100% 100 0
25. MOCKS
26. code analysis metrics
27. BUGS!
29. 2 Big Corp’s Video Academy The Aftermath
30. code coverage % 100 0 what do we learn?
31. code coverage % 100 0 what do we learn? not testing
32. code coverage % 100 0 what do we learn? metrics-driven incentives
33. code coverage % 100 0 what do we learn? nothing at all
34. Testing the Mocks public void Test_MyObj_With_Mocks() { Mock<MyObj> mockObj1 = new Mock<MyObj>(); Mock<OtherObj> mockObj2 = new Mock<OtherObj>(); MyObj obj = new MyObj(mockObj1, mockObj2); mockObj2.expect().toBeCalledWith(mockObj1).andReturn(mockObj1); obj.method(); mockObj1.assertExpectations(); mockObj2.assertExpectations(); // What are we testing again? }
35. Unit Tests
36. Test-Induced http://david.heinemeierhansson.com/2014/test-induced-design-damage.html Design Damage
37. 3 Acme’s Collaboration Computing Platform
38. 3 Develop a platform for Spies to Upload Confidential Data and Collaborate on missions.
39. Market and Technology FRONTIERS
40. $2M / Mo Over 200 of the Cleverest Designers, Developers, and Testers Burning
41. Advanced TECHNOLOGY
42. System Tests Integration Tests Unit Tests Selenium UI and Services Inter-Module Contract Tests TDD, Conversation-based tests Testing Pyramid
43. WE BUILT IT
44. NO ONE CAME
46. 3 Acme’s Collaboration Computing Platform The Aftermath
48. System Tests Integration Tests Unit Tests Testing Pyramid
49. System Tests Integration Tests Unit Tests Testing Pyramid Interface Risk Orchestration Risk Functional Risk
50. System Tests Integration Tests Unit Tests Testing Pyramid Interface Risk Orchestration Risk Functional Risk User Tests Market Risk
51. Interface Risk Orchestration Risk Functional Risk Market Risk Testing Scales
52. Interface Risk Orchestration Risk Functional Risk Market Risk Testing Scales System Tests Integration Tests User Tests Unit Tests
53. 4 Widget eCommerce Monolith’s Sell-a-majig
54. 4 Expand the Widget Ecommerce platform with more products
55. yet another ECOMMERCE project
57. lots of captive USERS
58. Interface Risk Orchestration Risk Functional Risk Market Risk Testing Scales
59. Interface Orchestration Functional Market Testing Scales System Tests Integration Tests User Tests Unit Tests
60. it shipped UGLY
61. customers kept BUYING
62. Prints Money the dirty old duct-taped machine that tm
63. 4 Widget eCommerce Monolith’s Sell-a-majig The Aftermath
65. Monolith ECommerce Acme Collaboration Information Technology Uninteresting Fragile Technology Poorly Tested Success! Engineering Advancing Latest Technology Well Tested Failure!
66. Monolith ECommerce Acme Collaboration Success! Failure! High Risk Low Risk
67. SYSTEM Purpose Criticality Lifespan Frequency CONFIDENCE
68. 5 My Startup
69. 5 find and fix JavaScript bugs in modern web applications …and build a better web
70. {Track:js} http://trackjs.com JavaScript Error Monitoring
71. User Activity AJAX Requests Console Logging Browser Context Inline Source
72. Interface Risk Orchestration Risk Functional Risk Market Risk Testing Scales
73. My Money! This time it’s
74. Launch Fast and measure
75. Simple Decoupled Obvious Abstract & &
76. Plumbing Services
77. Interface Risk Orchestration Risk Functional Risk Market Risk Testing Scales System Tests Integration Tests User Tests Unit Tests
78. Service Model Test Fixture Other Services Wrapper Data Unit Test Integration Test Testing Structure
79. “beta” Service Dragons be here! Testing Structure Canary Users Analytics Monitoring
80. FAST TO FIX is almost as good as NEVER BROKEN (and sometimes better)
81. PROJECTS SOFTWARE TESTING and what they taught me about 5
82. 1 Big Todd’s Awesome CRM METRICS don’t test to hit
83. 2 Big Corp’s Video Academy BREAKING test what is
84. 3 Acme’s Collaboration Computing Platform SCARIEST test the thing first
85. 4 Widget eCommerce Monolith’s Sell-a-majig CUSTOMER USE tests can’t substitute for
86. 5 JavaScript Error Tracking TrackJS’s MONITORING can substitute for tests
87. {Track:js} http://trackjs.com JavaScript Error Monitoring
88. TERRIBLE TESTING case studies in toddhgardner ! " todd@trackjs.com