Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Case Studies in Terrible Testing

Case Studies in Terrible Testing

Projects fail because they don’t test. Some fail because they test the wrong things. Others fail because they test too much. In this session, an enterprise consultant turned startup entrepreneur will share project case studies in testing atrocities and what can be learned from them. You’ll come away questioning your own testing. Check your dogma and let’s build better software.

Check out TrackJS JavaScript Error Monitoring
https://trackjs.com/javascript?utm_source=slides&utm_medium=speakerdeck&utm_term=testing

Todd Gardner

April 13, 2015
Tweet

More Decks by Todd Gardner

Other Decks in Technology

Transcript

  1. TERRIBLE
    TESTING
    case studies in

    View full-size slide

  2. are you
    testing
    the right things?

    View full-size slide

  3. I have
    no idea

    View full-size slide

  4. PROJECTS
    SOFTWARE TESTING
    and what they taught me about
    5

    View full-size slide

  5. toddhgardner
    Todd H Gardner
    !

    View full-size slide

  6. 1
    Big Todd’s
    Awesome CRM

    View full-size slide

  7. 1
    Deliver a
    Customer Relationship Management
    system to replace ad-hoc
    processes for internal sales teams

    View full-size slide

  8. I have no idea
    what I’m doing

    View full-size slide

  9. They have no idea
    what they’re doing

    View full-size slide

  10. 70% Code
    Coverage
    REQUIRED

    View full-size slide

  11. Assertion-Free Testing
    public void Test_MyObj_For_Coverage() {
    MyObj obj = new MyObj();
    try { obj.method1(); } catch (Exception e) {}
    try { obj.method2(); } catch (Exception e) {}
    try { obj.method3(); } catch (Exception e) {}
    // 100% Coverage, W00t!
    }

    View full-size slide

  12. Cool Tests Bro

    View full-size slide

  13. 1
    Big Todd’s
    Awesome CRM
    The Aftermath

    View full-size slide

  14. They all had their rules…
    1. Thou shall test everything
    2. Thou shalt not ship code
    3. Thou shall have more sys
    4. Thou shall abstract test f
    5. Thou shalt not duplicate

    View full-size slide

  15. goodness
    testiness
    coverage
    test code ratio

    View full-size slide

  16. It’s unprofessional to
    ship untested code

    View full-size slide

  17. 2
    Big Corp’s
    Video Academy

    View full-size slide

  18. 2
    Create a online
    Video Training Platform
    for crypto-zoologists

    View full-size slide

  19. unit code coverage 100%
    100
    0

    View full-size slide

  20. code analysis
    metrics

    View full-size slide

  21. 2
    Big Corp’s
    Video Academy
    The Aftermath

    View full-size slide

  22. code coverage %
    100
    0
    what do we learn?

    View full-size slide

  23. code coverage %
    100
    0
    what do we learn?
    not testing

    View full-size slide

  24. code coverage %
    100
    0
    what do we learn?
    metrics-driven
    incentives

    View full-size slide

  25. code coverage %
    100
    0
    what do we learn?
    nothing at all

    View full-size slide

  26. Testing the Mocks
    public void Test_MyObj_With_Mocks() {
    Mock mockObj1 = new Mock();
    Mock mockObj2 = new Mock();
    MyObj obj = new MyObj(mockObj1, mockObj2);
    mockObj2.expect().toBeCalledWith(mockObj1).andReturn(mockObj1);
    obj.method();
    mockObj1.assertExpectations();
    mockObj2.assertExpectations();
    // What are we testing again?
    }

    View full-size slide

  27. Test-Induced
    http://david.heinemeierhansson.com/2014/test-induced-design-damage.html
    Design Damage

    View full-size slide

  28. 3
    Acme’s
    Collaboration
    Computing Platform

    View full-size slide

  29. 3
    Develop a platform for Spies to
    Upload Confidential Data
    and Collaborate on missions.

    View full-size slide

  30. Market and Technology
    FRONTIERS

    View full-size slide

  31. $2M / Mo
    Over 200 of the
    Cleverest
    Designers,
    Developers,
    and Testers
    Burning

    View full-size slide

  32. Advanced
    TECHNOLOGY

    View full-size slide

  33. System Tests
    Integration Tests
    Unit Tests
    Selenium UI and Services
    Inter-Module Contract Tests
    TDD, Conversation-based tests
    Testing Pyramid

    View full-size slide

  34. 3
    Acme’s
    Collaboration
    Computing Platform
    The Aftermath

    View full-size slide

  35. System Tests
    Integration Tests
    Unit Tests
    Testing Pyramid

    View full-size slide

  36. System Tests
    Integration Tests
    Unit Tests
    Testing Pyramid
    Interface Risk
    Orchestration Risk
    Functional Risk

    View full-size slide

  37. System Tests
    Integration Tests
    Unit Tests
    Testing Pyramid
    Interface Risk
    Orchestration Risk
    Functional Risk
    User Tests
    Market Risk

    View full-size slide

  38. Interface Risk
    Orchestration Risk
    Functional Risk
    Market Risk
    Testing Scales

    View full-size slide

  39. Interface Risk
    Orchestration Risk
    Functional Risk
    Market Risk
    Testing Scales
    System Tests
    Integration Tests
    User Tests
    Unit Tests

    View full-size slide

  40. 4
    Widget eCommerce
    Monolith’s
    Sell-a-majig

    View full-size slide

  41. 4
    Expand the
    Widget Ecommerce
    platform with more products

    View full-size slide

  42. yet another
    ECOMMERCE
    project

    View full-size slide

  43. lots of captive
    USERS

    View full-size slide

  44. Interface Risk
    Orchestration Risk
    Functional Risk
    Market Risk
    Testing Scales

    View full-size slide

  45. Interface
    Orchestration
    Functional
    Market
    Testing Scales
    System Tests
    Integration Tests
    User Tests
    Unit Tests

    View full-size slide

  46. it shipped
    UGLY

    View full-size slide

  47. customers kept
    BUYING

    View full-size slide

  48. Prints Money
    the dirty old duct-taped machine that
    tm

    View full-size slide

  49. 4
    Widget eCommerce
    Monolith’s
    Sell-a-majig
    The Aftermath

    View full-size slide

  50. Monolith ECommerce Acme Collaboration
    Information Technology
    Uninteresting
    Fragile Technology
    Poorly Tested
    Success!
    Engineering
    Advancing
    Latest Technology
    Well Tested
    Failure!

    View full-size slide

  51. Monolith ECommerce Acme Collaboration
    Success! Failure!
    High Risk
    Low Risk

    View full-size slide

  52. SYSTEM
    Purpose
    Criticality
    Lifespan
    Frequency
    CONFIDENCE

    View full-size slide

  53. 5
    find and fix JavaScript bugs
    in modern web applications
    …and build a better web

    View full-size slide

  54. {Track:js}
    http://trackjs.com
    JavaScript Error Monitoring

    View full-size slide

  55. User Activity
    AJAX Requests
    Console Logging
    Browser Context
    Inline Source

    View full-size slide

  56. Interface Risk
    Orchestration Risk
    Functional Risk
    Market Risk
    Testing Scales

    View full-size slide

  57. My Money!
    This time it’s

    View full-size slide

  58. Launch Fast
    and measure

    View full-size slide

  59. Simple
    Decoupled
    Obvious
    Abstract
    &
    &

    View full-size slide

  60. Plumbing Services

    View full-size slide

  61. Interface Risk
    Orchestration Risk
    Functional Risk
    Market Risk
    Testing Scales
    System Tests
    Integration Tests
    User Tests
    Unit Tests

    View full-size slide

  62. Service
    Model
    Test Fixture
    Other
    Services
    Wrapper Data
    Unit
    Test
    Integration
    Test
    Testing Structure

    View full-size slide

  63. “beta”
    Service
    Dragons be
    here!
    Testing Structure
    Canary
    Users
    Analytics
    Monitoring

    View full-size slide

  64. FAST TO FIX
    is almost as good as
    NEVER BROKEN
    (and sometimes better)

    View full-size slide

  65. PROJECTS
    SOFTWARE TESTING
    and what they taught me about
    5

    View full-size slide

  66. 1
    Big Todd’s
    Awesome CRM
    METRICS
    don’t test to hit

    View full-size slide

  67. 2
    Big Corp’s
    Video Academy
    BREAKING
    test what is

    View full-size slide

  68. 3
    Acme’s
    Collaboration
    Computing Platform
    SCARIEST
    test the
    thing first

    View full-size slide

  69. 4
    Widget eCommerce
    Monolith’s
    Sell-a-majig
    CUSTOMER USE
    tests can’t substitute for

    View full-size slide

  70. 5
    JavaScript Error Tracking
    TrackJS’s
    MONITORING
    can substitute for tests

    View full-size slide

  71. {Track:js}
    http://trackjs.com
    JavaScript Error Monitoring

    View full-size slide

  72. TERRIBLE
    TESTING
    case studies in
    toddhgardner
    !
    " [email protected]

    View full-size slide