Technical Debt - The code monster in your closet - PyParis 2018

PyParis - November 14, 2018

Technical debt is the code monster hiding in every Python programmer’s closet. If you ignore it, it will terrorize you at night.

I’ve worked at many institutions with many programming languages over the past 12 years. They all have technical debt. Putting on a band-aid and ignoring the real issues can be disastrous. We’ll go through several case studies, review big red flags, and learn how to start chipping away at the problem with confidence.

Transcript

1. PyParis 2018
   Nina Zakharenko
   @nnja
   bit.ly/ParisTechDebt
   

   View Slide

2. Livetweet!
   use #PyParis
   @nnja
   

   View Slide

3. Technical Debt
   The code monster in your
   closet
   slides: bit.ly/ParisTechDebt
   @nnja
   

   View Slide

4. What is
   technical debt?
   @nnja
   

   View Slide

5. A series of bad decisions
   (Both business & technical)
   @nnja
   

   View Slide

6. Which lead to ->
   Error prone code & architecture
   @nnja
   

   View Slide

7. ... and using more
   Resources
   to accomplish
   Less
   @nnja
   

   View Slide

8. What decisions were made
   in the past that prevent me
   from getting sh** done
   today?
   @nnja
   

   View Slide

9. What causes
   technical debt?
   @nnja
   

   View Slide

10. Me.
    And you.
    @nnja
    

    View Slide

11. Mistakes I Made Early On
    4 Not seeing the value in unit tests
    4 Not knowing how to say NO to features
    @nnja
    

    View Slide

12. Mistakes I Made Early On
    4 Overly optimistic estimates
    4 Putting releases over good design & reusable code
    @nnja
    

    View Slide

13. Time Crunch
    That project was due yesterday!
    I'll take a shortcut, and clean up the
    mess tomorrow.
    @nnja
    

    View Slide

14. Unneeded Complexity
    Lines of code committed != amount of
    work accomplished
    @nnja
    

    View Slide

15. Lack of understanding
    1. Have a problem
    2. Look up a solution on stackoverflow
    3. Copy & paste it into your code
    4. ???
    5. Bugs!
    @nnja
    

    View Slide

16. Culture of Despair
    This is already a heap of trash.
    Will anyone really notice if I add one
    more thing to the top?
    @nnja
    

    View Slide

17. Red Flags
    Houston, we have a problem.
    @nnja
    

    View Slide

18. Code Smells
    4 Not Bugs
    4 An indication of a deeper problem
    @nnja
    

    View Slide

19. Code Smells
    4 Half implemented features
    4 No documentation, or poor documentation
    @nnja
    

    View Slide

20. Code Smells
    4 Commented out code
    4 Incorrect comments
    4 No tests, or worse: broken tests
    @nnja
    

    View Slide

21. Restore deleted code with git!
    Find by content:
    $ git log --summary -G'(D|d)jango'
    Find the commit that deleted a file:
    ```shell
    git log --diff-filter=D --summary --
    @nnja
    

    View Slide

22. No more
    commented out
    code!
    @nnja
    

    View Slide

23. Poor Documentation
    class OrganicGlutenFreePizzaFactory:
    def get_dough(self):
    """
    Return amazing, organic, GMO and Gluten Free Dough
    """
    # ran out of organic gluten free, use the other stuff.
    # return 'organic gluten free dough'
    return 'gmo pesticide processed gluten-full dough'
    @nnja
    

    View Slide

24. Architecture & Design... Smells
    4 Parts of the code no one wants to touch
    4 Brittle codebase -- changing code in one area breaks
    other parts of the system
    4 Severe outages caused by frequent & unexpected
    bugs
    @nnja
    

    View Slide

25. Good Design -> Implementing new
    features comes easily
    Poor Design -> New features are shoe-
    horned into the system
    @nnja
    

    View Slide

26. Python Specific
    @nnja
    

    View Slide

27. Functionality changes, but variable
    names don't
    employees = ['John', 'Mary', 'Dale']
    employees = 'Bob'
    employees[0]
    @nnja
    

    View Slide

28. Monkey Patching
    !"
    def new_init(self):
    pass
    some_library.SomeClass.__init__ = new_init
    @nnja
    

    View Slide

29. What exactly does this decorator do?
    def decorator_evil(func):
    return False
    @decorator_evil
    def target(a,b):
    return a + b
    >>> target(1,2)
    TypeError: 'bool' object is not callable
    >>> target
    False
    

    View Slide

30. Circular Dependencies
    # Circumvent circular dependency warnings
    def some_function(x):
    from some.module import some_method
    some_method(x)
    @nnja
    

    View Slide

31. Case Studies
    @nnja
    

    View Slide

32. IRS Chief:
    "We still have applications that were
    running when JFK was President"
    Tech at the IRS
    

    View Slide

33. 50 Year Old Technology
    "And we continue to use the COBOL
    programming language, it is extremely
    difficult to find IT experts who are
    versed in this language."
    @nnja
    

    View Slide

34. It's not just the IRS
    4 Banks & Financial Institutions
    4 Universities
    4 Air Traffic Control
    4 ... many still use COBOL
    @nnja
    

    View Slide

35. Story Time
    4 I used to work in finance.
    4 At the time I was there, all of the banking systems
    were run on mainframes.
    4 The bankers were getting frustrated. They wanted a
    UI.
    @nnja
    

    View Slide

36. Big Idea!
    4 Let’s write a fancy new web front end
    4 It’ll do ALL the things
    @nnja
    

    View Slide

37. But
    4 Rewriting the backend is too expensive
    4 It already does what we need
    4 Let's leave the mainframe as the backend
    @nnja
    

    View Slide

38. Cursors
    4 The mainframe would output a text screen from a
    program result, based on a query.
    4 The results would be parsed by reading variables
    from the screen in certain positions.
    @nnja
    

    View Slide

39. Result?
    4 The new system was incredibly slow
    4 And error prone
    4 After months of work, the multi-million dollar
    rewrite was scrapped
    @nnja
    

    View Slide

40. You can try to cover up debt...
    (but it probably won't work)
    @nnja
    

    View Slide

41. The MVP
    4 (Minimum Viable Product)
    4 Get the product to market as soon as possible
    @nnja
    

    View Slide

42. A Great Idea
    4 A successful project that was created by a lone
    developer in a coffee fueled 48 hours.
    @nnja
    

    View Slide

43. There Was a Problem
    4 Years went on, but the initial code and design didn’t
    go away.
    4 Instead, it became the base for an expanding project,
    with expanding features.
    4 There was never any time to refactor.
    @nnja
    

    View Slide

44. !"#
    @nnja
    

    View Slide

45. Scope Creep
    4 Features that someone thought was a good idea one
    day, stuck around forever.
    4 > “In case we need them. Later.”
    @nnja
    

    View Slide

46. Sad Developers
    4 Minimal working tests (no time to write them).
    4 When a release was pushed, something was bound to
    break.
    4 Made everything feel like it was your fault.
    @nnja
    

    View Slide

47. Grinding To a Halt
    4 Development time for new features skyrocketed
    4 The project was deemed too difficult to maintain
    4 ... and cancelled.
    @nnja
    

    View Slide

48. Sometimes you need to
    burn it.
    With fire.
    @nnja
    

    View Slide

49. Battling The Monster
    @nnja
    

    View Slide

50. Don't point fingers
    Technical debt is a team-wide problem.
    Everybody needs to be part of the
    solution.
    @nnja
    

    View Slide

51. Work Together
    4 Code Standards
    4 Pair Programming
    4 Code Reviews
    @nnja
    

    View Slide

52. Unless something is on fire,
    or you’re losing money,
    don't merge unreviewed
    code into master.
    @nnja
    

    View Slide

53. Be Accountable
    4 Unit & Integration Tests
    4 Pre-Commit Hooks
    4 Continuous Integration
    @nnja
    

    View Slide

54. Make a Commitment
    Company tried to fight debt, but they
    didn't make a commitment.
    @nnja
    

    View Slide

55. Ended up with twice as
    many technologies in their
    stack as needed, and twice
    as big of a mess.
    @nnja
    

    View Slide

56. Sell It To Decision Makers
    By allocating project time to tackling debt,
    the end result will be less error prone, easier
    to maintain, and easier to add features to.
    @nnja
    

    View Slide

57. Not broken, why fix it?
    Source
    

    View Slide

58. Ski Rental Problem
    You’re going skiing for an unknown
    number of days.
    It costs $1 a day to rent, or $20 to
    buy.
    Source
    

    View Slide

59. Hiring developers is hard.
    Technical debt frustrates developers.
    Frustrated developers are more likely
    to leave.
    @nnja
    

    View Slide

60. Some lingering debt is inevitable.
    Don't be a perfectionist.
    Figure out the project tolerance, and
    work with it.
    @nnja
    

    View Slide

61. Use these arguments to
    justify the additional time
    it takes to do things right
    @nnja
    

    View Slide

62. To Win The Fight, Pay
    Down Your Debt
    @nnja
    

    View Slide

63. Refactoring
    The single greatest tool in your toolbox
    @nnja
    

    View Slide

64. What is it?
    Systematically changing the code
    without changing functionality, while
    improving design and readability.
    @nnja
    

    View Slide

65. Refactoring
    4 Slow and steady wins the race.
    4 The end goal is to refactor without breaking existing
    functionality.
    @nnja
    

    View Slide

66. Refactoring
    4 Replace functions and modules incrementally.
    4 Test as you go.
    4 Tests are mandatory at this step.
    @nnja
    

    View Slide

67. github.com/Yelp/undebt, yelp
    refactoring
    

    View Slide

68. Use proper design patterns
    github.com/faif/python-patterns
    

    View Slide

69. Use depreciation patterns
    Like openstack debtcollector
    class removed_property(object):
    """Property descriptor that deprecates a property.
    This works like the ``@property`` descriptor but can
    be used instead to provide the same functionality
    and also interact with the :mod:`warnings`module to
    warn when a property is accessed, set and/or deleted.
    """
    @nnja
    

    View Slide

70. Use vulture.py
    to find dead or unreachable code
    $ pip install vulture
    $ vulture script.py package/
    or
    $ python -m vulture script.py package/
    github.com/jendrikseipp/vulture
    

    View Slide

71. sample code
    def foo():
    print("foo")
    def bar():
    print("bar")
    def baz():
    print("baz")
    foo()
    bar()
    vulture.py output
    › python -m vulture foo.py
    foo.py:7: unused function 'baz' (60% confidence)
    

    View Slide

72. Prioritize
    What causes the biggest & most
    frequent pain points for developers?
    @nnja
    

    View Slide

73. Just like with
    monetary debt,
    pay off the high interest
    loan first.
    @nnja
    

    View Slide

74. Shelf Life
    What's the life expectancy of this
    project?
    Longer shelf life -> higher debt
    interest
    @nnja
    

    View Slide

75. Technical debt can be strategic
    If you don't have to pay it off, you got
    something for nothing.
    @nnja
    

    View Slide

76. Making time for refactoring depends
    on the size of your team, and the size
    of your problem.
    @nnja
    

    View Slide

77. Guidelines
    4 Small
    4 Devote a week every 6-8 weeks
    4 Medium
    4 Devote a person every 1-4 weeks, rotate
    4 Large
    @nnja
    

    View Slide

78. A Few Last Tips
    @nnja
    

    View Slide

79. Code should be for humans
    @nnja
    

    View Slide

80. Boy Scout Rule
    "Always check in a module cleaner than
    when you checked it out."
    Source
    

    View Slide

81. Expect To Be Frustrated
    The process of cleaning up days /
    months / years of bad code can be
    analogous with untangling a ball of
    yarn.
    Don't give up.
    @nnja
    

    View Slide

82. View Slide

83. Thank You!
    Python @ Microsoft:
    bit.ly/parispython
    @nnja
    @nnja
    

    View Slide