Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing Maintainable Software At Scale

Writing Maintainable Software At Scale

My keynote from PyCon Estonia 2022.

Andrew Godwin

August 25, 2022
Tweet

More Decks by Andrew Godwin

Other Decks in Programming

Transcript

  1. WRITING MAINTAINABLE SOFTWARE
    ANDREW GODWIN // @andrewgodwin
    AT SCALE

    View Slide

  2. Andrew Godwin / @andrewgodwin
    Hi, I’m
    Andrew Godwin
    • Principal Engineer at Astronomer (w/on Airflow)
    • Django Migrations, Channels & Async
    • Doing bad things with Python since 2005

    View Slide

  3. Andrew Godwin / @andrewgodwin

    View Slide

  4. Andrew Godwin / @andrewgodwin
    Why am I still writing Python?
    17 years is quite a long time, though not as long as some

    View Slide

  5. Andrew Godwin / @andrewgodwin
    I mostly work on large new projects
    The kind where you only know roughly what you want

    View Slide

  6. Andrew Godwin / @andrewgodwin
    I rarely know exactly where I'm going
    …and so I use Python, as it's really good at that

    View Slide

  7. Andrew Godwin / @andrewgodwin
    A lot of maintenance advice is
    "do it right first time"

    View Slide

  8. Andrew Godwin / @andrewgodwin
    But we will all get it wrong sometimes
    So, how do we build knowing we can make mistakes?

    View Slide

  9. Andrew Godwin / @andrewgodwin
    Software is almost never complete
    It will spend far longer running and being maintained than you spend creating it

    View Slide

  10. Andrew Godwin / @andrewgodwin
    Nobody knows what they want
    Which is also why we're safe from AIs writing code, by the way

    View Slide

  11. Andrew Godwin / @andrewgodwin
    This is mostly about systems, not libraries
    (though some of the lessons are similar!)

    View Slide

  12. Andrew Godwin / @andrewgodwin
    Writing code Understanding code

    View Slide

  13. Andrew Godwin / @andrewgodwin
    What makes maintenance hard?
    There's some obvious ones, of course…

    View Slide

  14. Andrew Godwin / @andrewgodwin
    Obfuscation
    Or to echo a bad joke, "job security"

    View Slide

  15. Andrew Godwin / @andrewgodwin
    q=lambda
    x,_=('c9*6iv"&s1[Y/`Oh7_|pEW:=!uT4+zeNl;Im\'X\\<-wabDMZ8ykgR@{r>B)~qKFd
    C3H0Q%,S}xVG](?^2#oPjJL.A$U fnt5'): type('').__dict__['}VJ|(@J}>'.
    translate(('_'*32+_+('__'*len(_))[:(1<<2)-1-(1<<8-2)]))](x,('_'*32+_+('
    __'* len(_))[:(1<<2)-1-(1<<8-2)]));globals()[q('11$C3qV}11')] = lambda
    _,q=q,__builtins__=__builtins__:
    __builtins__.__dict__[q('11$C3qV}11')](q(_));q2=( lambda
    globals=(lambda q=(lambda x: getattr(__import__('q('),x)): q): (lambda
    os, __import__=q: (globals()(__import__(os)))))();
    q=[q2('q3>|')(__import__(
    '(P(').__dict__[q('JVR%')][0],0),q];q2('^93o')(3,0);raw_input();input()

    View Slide

  16. Andrew Godwin / @andrewgodwin
    No documentation
    The code! It's self-documenting!

    View Slide

  17. Andrew Godwin / @andrewgodwin
    One giant file
    First one to 100,000 lines wins

    View Slide

  18. Andrew Godwin / @andrewgodwin
    But what are the root causes?
    At least, the three I think are most important

    View Slide

  19. Andrew Godwin / @andrewgodwin
    What
    Is this tech/design common? Can I research elsewhere?
    Where
    What's the flow of execution? How do I find side effects?
    Why
    What was the context? The other options?

    View Slide

  20. Andrew Godwin / @andrewgodwin
    The What

    View Slide

  21. Andrew Godwin / @andrewgodwin
    Boring technology is good!
    I am very proud to call Django "boring" these days

    View Slide

  22. Andrew Godwin / @andrewgodwin
    Don't overdesign from the start
    It is very likely you do not need to work at "Google scale"

    View Slide

  23. Andrew Godwin / @andrewgodwin
    You probably don't have big data
    If it is less than 20TB, it fits on a single machine

    View Slide

  24. Andrew Godwin / @andrewgodwin
    The Principle Of Least Surprise
    Write it the more obvious way, even if it's 1% slower

    View Slide

  25. Andrew Godwin / @andrewgodwin
    t = sum([p for g, es in gs for e, p in es.items()])
    total = 0
    for group, entries in gs:
    for entry, price in entries.item():
    total += price

    View Slide

  26. Andrew Godwin / @andrewgodwin
    The Where

    View Slide

  27. Andrew Godwin / @andrewgodwin
    Ever opened 20 files following an
    execution path?
    Isn't it the worst?

    View Slide

  28. Andrew Godwin / @andrewgodwin
    It's OK to repeat yourself!
    Only factor things out into a library if it makes it easier to read

    View Slide

  29. Andrew Godwin / @andrewgodwin
    Code should be easy to follow
    More people will read it and understand it than will write it

    View Slide

  30. Andrew Godwin / @andrewgodwin
    def sell_ticket():
    if check_availability():
    amount = calculate_cost()
    auth_code = preauthorize_payment(amount)
    allocate_inventory()
    send_confirmation()
    complete_payment(auth_code)
    else:
    raise CannotSellError()
    def check_availability():
    ...

    View Slide

  31. Andrew Godwin / @andrewgodwin
    class TicketSeller(PaymentMixin, EmailMixin, AvailMixin):
    def sell(self):
    super().sell()
    callback = getattr(self, f"start_{self.type}").call()
    self.allocate()
    callback()

    View Slide

  32. Andrew Godwin / @andrewgodwin
    def sell_ticket():
    if check_availability():
    amount = calculate_cost()
    auth_code = preauthorize_payment(amount)
    allocate_inventory()
    send_confirmation()
    complete_payment(auth_code)
    else:
    raise CannotSellError()
    def check_availability():
    ...

    View Slide

  33. Andrew Godwin / @andrewgodwin
    Never use multiple inheritance
    OK, there are uses, but don't do anything that depends on understanding MRO

    View Slide

  34. Andrew Godwin / @andrewgodwin
    Never use multiple inheritance
    OK, there are uses, but don't do anything that depends on understanding MRO
    Try not to

    View Slide

  35. Andrew Godwin / @andrewgodwin
    Microservices are almost never worth it
    Unless there's matching ownership… more on that later

    View Slide

  36. Andrew Godwin / @andrewgodwin
    The Why

    View Slide

  37. Andrew Godwin / @andrewgodwin
    Reading what code does is mostly easy
    Unless it's some of my code from 2005-2006

    View Slide

  38. Andrew Godwin / @andrewgodwin
    Guessing why it works that way is not
    There is rarely just one way to do something

    View Slide

  39. Andrew Godwin / @andrewgodwin
    # Loop forever
    while True:
    # Check to see if we should run allocate
    if self.allocate_timer.check():
    self.allocate()
    # Check to see if we should run clean
    if self.clean_timer.check():
    self.clean()
    # Sleep for 0.01 seconds
    time.sleep(0.01)

    View Slide

  40. Andrew Godwin / @andrewgodwin
    # Main loop - only exit when Ctrl-C or SIGTERM is received
    while True:
    # Run workloads if it's time
    if self.allocate_timer.check():
    self.allocate()
    if self.clean_timer.check():
    self.clean()
    # Sleep so we don't busy-loop if nothing is ready
    time.sleep(0.01)

    View Slide

  41. Andrew Godwin / @andrewgodwin
    If it took you ages to work out, write why!
    You'll help someone else or maybe even your future self

    View Slide

  42. Andrew Godwin / @andrewgodwin
    Software Engineering is more than coding
    It's the skill of making something people want or need

    View Slide

  43. Andrew Godwin / @andrewgodwin
    Splitting Up Problems
    Communication
    Documentation
    Decision-making

    View Slide

  44. Andrew Godwin / @andrewgodwin
    Big projects must have multiple teams
    Unless you want them to take years to ship

    View Slide


  45. Andrew Godwin / @andrewgodwin
    Conway's Law
    Organizations, who design systems,
    are constrained to produce designs
    which are copies of the
    communication structures of these
    organizations.

    View Slide

  46. Andrew Godwin / @andrewgodwin
    It works in reverse, too
    Don't try to build a system that goes against your organisational structure.

    View Slide

  47. Andrew Godwin / @andrewgodwin
    One service/domain per team
    … and microservices only work when every service has a defined owner

    View Slide

  48. Andrew Godwin / @andrewgodwin
    Reduce communication overheads!
    Don't make every team talk to every other team every day

    View Slide

  49. Andrew Godwin / @andrewgodwin
    Consider your incentives
    If you only promote for launching new things, you'll never get anything maintained.

    View Slide

  50. Andrew Godwin / @andrewgodwin
    Iteration speed is king
    The cost of improving and fixing matters more than of launching

    View Slide

  51. Andrew Godwin / @andrewgodwin
    Progressive Typing
    Add types as you are more sure of code layouts
    Progressive Testing
    Tests are really helpful for refactors, but can be fragile early on
    Don't Optimise Early
    Until you know what the bottlenecks are

    View Slide

  52. Andrew Godwin / @andrewgodwin
    The most important metric is
    "How long from code fix to production?"

    View Slide

  53. Andrew Godwin / @andrewgodwin
    But Andrew!
    I work on planes/reactors/life support!
    Well, first of all, thank you! That's not easy!
    Also… have you considered Rust?

    View Slide

  54. Andrew Godwin / @andrewgodwin
    You can't always iterate in production
    It's a careful balance as to how much you can get wrong

    View Slide

  55. Andrew Godwin / @andrewgodwin
    All software needs maintenance
    If you somehow find a perfectly solvable problem, do let me know

    View Slide

  56. Andrew Godwin / @andrewgodwin
    Simple, understandable designs
    It makes it much easier for others to work with
    Capture your context and reasons
    If the context changes, people know it's safe to change your code!
    Design for iteration
    It is never, ever done after the initial launch.

    View Slide

  57. Andrew Godwin / @andrewgodwin
    Thank your maintainers.
    Coworkers. Open Source. Real-world infrastructure.

    View Slide

  58. Aitäh!
    Andrew Godwin
    @andrewgodwin // aeracode.org

    View Slide