Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Lost Chapter: Rediscovering Foundations
 in Software Design

The Lost Chapter: Rediscovering Foundations
 in Software Design

Lemi Orhan Ergin

October 07, 2023
Tweet

More Decks by Lemi Orhan Ergin

Other Decks in Technology

Transcript

  1. THE LOST CHAPTER
    Rediscovering Foundations
    in So
    ft
    LEMİ ORHAN ERGİN
    co-founder, craftgate

    View full-size slide

  2. THE LOST CHAPTER
    Rediscovering Foundations
    in So
    ft
    LEMİ ORHAN ERGİN
    co-founder, craftgate

    View full-size slide

  3. We make the most significant
    decisions when we know least
    Adding new features becomes
    harder and harder in time
    Complexity rises till the dead end
    You either redesign or rewrite to
    continue adding new features
    Microservices transformation
    projects usually aim to fix these
    kinds of legacy projects
    LIFE OF A SOFTWARE
    Feature
    Count
    Time
    Growth
    Limit
    Growth
    Limit
    let’s split it into
    microservices
    we need to
    rewrite it again
    Difficulty Level
    While Adding
    New Features
    Time
    refactor
    redesign
    zone
    Growth
    Limit
    Growth
    Limit
    the increase in
    endurance level is
    due to the new senior
    endurance level:
    do not touch
    if it works
    transfor-
    mation
    project
    zone
    go-live
    night

    View full-size slide

  4. No need for big redesigns or
    rewrites, refactoring is enough
    Refactoring should be continuous
    The system should be open to
    extension and close to modification
    We learn from experiences and
    adapt the system accordingly
    No direct need for microservices,
    expansion in monolith should be fine
    Feature
    Count
    Time
    Difficulty Level
    While Adding
    New Features
    Time
    LIFE OF A SOFTWARE
    HOW IT SHOULD BE

    View full-size slide

  5. AN ORDINARY
    STORY OF
    LOVE & HATE
    OF AN API
    EVOLUTION

    View full-size slide

  6. Payment API
    after a payment are completed,
    we save the data to database
    every design starts with
    the simpliest solution
    IMPLEMENT PAYMENT OPERATIONS 1
    Save payment
    Retrieve payment
    AN ORDINARY
    STORY OF
    LOVE &
    HATE
    OF AN API
    EVOLUTION

    View full-size slide

  7. Payment API
    We need to search existing payments
    FEATURE
    REQUEST
    where do we need to add the search operation ?
    It is a payment operation, so add it to same service ?
    Save payment
    Retrieve payment

    View full-size slide

  8. Payment API
    it’s a simple as querying the database
    with a search criteria and returning the results
    we add it to payment component as
    a new responsibility, a new flow
    2
    ADD SEARCH TO EXISTING FLOW
    Save payment
    Retrieve payment
    Search payment
    start with the simplest solution

    View full-size slide

  9. Payment API
    Search service requires additional models, introduces new dependencies and differentiates
    from core payment flow
    payment service starts to be huge and unmaintainable
    (new dependencies for search operation are not
    used and relevant with core payment flow)
    search flow has different business needs
    and triggered by different roles
    a change in the repository for search improvements
    directly triggers change in core payment flow
    Save payment
    Retrieve payment
    Search payment

    View full-size slide

  10. Payment API
    SEPARATE SEARCH AS A NEW SERVICE 3
    separate the search logic from core payment logic,
    introduce a new service with its own dependencies
    Save payment
    Retrieve payment
    Search payment

    View full-size slide

  11. Payment API
    Single place for doing db operations becomes unmaintainable, includes many responsibilities
    too many reponsibilities included, attracts more
    functionality as a single point for DB operations
    Save payment
    Retrieve payment
    Search payment

    View full-size slide

  12. Payment API
    separate the search db operation from other db
    operations
    SPLIT INTO BOUNDED CONTEXTS 4
    Save payment
    Retrieve payment
    Search payment
    H
    E
    X
    A O
    G
    N

    View full-size slide

  13. Search queries takes time and start to have impact on save queries at database level
    both services depend on the same data and
    the performance of one side affects the other
    we moved read operations into a new repository
    Payment API
    Save payment
    Retrieve payment
    Search payment

    View full-size slide

  14. MASTER REPLICA
    search and payment flows do not depend on
    the same database anymore
    Payment API
    Save payment
    Retrieve payment
    Search payment
    CHANGE THE DATA SOURCE 5
    Read
    Write

    View full-size slide

  15. MASTER REPLICA
    Search results are expected to be returned as files in csv and excel formats
    FEATURE
    REQUEST
    Payment API
    Save payment
    Retrieve payment
    Search payment
    Read
    Write

    View full-size slide

  16. Payment API
    MASTER REPLICA
    generate report simply searches the data and
    fills it in a file with the given extension
    ADD NEW FLOW TO SEARCH MODULE 6
    Save payment
    Retrieve payment
    Search payment
    Generate report
    Read
    Write

    View full-size slide

  17. Payment API
    MASTER REPLICA
    File generation is time and memory consuming, and affects performance of other operations
    deploying the whole api is required whenever something
    added to report logic, that increases the risc on payment
    Save payment
    Retrieve payment
    Search payment
    Generate report
    Read
    Write

    View full-size slide

  18. Payment API
    MASTER REPLICA
    Report API
    it’s time to introduce a new reporting api
    with generate report capabilities
    MOVE TO A NEW MICRO SERVICE 7
    Save payment
    Retrieve payment
    Generate report
    Read
    Write Read
    Search payment

    View full-size slide

  19. Report generation takes time and users wait too long, timeout errors start to occur
    Payment API
    MASTER REPLICA
    Report API
    things go even worse when multiple users
    trigger reports at the same time
    Save payment
    Retrieve payment
    Generate report
    Search payment
    Read
    Write Read

    View full-size slide

  20. Payment API
    MASTER REPLICA
    Report API
    async
    users just fire and forget,
    then check if report is ready
    TRIGGER GENERATE REPORT ASYNC 8
    Save payment
    Retrieve payment
    Generate report
    Search payment
    Read
    Write Read

    View full-size slide

  21. Users re-demand reports before previous ones complete, that creates unnecessary load
    Payment API
    MASTER REPLICA
    Report API
    async
    users are usually impatient,
    they re-trigger report generation
    again and again
    Save payment
    Retrieve payment
    Generate report
    Search payment
    Read
    Write Read

    View full-size slide

  22. Payment API
    MASTER REPLICA
    Report API
    async
    lock
    adding a lock per user lets us to
    prevent from being overloaded by
    the same duplicate report generations
    Save payment
    Retrieve payment
    Generate report
    Search payment
    THROTTLE DEMANDS FROM PER USER 9
    Read
    Write Read

    View full-size slide

  23. Payment API
    MASTER REPLICA
    Report API
    async
    lock
    Throttling demands per user does not prevent system from overloading by report demands
    by multiple users. Limit parallel report generations but not the demands.
    throttling demands per user is not enough,
    multiple demands overload the system
    Save payment
    Retrieve payment
    Generate report
    Search payment
    Read
    Write Read

    View full-size slide

  24. Payment API
    MASTER REPLICA
    Report API
    QUEUE
    each demand is queued,
    the throttlling logic can
    be managed
    Save payment
    Retrieve payment
    Generate report
    Search payment
    Demand report
    QUEUE DEMANDS AND PROCESS 10
    Read
    Write Read

    View full-size slide

  25. Payment API
    MASTER REPLICA
    Report API
    QUEUE
    Report generation affects performance of search operations
    since the data source is shared, the load of reporting
    api has direct impact on payment operations
    Generate report
    Demand report
    Save payment
    Retrieve payment
    Search payment
    Read
    Write Read

    View full-size slide

  26. Payment API
    MASTER REPLICA
    Write Read
    Report API
    Read
    QUEUE
    ELASTIC
    separate the read model of
    report generation from the
    model of payment operations
    Generate report
    Demand report
    Save payment
    Retrieve payment
    Search payment
    THROTTLE DEMANDS FROM PER USER 11

    View full-size slide

  27. Payment API
    MASTER REPLICA
    Report API
    ELASTIC
    Inform the user when the report is generated
    FEATURE
    REQUEST
    Save payment
    Retrieve payment
    Search payment
    QUEUE
    Generate report
    Demand report
    Report API
    Write Read Read

    View full-size slide

  28. Payment API
    MASTER REPLICA ELASTIC
    Notification API
    call
    email
    notification api directly sends
    emails and slack messages
    that provided by the report api
    INTRODUCE NOTIFICATION SERVICE 12
    Save payment
    Retrieve payment
    Search payment
    Report API
    QUEUE
    Generate report
    Demand report
    Report API
    Send Slack message
    Send email
    Write Read Read

    View full-size slide

  29. Payment API
    MASTER REPLICA ELASTIC
    Notification API
    call
    email
    Report needs to know notification api and how to trigger its functionalities
    If notification api stops
    working, we lose messages
    Save payment
    Retrieve payment
    Search payment
    Report API
    QUEUE
    Generate report
    Demand report
    Report API
    Send Slack message
    Send email
    Write Read Read

    View full-size slide

  30. Payment API
    MASTER REPLICA ELASTIC
    email
    QUEUE
    keep messages in queue and manage
    the error logic with retries
    QUEUE NOTIFICATION MESSAGES 13
    Save payment
    Retrieve payment
    Search payment
    Report API
    QUEUE
    Generate report
    Demand report
    Report API Notification API
    Send Slack message
    Send email
    Write Read Read

    View full-size slide

  31. Payment API
    MASTER REPLICA ELASTIC
    email
    QUEUE
    Whenever queue stops responding, the whole flow stops and messages are lost
    we lose messages when the queue is
    down :(
    Save payment
    Retrieve payment
    Search payment
    Report API
    QUEUE
    Generate report
    Demand report
    Report API Notification API
    Send Slack message
    Send email
    Write Read Read

    View full-size slide

  32. Payment API
    MASTER REPLICA ELASTIC
    email
    Outbox
    Inbox
    make messaging transactional and
    be sure you create messages
    MAKE MESSAGING TRANSACTIONAL 14
    Save payment
    Retrieve payment
    Search payment
    Report API
    Generate report
    Demand report
    Report API Notification API
    Send Slack message
    Send email
    Generate
    report
    Send
    notification
    Write Read Read

    View full-size slide

  33. Payment API
    MASTER REPLICA ELASTIC
    email
    Outbox
    Inbox
    We design a kind-a maintainable, expandable, robust system at the end
    make messaging transactional and
    be sure you create messages
    Save payment
    Retrieve payment
    Search payment
    Report API
    Generate report
    Demand report
    Report API Notification API
    Send Slack message
    Send email
    Generate
    report
    Send
    notification
    Write Read Read

    View full-size slide

  34. https://twitter.com/KentBeck/status/1218307926818869248
    %80 of software development
    is maintenance, pareto ?
    Adding every new feature and
    every trouble you got trigger
    a change in the system, and the
    way you refactor the existing
    system determines how easy
    that can be.

    View full-size slide

  35. https://twitter.com/alexbunardzic/status/1697259515593564178
    REFACTORING
    MAXIMIZES COHESION
    AND MINIMIZES COUPLING

    View full-size slide

  36. Extract to method, to classes, to functions
    Centralize decision making (behavior)
    Make responsible of one behavior
    Apply design patters
    Apply design principles like SOLID
    Clearify boundaries of contexts
    Extract functionality as a new microservice
    Seperate by behaviors, not models or packages
    Avoid layered arc, build with hexagonal arch
    Avoid wrappers around CRUD operations
    Do not depend on same datasource
    Avoid distributed monoliths
    Avoid centralized common modules
    Do not depend on implementations
    Let the control of flow inverted (IoC)
    Prefer duplication to make independent
    Introduce abstrations to eliminate down times
    WE MANAGED
    COMPLEXITY
    BY DECREASING
    COUPLING AND
    INCREASING
    COHESION WHILE
    REFACTORING
    AND ADDING
    NEW CODE

    View full-size slide

  37. COUPLING & COHESION
    Structured Design book copyrighted on 1978, 1975
    the most important fundemantal
    in software design
    Minimum Cost Systems
    Black Boxes
    Modular Design
    Modularity
    Software Engineering
    Complexity
    Coupling
    Cohesion
    Morphology of Simple Systems
    Design Heuristics
    Refactoring
    Transaction Analysis
    Communication in Modular Systems
    Packaging
    Optimization of Modular Systems
    Typology of System Components
    Recursive Structures
    Incremental Structures
    Program Quality
    Top-down Implementation
    Bottom-up Development
    Management Benefits
    THE LOST CHAPTER
    * the chapter has already been there for 40+ years
    FOR MANAGING SOFTWARE COMPLEXITY

    View full-size slide

  38. https://twitter.com/KentBeck/status/10668695983
    THE LAWS OF PHYSICS OF
    SOFTWARE DESIGN

    View full-size slide

  39. COUPLING
    In highly coupled systems, one change
    triggers a change in somewhere else
    The measure of the strength of
    interconnection, measure of independence
    If one system waits, changes, works, fails,
    starts, stops due to another systems action,
    these systems are highly coupled
    with the butterfly effect, every change
    thiggers something bigger
    Controller
    Service Service Service
    Repository Repository
    Utility
    Rest
    Client

    View full-size slide

  40. COHESION
    The code that changes together (i.e. in the
    same feature), stays together (only coupled
    to each other)
    Belonging together, relatedness, sharing
    same purpose/responsibility
    Cohesiveness exists inside boundaries, that
    makes it a module, a component, an element

    View full-size slide

  41. Controller
    Service Service Service
    Repository Repository
    Utility
    Rest
    Client
    Layered communication increases coupling
    and destroys cohesion by every change in time

    View full-size slide

  42. WE ARE
    HIRING
    mail: [email protected]
    CRAFTGATE
    one-stop shop payment orchestration
    payment hub for your virtual poses
    with dynamic pos routing, you always
    pay the lowest commission to banks
    craftgate.io @craftgateio
    https://tidyfirst.substack.com/p/how-i-came-to-write-tidy-first
    The goal of software design is to create
    chunks or slices that fit into a human
    mind. The software keeps growing but
    the human mind maxes out, so we have
    to keep chunking and slicing differently
    if we want to keep making changes.
    https://twitter.com/KentBeck/status/1354418068869398538
    Kent Beck

    View full-size slide

  43. One-Stop-Shop
    Payment Orchestration
    [email protected]
    we are hiring!
    LEMİ ORHAN ERGİN
    co-founder, craftgate

    View full-size slide