Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Lost Chapter: Rediscovering Foundations
 in...

The Lost Chapter: Rediscovering Foundations
 in Software Design

Lemi Orhan Ergin

October 07, 2023
Tweet

More Decks by Lemi Orhan Ergin

Other Decks in Technology

Transcript

  1. We make the most significant decisions when we know least

    Adding new features becomes harder and harder in time Complexity rises till the dead end You either redesign or rewrite to continue adding new features Microservices transformation projects usually aim to fix these kinds of legacy projects LIFE OF A SOFTWARE Feature Count Time Growth Limit Growth Limit let’s split it into microservices we need to rewrite it again Difficulty Level While Adding New Features Time refactor redesign zone Growth Limit Growth Limit the increase in endurance level is due to the new senior endurance level: do not touch if it works transfor- mation project zone go-live night
  2. No need for big redesigns or rewrites, refactoring is enough

    Refactoring should be continuous The system should be open to extension and close to modification We learn from experiences and adapt the system accordingly No direct need for microservices, expansion in monolith should be fine Feature Count Time Difficulty Level While Adding New Features Time LIFE OF A SOFTWARE HOW IT SHOULD BE
  3. Payment API after a payment are completed, we save the

    data to database every design starts with the simpliest solution IMPLEMENT PAYMENT OPERATIONS 1 Save payment Retrieve payment AN ORDINARY STORY OF LOVE & HATE OF AN API EVOLUTION
  4. Payment API We need to search existing payments FEATURE REQUEST

    where do we need to add the search operation ? It is a payment operation, so add it to same service ? Save payment Retrieve payment
  5. Payment API it’s a simple as querying the database with

    a search criteria and returning the results we add it to payment component as a new responsibility, a new flow 2 ADD SEARCH TO EXISTING FLOW Save payment Retrieve payment Search payment start with the simplest solution
  6. Payment API Search service requires additional models, introduces new dependencies

    and differentiates from core payment flow payment service starts to be huge and unmaintainable (new dependencies for search operation are not used and relevant with core payment flow) search flow has different business needs and triggered by different roles a change in the repository for search improvements directly triggers change in core payment flow Save payment Retrieve payment Search payment
  7. Payment API SEPARATE SEARCH AS A NEW SERVICE 3 separate

    the search logic from core payment logic, introduce a new service with its own dependencies Save payment Retrieve payment Search payment
  8. Payment API Single place for doing db operations becomes unmaintainable,

    includes many responsibilities too many reponsibilities included, attracts more functionality as a single point for DB operations Save payment Retrieve payment Search payment
  9. Payment API separate the search db operation from other db

    operations SPLIT INTO BOUNDED CONTEXTS 4 Save payment Retrieve payment Search payment H E X A O G N
  10. Search queries takes time and start to have impact on

    save queries at database level both services depend on the same data and the performance of one side affects the other we moved read operations into a new repository Payment API Save payment Retrieve payment Search payment
  11. MASTER REPLICA search and payment flows do not depend on

    the same database anymore Payment API Save payment Retrieve payment Search payment CHANGE THE DATA SOURCE 5 Read Write
  12. MASTER REPLICA Search results are expected to be returned as

    files in csv and excel formats FEATURE REQUEST Payment API Save payment Retrieve payment Search payment Read Write
  13. Payment API MASTER REPLICA generate report simply searches the data

    and fills it in a file with the given extension ADD NEW FLOW TO SEARCH MODULE 6 Save payment Retrieve payment Search payment Generate report Read Write
  14. Payment API MASTER REPLICA File generation is time and memory

    consuming, and affects performance of other operations deploying the whole api is required whenever something added to report logic, that increases the risc on payment Save payment Retrieve payment Search payment Generate report Read Write
  15. Payment API MASTER REPLICA Report API it’s time to introduce

    a new reporting api with generate report capabilities MOVE TO A NEW MICRO SERVICE 7 Save payment Retrieve payment Generate report Read Write Read Search payment
  16. Report generation takes time and users wait too long, timeout

    errors start to occur Payment API MASTER REPLICA Report API things go even worse when multiple users trigger reports at the same time Save payment Retrieve payment Generate report Search payment Read Write Read
  17. Payment API MASTER REPLICA Report API async users just fire

    and forget, then check if report is ready TRIGGER GENERATE REPORT ASYNC 8 Save payment Retrieve payment Generate report Search payment Read Write Read
  18. Users re-demand reports before previous ones complete, that creates unnecessary

    load Payment API MASTER REPLICA Report API async users are usually impatient, they re-trigger report generation again and again Save payment Retrieve payment Generate report Search payment Read Write Read
  19. Payment API MASTER REPLICA Report API async lock adding a

    lock per user lets us to prevent from being overloaded by the same duplicate report generations Save payment Retrieve payment Generate report Search payment THROTTLE DEMANDS FROM PER USER 9 Read Write Read
  20. Payment API MASTER REPLICA Report API async lock Throttling demands

    per user does not prevent system from overloading by report demands by multiple users. Limit parallel report generations but not the demands. throttling demands per user is not enough, multiple demands overload the system Save payment Retrieve payment Generate report Search payment Read Write Read
  21. Payment API MASTER REPLICA Report API QUEUE each demand is

    queued, the throttlling logic can be managed Save payment Retrieve payment Generate report Search payment Demand report QUEUE DEMANDS AND PROCESS 10 Read Write Read
  22. Payment API MASTER REPLICA Report API QUEUE Report generation affects

    performance of search operations since the data source is shared, the load of reporting api has direct impact on payment operations Generate report Demand report Save payment Retrieve payment Search payment Read Write Read
  23. Payment API MASTER REPLICA Write Read Report API Read QUEUE

    ELASTIC separate the read model of report generation from the model of payment operations Generate report Demand report Save payment Retrieve payment Search payment THROTTLE DEMANDS FROM PER USER 11
  24. Payment API MASTER REPLICA Report API ELASTIC Inform the user

    when the report is generated FEATURE REQUEST Save payment Retrieve payment Search payment QUEUE Generate report Demand report Report API Write Read Read
  25. Payment API MASTER REPLICA ELASTIC Notification API call email notification

    api directly sends emails and slack messages that provided by the report api INTRODUCE NOTIFICATION SERVICE 12 Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Send Slack message Send email Write Read Read
  26. Payment API MASTER REPLICA ELASTIC Notification API call email Report

    needs to know notification api and how to trigger its functionalities If notification api stops working, we lose messages Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Send Slack message Send email Write Read Read
  27. Payment API MASTER REPLICA ELASTIC email QUEUE keep messages in

    queue and manage the error logic with retries QUEUE NOTIFICATION MESSAGES 13 Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Notification API Send Slack message Send email Write Read Read
  28. Payment API MASTER REPLICA ELASTIC email QUEUE Whenever queue stops

    responding, the whole flow stops and messages are lost we lose messages when the queue is down :( Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Notification API Send Slack message Send email Write Read Read
  29. Payment API MASTER REPLICA ELASTIC email Outbox Inbox make messaging

    transactional and be sure you create messages MAKE MESSAGING TRANSACTIONAL 14 Save payment Retrieve payment Search payment Report API Generate report Demand report Report API Notification API Send Slack message Send email Generate report Send notification Write Read Read
  30. Payment API MASTER REPLICA ELASTIC email Outbox Inbox We design

    a kind-a maintainable, expandable, robust system at the end make messaging transactional and be sure you create messages Save payment Retrieve payment Search payment Report API Generate report Demand report Report API Notification API Send Slack message Send email Generate report Send notification Write Read Read
  31. https://twitter.com/KentBeck/status/1218307926818869248 %80 of software development is maintenance, pareto ? Adding

    every new feature and every trouble you got trigger a change in the system, and the way you refactor the existing system determines how easy that can be.
  32. Extract to method, to classes, to functions Centralize decision making

    (behavior) Make responsible of one behavior Apply design patters Apply design principles like SOLID Clearify boundaries of contexts Extract functionality as a new microservice Seperate by behaviors, not models or packages Avoid layered arc, build with hexagonal arch Avoid wrappers around CRUD operations Do not depend on same datasource Avoid distributed monoliths Avoid centralized common modules Do not depend on implementations Let the control of flow inverted (IoC) Prefer duplication to make independent Introduce abstrations to eliminate down times WE MANAGED COMPLEXITY BY DECREASING COUPLING AND INCREASING COHESION WHILE REFACTORING AND ADDING NEW CODE
  33. COUPLING & COHESION Structured Design book copyrighted on 1978, 1975

    the most important fundemantal in software design Minimum Cost Systems Black Boxes Modular Design Modularity Software Engineering Complexity Coupling Cohesion Morphology of Simple Systems Design Heuristics Refactoring Transaction Analysis Communication in Modular Systems Packaging Optimization of Modular Systems Typology of System Components Recursive Structures Incremental Structures Program Quality Top-down Implementation Bottom-up Development Management Benefits THE LOST CHAPTER * the chapter has already been there for 40+ years FOR MANAGING SOFTWARE COMPLEXITY
  34. COUPLING In highly coupled systems, one change triggers a change

    in somewhere else The measure of the strength of interconnection, measure of independence If one system waits, changes, works, fails, starts, stops due to another systems action, these systems are highly coupled with the butterfly effect, every change thiggers something bigger Controller Service Service Service Repository Repository Utility Rest Client
  35. COHESION The code that changes together (i.e. in the same

    feature), stays together (only coupled to each other) Belonging together, relatedness, sharing same purpose/responsibility Cohesiveness exists inside boundaries, that makes it a module, a component, an element
  36. Controller Service Service Service Repository Repository Utility Rest Client Layered

    communication increases coupling and destroys cohesion by every change in time
  37. WE ARE HIRING mail: [email protected] CRAFTGATE one-stop shop payment orchestration

    payment hub for your virtual poses with dynamic pos routing, you always pay the lowest commission to banks craftgate.io @craftgateio https://tidyfirst.substack.com/p/how-i-came-to-write-tidy-first The goal of software design is to create chunks or slices that fit into a human mind. The software keeps growing but the human mind maxes out, so we have to keep chunking and slicing differently if we want to keep making changes. https://twitter.com/KentBeck/status/1354418068869398538 Kent Beck