THE LOST CHAPTER Rediscovering Foundations in So ft LEMİ ORHAN ERGİN co-founder, craftgate

THE LOST CHAPTER Rediscovering Foundations in So ft LEMİ ORHAN ERGİN co-founder, craftgate

We make the most significant decisions when we know least Adding new features becomes harder and harder in time Complexity rises till the dead end You either redesign or rewrite to continue adding new features Microservices transformation projects usually aim to fix these kinds of legacy projects LIFE OF A SOFTWARE Feature Count Time Growth Limit Growth Limit let’s split it into microservices we need to rewrite it again Difficulty Level While Adding New Features Time refactor redesign zone Growth Limit Growth Limit the increase in endurance level is due to the new senior endurance level: do not touch if it works transfor- mation project zone go-live night

No need for big redesigns or rewrites, refactoring is enough Refactoring should be continuous The system should be open to extension and close to modification We learn from experiences and adapt the system accordingly No direct need for microservices, expansion in monolith should be fine Feature Count Time Difficulty Level While Adding New Features Time LIFE OF A SOFTWARE HOW IT SHOULD BE

AN ORDINARY STORY OF LOVE & HATE OF AN API EVOLUTION

Payment API after a payment are completed, we save the data to database every design starts with the simpliest solution IMPLEMENT PAYMENT OPERATIONS 1 Save payment Retrieve payment AN ORDINARY STORY OF LOVE & HATE OF AN API EVOLUTION

Payment API We need to search existing payments FEATURE REQUEST where do we need to add the search operation ? It is a payment operation, so add it to same service ? Save payment Retrieve payment

Payment API it’s a simple as querying the database with a search criteria and returning the results we add it to payment component as a new responsibility, a new flow 2 ADD SEARCH TO EXISTING FLOW Save payment Retrieve payment Search payment start with the simplest solution

Payment API Search service requires additional models, introduces new dependencies and differentiates from core payment flow payment service starts to be huge and unmaintainable (new dependencies for search operation are not used and relevant with core payment flow) search flow has different business needs and triggered by different roles a change in the repository for search improvements directly triggers change in core payment flow Save payment Retrieve payment Search payment

Payment API SEPARATE SEARCH AS A NEW SERVICE 3 separate the search logic from core payment logic, introduce a new service with its own dependencies Save payment Retrieve payment Search payment

Payment API Single place for doing db operations becomes unmaintainable, includes many responsibilities too many reponsibilities included, attracts more functionality as a single point for DB operations Save payment Retrieve payment Search payment

Payment API separate the search db operation from other db operations SPLIT INTO BOUNDED CONTEXTS 4 Save payment Retrieve payment Search payment H E X A O G N

Search queries takes time and start to have impact on save queries at database level both services depend on the same data and the performance of one side affects the other we moved read operations into a new repository Payment API Save payment Retrieve payment Search payment

MASTER REPLICA search and payment flows do not depend on the same database anymore Payment API Save payment Retrieve payment Search payment CHANGE THE DATA SOURCE 5 Read Write

MASTER REPLICA Search results are expected to be returned as files in csv and excel formats FEATURE REQUEST Payment API Save payment Retrieve payment Search payment Read Write

Payment API MASTER REPLICA generate report simply searches the data and fills it in a file with the given extension ADD NEW FLOW TO SEARCH MODULE 6 Save payment Retrieve payment Search payment Generate report Read Write

Payment API MASTER REPLICA File generation is time and memory consuming, and affects performance of other operations deploying the whole api is required whenever something added to report logic, that increases the risc on payment Save payment Retrieve payment Search payment Generate report Read Write

Payment API MASTER REPLICA Report API it’s time to introduce a new reporting api with generate report capabilities MOVE TO A NEW MICRO SERVICE 7 Save payment Retrieve payment Generate report Read Write Read Search payment

Report generation takes time and users wait too long, timeout errors start to occur Payment API MASTER REPLICA Report API things go even worse when multiple users trigger reports at the same time Save payment Retrieve payment Generate report Search payment Read Write Read

Payment API MASTER REPLICA Report API async users just fire and forget, then check if report is ready TRIGGER GENERATE REPORT ASYNC 8 Save payment Retrieve payment Generate report Search payment Read Write Read

Users re-demand reports before previous ones complete, that creates unnecessary load Payment API MASTER REPLICA Report API async users are usually impatient, they re-trigger report generation again and again Save payment Retrieve payment Generate report Search payment Read Write Read

Payment API MASTER REPLICA Report API async lock adding a lock per user lets us to prevent from being overloaded by the same duplicate report generations Save payment Retrieve payment Generate report Search payment THROTTLE DEMANDS FROM PER USER 9 Read Write Read

Payment API MASTER REPLICA Report API async lock Throttling demands per user does not prevent system from overloading by report demands by multiple users. Limit parallel report generations but not the demands. throttling demands per user is not enough, multiple demands overload the system Save payment Retrieve payment Generate report Search payment Read Write Read

Payment API MASTER REPLICA Report API QUEUE each demand is queued, the throttlling logic can be managed Save payment Retrieve payment Generate report Search payment Demand report QUEUE DEMANDS AND PROCESS 10 Read Write Read

Payment API MASTER REPLICA Report API QUEUE Report generation affects performance of search operations since the data source is shared, the load of reporting api has direct impact on payment operations Generate report Demand report Save payment Retrieve payment Search payment Read Write Read

Payment API MASTER REPLICA Write Read Report API Read QUEUE ELASTIC separate the read model of report generation from the model of payment operations Generate report Demand report Save payment Retrieve payment Search payment THROTTLE DEMANDS FROM PER USER 11

Payment API MASTER REPLICA Report API ELASTIC Inform the user when the report is generated FEATURE REQUEST Save payment Retrieve payment Search payment QUEUE Generate report Demand report Report API Write Read Read

Payment API MASTER REPLICA ELASTIC Notification API call email notification api directly sends emails and slack messages that provided by the report api INTRODUCE NOTIFICATION SERVICE 12 Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Send Slack message Send email Write Read Read

Payment API MASTER REPLICA ELASTIC Notification API call email Report needs to know notification api and how to trigger its functionalities If notification api stops working, we lose messages Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Send Slack message Send email Write Read Read

Payment API MASTER REPLICA ELASTIC email QUEUE keep messages in queue and manage the error logic with retries QUEUE NOTIFICATION MESSAGES 13 Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Notification API Send Slack message Send email Write Read Read

Payment API MASTER REPLICA ELASTIC email QUEUE Whenever queue stops responding, the whole flow stops and messages are lost we lose messages when the queue is down :( Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Notification API Send Slack message Send email Write Read Read

Payment API MASTER REPLICA ELASTIC email Outbox Inbox make messaging transactional and be sure you create messages MAKE MESSAGING TRANSACTIONAL 14 Save payment Retrieve payment Search payment Report API Generate report Demand report Report API Notification API Send Slack message Send email Generate report Send notification Write Read Read

Payment API MASTER REPLICA ELASTIC email Outbox Inbox We design a kind-a maintainable, expandable, robust system at the end make messaging transactional and be sure you create messages Save payment Retrieve payment Search payment Report API Generate report Demand report Report API Notification API Send Slack message Send email Generate report Send notification Write Read Read

https://twitter.com/KentBeck/status/1218307926818869248 %80 of software development is maintenance, pareto ? Adding every new feature and every trouble you got trigger a change in the system, and the way you refactor the existing system determines how easy that can be.

https://twitter.com/alexbunardzic/status/1697259515593564178 REFACTORING MAXIMIZES COHESION AND MINIMIZES COUPLING

Extract to method, to classes, to functions Centralize decision making (behavior) Make responsible of one behavior Apply design patters Apply design principles like SOLID Clearify boundaries of contexts Extract functionality as a new microservice Seperate by behaviors, not models or packages Avoid layered arc, build with hexagonal arch Avoid wrappers around CRUD operations Do not depend on same datasource Avoid distributed monoliths Avoid centralized common modules Do not depend on implementations Let the control of flow inverted (IoC) Prefer duplication to make independent Introduce abstrations to eliminate down times WE MANAGED COMPLEXITY BY DECREASING COUPLING AND INCREASING COHESION WHILE REFACTORING AND ADDING NEW CODE

COUPLING & COHESION Structured Design book copyrighted on 1978, 1975 the most important fundemantal in software design Minimum Cost Systems Black Boxes Modular Design Modularity Software Engineering Complexity Coupling Cohesion Morphology of Simple Systems Design Heuristics Refactoring Transaction Analysis Communication in Modular Systems Packaging Optimization of Modular Systems Typology of System Components Recursive Structures Incremental Structures Program Quality Top-down Implementation Bottom-up Development Management Benefits THE LOST CHAPTER * the chapter has already been there for 40+ years FOR MANAGING SOFTWARE COMPLEXITY

https://twitter.com/KentBeck/status/10668695983 THE LAWS OF PHYSICS OF SOFTWARE DESIGN

COUPLING In highly coupled systems, one change triggers a change in somewhere else The measure of the strength of interconnection, measure of independence If one system waits, changes, works, fails, starts, stops due to another systems action, these systems are highly coupled with the butterfly effect, every change thiggers something bigger Controller Service Service Service Repository Repository Utility Rest Client

COHESION The code that changes together (i.e. in the same feature), stays together (only coupled to each other) Belonging together, relatedness, sharing same purpose/responsibility Cohesiveness exists inside boundaries, that makes it a module, a component, an element

Controller Service Service Service Repository Repository Utility Rest Client Layered communication increases coupling and destroys cohesion by every change in time

WE ARE HIRING mail: [email protected] CRAFTGATE one-stop shop payment orchestration payment hub for your virtual poses with dynamic pos routing, you always pay the lowest commission to banks craftgate.io @craftgateio https://tidyfirst.substack.com/p/how-i-came-to-write-tidy-first The goal of software design is to create chunks or slices that fit into a human mind. The software keeps growing but the human mind maxes out, so we have to keep chunking and slicing differently if we want to keep making changes. https://twitter.com/KentBeck/status/1354418068869398538 Kent Beck

One-Stop-Shop Payment Orchestration [email protected] we are hiring! LEMİ ORHAN ERGİN co-founder, craftgate