Fulfilling Data Needs between Microservices

Sometimes an application is split into multiple smaller services, each responsible for a certain aspect of the application. Doing this correctly mandates that each service has its own storage. But what if one service needs data from another service? Let's find out some ways of achieving this!

Transcript

1. Fulfilling Data Needs between Microservices
2. Need an idea for a talk...
3. Something about how we do Microservices!
4. https://www.primevideotech.com/video-​ streaming/scaling-​ up-​ the-​ prime-​ video-​ audio-​ video-​ monitoring-​ service-​ and-​ reducing-​ costs-​ by-​ 90 Then this happened...
5. Now Microservices are basically dead and my talk is pointless
6. Fullfilling Data Needs between Microservices Bounded Contexts
7. What is a "microservice"? What are "data needs"?
8. What is a "microservice"? Autonomous Specialized Develop, Build, Deploy independently Data ownership Communicate through well-​ defined API's Knowledge isolation https://aws.amazon.com/microservices/
9. What are "data needs"? When a service needs Data it doesn't own To perform a task from another service
10. Task 1 Return data you don't own
11. Task 1 Return data you don't own Request
12. Concrete example Service Projects is requested to return the details of a specific project, together with the name of the customer
13. Concrete example Projects Customers Customer ID Name Address Project ID Title Customer 443f16e3-​ e398-4e57-88ea-9fa4972be6a0 443f16e3-​ e398-4e57-88ea-9fa4972b Bezos Company Parkway Drive 666, 90210 Bvrly Hills 17a4b5ac-38f0-4945-​ b3a5-5aed968ce0b0 Install Solar Panels
14. Solution Projects Customers API Gateway one of many solutions actually Routes the request Consolidates responses Protocol translation {   ​ "id" : "17a4b5ac-38f0-4945-​ b3a5-5aed968ce0b0",   ​ "title" : "Install Solar Panels",   ​ "customer" : {   ​   ​ "type": "customer",   ​   ​ "id" : "443f16e3-​ e398-4e57-88ea-9fa4972be6a0"   ​ } } {   ​ "id" : "443f16e3-​ e398-4e57-88ea-9fa4972be6a0",   ​ "Name" : "Bezos Company",   ​ "address" : "Parkway Drive 666, 90210 Bvrly Hills" } {   ​ "id" : "17a4b5ac-38f0-4945-​ b3a5-5aed968ce0b0",   ​ "title" : "Install Solar Panels",   ​ "customer" : {   ​   ​ "id" : "443f16e3-​ e398-4e57-88ea-9fa4972be6a0",   ​   ​ "Name" : "Bezos Company",   ​   ​ "address" : "Parkway Drive 666, 90210 Bvrly Hills"   ​ } } API Gateway
15. Task 1 Return data you don't own Request Solution: introduce a separate component AKA not a microservice's task
16. Task 2 Use data for a business process
17. Concrete example A project's initial title is constructed from the customer name and an incrementing number
18. Projects Solution #1 Customers Database sharing Just read from their database!
19. Solution #1Database sharing Pro's Cons Why even do microservices?? Easy Fast High "implementation" coupling
20. What is a "microservice"? Autonomous Specialized Develop, Build, Deploy independently Data ownership Communicate through well-​ defined API's Knowledge isolation https://aws.amazon.com/microservices/
21. Should not even be considered as a solution
22. Solving the implementation coupling leads to...
23. Projects Solution #2API Calls Just use their API! Customers
24. Solution #2API Calls Pro's Cons Loose "implementation" coupling High "temporal" coupling aka "latency" But... seems acceptable Very microservice'y! API Contract
25. Fast-​ forward in time...
26. Projects Solution #2API Calls Customers Invoicing The Internet Currency conversion
27. Solution #2API Calls Pro's Cons Loose "implementation" coupling High "temporal" coupling But... seems acceptable Very microservice'y!
28. But what about caching?
30. Solution #2API Calls Sync calls Async calls Eventually needs caching UX Hell
31. How to do it well?
32. API Contracts Make your API a 1st Class Citizen it was implied in the definition of a Microservice JSON Schema
33. Solving the temporal coupling leads to...
34. Projects Solution #3Event-​ Carried State Transfer Customers Service Bus event event Just use events to transfer data!
35. Solution #3Event-​ Carried State Transfer Customers Service Bus T r a n s l a t e Domain Event Customer Activated Customer Updated Produce Domain events 1. 3. Send out integration event 2. Translate into 1 integration event with all entity data {   ​ "action" : "UPDATE",   ​ "id" : "17a4b5ac-38f0-4945-​ b3a5-5aed968ce0b0",   ​ "data" : {   ​   ​ "first_name" : "Josh",   ​   ​ "last_name" : "Holme",   ​   ​ // ...   ​ } } Producing Events
36. Solution #3Event-​ Carried State Transfer Fat Event https://verraes.net/2019/05/patterns-​ for-​ decoupling-​ distsys-​ segregated-​ event-​ layers/ https://verraes.net/2019/05/patterns-​ for-​ decoupling-​ distsys-​ fat-​ event/ Segregated Event Layers Rich
37. Projects Solution #3Event-​ Carried State Transfer Service Bus {   ​ "action" : "UPDATE",   ​ "id" : "17a4b5ac-38f0-4945-​ b3a5-5aed968ce0b0",   ​ "data" : {   ​   ​ "first_name" : "Josh",   ​   ​ "last_name" : "Holme",   ​   ​ // ...   ​ } } Customer Updated A C L Ingest integration event 1. 2. Pick interesting data 3. Store in a read model ID | NAME 17a4b5ac-38f0-4945-​ b3a5-5aed968ce0b0 | Josh Holme Consuming Events
38. Solution #3Event-​ Carried State Transfer Anti-​ Corruption Layer https://learn.microsoft.com/en-​ us/azure/architecture/patterns/cqrs https://www.youtube.com/watch?​ v=Dok2Ikcjaro Read Model
39. Solution #3Event-​ Carried State Transfer https://www.youtube.com/watch?​ v=IzBEbfSg0uY
40. Solution #3Event-​ Carried State Transfer Pro's Cons Loose "implementation" coupling Loose "temporal" coupling Data duplication Is this really a problem these days? Eventual Consistency
41. How to do it well?
42. Event Contracts Make your Events 1st Class Citizens
43. Solving the data duplication & eventual consistency leads to...
46. https://particular.net/webinars/2023-​ live-​ qa-​ with-​ udi
48. Projects Solution #4Service Composition Customers C C Just inject a component!
49. Solution #4Service Composition Component Piece of functionality, exposed as a reusable package
50. Solution #4Service Composition Component Created & maintained where it logically belongs Inside the knowledge boundary or "system"
51. How to do it well? experimental ! in PHP
52. Guard the boundaries C o m p o n e n t O t h e r c o d e exposed hidden Cannot depend on each other https://qossmic.github.io/deptrac/ same codebase
53. e.g. Github Actions Mirror to Read-​ Only repository C o m p o n e n t O t h e r c o d e C o m p o n e n t same repository separate repository mirror
54. Re-​ use s e r v i c e C o m p o n e n t composer require // composer.json {   ​ "repositories": [   ​   ​ {   ​   ​   ​ "type": "vcs", "no-​ api": true, "url": "<read-​ only repository>"   ​   ​ } } (the Read-​ only repository)
55. Decouple s e r v i c e Component Interfaces depends on & uses for typehinting depends on implements Dependency-​ inversion principle (separate repository)
56. Just like PSR!
57. Why?
58. To avoid cyclic dependencies Customers C L i b r a r y depends on depends on v1 depends on v2 conflict!
59. Some heuristics
60. Solution #4Service Composition Component Should only do reads and/or business logic No writes in its knowledge boundary
61. Solution #4Service Composition Component Should not do writes to your database Service no longer retains data ownership
62. Solution #4Service Composition Component Interface(s) are distributed separately To break cyclic dependencies
63. Solution #4Service Composition Component Caching is not its responsibility Use decorator pattern & cache locally
64. Solution #4Service Composition Component Ideally has a single purpose It's not a library!
65. Quick Recap
66. Task 1 Return data you don't own
67. Solution Projects Customers API Gateway API Gateway
68. Task 2 Use data for a business process
69. Projects Solution #1 Customers Database sharing seriously... don't!
70. Projects Solution #2API Calls Customers
71. Projects Solution #3Event-​ Carried State Transfer Customers Service Bus event event
72. Projects Solution #4Service Composition Customers C C
73. Questions?
74. THANK YOU!