Fulfilling Data Needs between Microservices

Sometimes an application is split into multiple smaller services, each responsible for a certain aspect of the application. Doing this correctly mandates that each service has its own storage. But what if one service needs data from another service? Let's find out some ways of achieving this!

Transcript

1. Fulfilling Data
   Needs
   between
   Microservices
   

   View Slide

2. Need an idea
   for a talk...
   

   View Slide

3. Something about how
   we do Microservices!
   

   View Slide

4. https://www.primevideotech.com/video-​
   streaming/scaling-​
   up-​
   the-​
   prime-​
   video-​
   audio-​
   video-​
   monitoring-​
   service-​
   and-​
   reducing-​
   costs-​
   by-​
   90
   Then this happened...
   

   View Slide

5. Now Microservices
   are basically dead
   and my talk is
   pointless
   

   View Slide

6. Fullfilling Data
   Needs
   between
   Microservices
   Bounded Contexts
   

   View Slide

7. What is a "microservice"?
   What are "data needs"?
   

   View Slide

8. What is a "microservice"?
   Autonomous
   Specialized
   Develop, Build, Deploy independently
   Data ownership
   Communicate through well-​
   defined API's
   Knowledge isolation
   https://aws.amazon.com/microservices/
   

   View Slide

9. What are "data needs"?
   When a service needs
   Data it doesn't own
   To perform a task from another service
   

   View Slide

10. Task 1
    Return data you
    don't own
    

    View Slide

11. Task 1
    Return data you
    don't own
    Request
    

    View Slide

12. Concrete example
    Service Projects is requested
    to return the details of a specific
    project, together with the name
    of the customer
    

    View Slide

13. Concrete example
    Projects Customers
    Customer ID
    Name
    Address
    Project ID
    Title
    Customer 443f16e3-​
    e398-4e57-88ea-9fa4972be6a0
    443f16e3-​
    e398-4e57-88ea-9fa4972b
    Bezos Company
    Parkway Drive 666, 90210 Bvrly Hills
    17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0
    Install Solar Panels
    

    View Slide

14. Solution
    Projects Customers
    API Gateway
    one of many solutions actually
    Routes the request
    Consolidates responses
    Protocol translation
    {
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "title" : "Install Solar Panels",
      ​
    "customer" : {
      ​
      ​
    "type": "customer",
      ​
      ​
    "id" : "443f16e3-​
    e398-4e57-88ea-9fa4972be6a0"
      ​
    }
    }
    {
      ​
    "id" : "443f16e3-​
    e398-4e57-88ea-9fa4972be6a0",
      ​
    "Name" : "Bezos Company",
      ​
    "address" : "Parkway Drive 666, 90210 Bvrly Hills"
    }
    {
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "title" : "Install Solar Panels",
      ​
    "customer" : {
      ​
      ​
    "id" : "443f16e3-​
    e398-4e57-88ea-9fa4972be6a0",
      ​
      ​
    "Name" : "Bezos Company",
      ​
      ​
    "address" : "Parkway Drive 666, 90210 Bvrly Hills"
      ​
    }
    }
    API Gateway
    

    View Slide

15. Task 1
    Return data you
    don't own
    Request
    Solution: introduce a separate
    component
    AKA not a microservice's task
    

    View Slide

16. Task 2
    Use data for a
    business process
    

    View Slide

17. Concrete example
    A project's initial title is
    constructed from the customer
    name and an incrementing
    number
    

    View Slide

18. Projects
    Solution #1
    Customers
    Database sharing
    Just read from their database!
    

    View Slide

19. Solution #1Database sharing
    Pro's Cons
    Why even do
    microservices??
    Easy Fast High
    "implementation"
    coupling
    

    View Slide

20. What is a "microservice"?
    Autonomous
    Specialized
    Develop, Build, Deploy independently
    Data ownership
    Communicate through well-​
    defined API's
    Knowledge isolation
    https://aws.amazon.com/microservices/
    

    View Slide

21. Should not even be
    considered as a
    solution
    

    View Slide

22. Solving the
    implementation
    coupling leads to...
    

    View Slide

23. Projects
    Solution #2API Calls
    Just use their API!
    Customers
    

    View Slide

24. Solution #2API Calls
    Pro's Cons
    Loose
    "implementation"
    coupling
    High
    "temporal"
    coupling
    aka "latency"
    But... seems acceptable
    Very
    microservice'y!
    API Contract
    

    View Slide

25. Fast-​
    forward in
    time...
    

    View Slide

26. Projects
    Solution #2API Calls
    Customers
    Invoicing
    The
    Internet
    Currency
    conversion
    

    View Slide

27. Solution #2API Calls
    Pro's Cons
    Loose
    "implementation"
    coupling
    High
    "temporal"
    coupling
    But... seems
    acceptable
    Very
    microservice'y!
    

    View Slide

28. But what about
    caching?
    

    View Slide

29. View Slide

30. Solution #2API Calls
    Sync calls
    Async calls
    Eventually needs caching
    UX Hell
    

    View Slide

31. How to do it well?
    

    View Slide

32. API Contracts
    Make your API a 1st Class Citizen
    it was implied in the definition of a Microservice
    JSON Schema
    

    View Slide

33. Solving the
    temporal coupling
    leads to...
    

    View Slide

34. Projects
    Solution #3Event-​
    Carried State Transfer
    Customers
    Service Bus
    event
    event
    Just use events to transfer data!
    

    View Slide

35. Solution #3Event-​
    Carried State Transfer
    Customers
    Service Bus
    T
    r
    a
    n
    s
    l
    a
    t
    e
    Domain
    Event
    Customer
    Activated
    Customer
    Updated
    Produce Domain events
    1.
    3. Send out
    integration event
    2. Translate into 1 integration
    event with all entity data
    {
      ​
    "action" : "UPDATE",
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "data" : {
      ​
      ​
    "first_name" : "Josh",
      ​
      ​
    "last_name" : "Holme",
      ​
      ​
    // ...
      ​
    }
    }
    Producing Events
    

    View Slide

36. Solution #3Event-​
    Carried State Transfer
    Fat Event
    https://verraes.net/2019/05/patterns-​
    for-​
    decoupling-​
    distsys-​
    segregated-​
    event-​
    layers/
    https://verraes.net/2019/05/patterns-​
    for-​
    decoupling-​
    distsys-​
    fat-​
    event/
    Segregated
    Event Layers
    Rich
    

    View Slide

37. Projects
    Solution #3Event-​
    Carried State Transfer
    Service Bus
    {
      ​
    "action" : "UPDATE",
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "data" : {
      ​
      ​
    "first_name" : "Josh",
      ​
      ​
    "last_name" : "Holme",
      ​
      ​
    // ...
      ​
    }
    }
    Customer
    Updated
    A
    C
    L
    Ingest integration event
    1.
    2. Pick interesting data
    3. Store in a
    read model
    ID | NAME
    17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0 | Josh Holme
    Consuming Events
    

    View Slide

38. Solution #3Event-​
    Carried State Transfer
    Anti-​
    Corruption Layer
    https://learn.microsoft.com/en-​
    us/azure/architecture/patterns/cqrs
    https://www.youtube.com/watch?​
    v=Dok2Ikcjaro
    Read Model
    

    View Slide

39. Solution #3Event-​
    Carried State Transfer
    https://www.youtube.com/watch?​
    v=IzBEbfSg0uY
    

    View Slide

40. Solution #3Event-​
    Carried State Transfer
    Pro's Cons
    Loose
    "implementation"
    coupling
    Loose
    "temporal"
    coupling
    Data
    duplication
    Is this really a
    problem these days?
    Eventual
    Consistency
    

    View Slide

41. How to do it well?
    

    View Slide

42. Event Contracts
    Make your Events 1st Class Citizens
    

    View Slide

43. Solving the data
    duplication &
    eventual
    consistency leads
    to...
    

    View Slide

44. View Slide

45. View Slide

46. https://particular.net/webinars/2023-​
    live-​
    qa-​
    with-​
    udi
    

    View Slide

47. View Slide

48. Projects
    Solution #4Service Composition
    Customers
    C
    C
    Just inject a component!
    

    View Slide

49. Solution #4Service Composition
    Component Piece of functionality,
    exposed as a
    reusable package
    

    View Slide

50. Solution #4Service Composition
    Component Created & maintained
    where it logically
    belongs
    Inside the knowledge
    boundary or "system"
    

    View Slide

51. How to do it well?
    experimental !
    in PHP
    

    View Slide

52. Guard the boundaries
    C
    o
    m
    p
    o
    n
    e
    n
    t
    O
    t
    h
    e
    r
    c
    o
    d
    e
    exposed
    hidden
    Cannot depend on each other
    https://qossmic.github.io/deptrac/
    same codebase
    

    View Slide

53. e.g. Github Actions
    Mirror to Read-​
    Only repository
    C
    o
    m
    p
    o
    n
    e
    n
    t
    O
    t
    h
    e
    r
    c
    o
    d
    e
    C
    o
    m
    p
    o
    n
    e
    n
    t
    same repository separate repository
    mirror
    

    View Slide

54. Re-​
    use
    s
    e
    r
    v
    i
    c
    e
    C
    o
    m
    p
    o
    n
    e
    n
    t
    composer require
    // composer.json
    {
      ​
    "repositories": [
      ​
      ​
    {
      ​
      ​
      ​
    "type": "vcs",
    "no-​
    api": true,
    "url": "only repository>"
      ​
      ​
    }
    }
    (the Read-​
    only repository)
    

    View Slide

55. Decouple
    s
    e
    r
    v
    i
    c
    e
    Component
    Interfaces
    depends on
    & uses for
    typehinting
    depends on
    implements
    Dependency-​
    inversion
    principle
    (separate repository)
    

    View Slide

56. Just like PSR!
    

    View Slide

57. Why?
    

    View Slide

58. To avoid cyclic dependencies
    Customers
    C
    L
    i
    b
    r
    a
    r
    y
    depends on
    depends on
    v1
    depends on v2
    conflict!
    

    View Slide

59. Some heuristics
    

    View Slide

60. Solution #4Service Composition
    Component Should only do reads
    and/or business logic
    No writes in its
    knowledge boundary
    

    View Slide

61. Solution #4Service Composition
    Component Should not do writes to
    your database
    Service no longer
    retains data ownership
    

    View Slide

62. Solution #4Service Composition
    Component Interface(s) are
    distributed separately
    To break cyclic
    dependencies
    

    View Slide

63. Solution #4Service Composition
    Component Caching is not its
    responsibility
    Use decorator
    pattern & cache
    locally
    

    View Slide

64. Solution #4Service Composition
    Component Ideally has a single
    purpose
    It's not a library!
    

    View Slide

65. Quick Recap
    

    View Slide

66. Task 1
    Return data you
    don't own
    

    View Slide

67. Solution
    Projects Customers
    API Gateway
    API Gateway
    

    View Slide

68. Task 2
    Use data for a
    business process
    

    View Slide

69. Projects
    Solution #1
    Customers
    Database sharing
    seriously... don't!
    

    View Slide

70. Projects
    Solution #2API Calls
    Customers
    

    View Slide

71. Projects
    Solution #3Event-​
    Carried State Transfer
    Customers
    Service Bus
    event
    event
    

    View Slide

72. Projects
    Solution #4Service Composition
    Customers
    C
    C
    

    View Slide

73. Questions?
    

    View Slide

74. THANK YOU!
    

    View Slide