$30 off During Our Annual Pro Sale. View Details »

Fulfilling Data Needs between Microservices

Fulfilling Data Needs between Microservices

Sometimes an application is split into multiple smaller services, each responsible for a certain aspect of the application. Doing this correctly mandates that each service has its own storage. But what if one service needs data from another service? Let's find out some ways of achieving this!

Tom Van Herreweghe

May 17, 2023
Tweet

More Decks by Tom Van Herreweghe

Other Decks in Technology

Transcript

  1. Fulfilling Data
    Needs
    between
    Microservices

    View Slide

  2. Need an idea
    for a talk...

    View Slide

  3. Something about how
    we do Microservices!

    View Slide

  4. https://www.primevideotech.com/video-​
    streaming/scaling-​
    up-​
    the-​
    prime-​
    video-​
    audio-​
    video-​
    monitoring-​
    service-​
    and-​
    reducing-​
    costs-​
    by-​
    90
    Then this happened...

    View Slide

  5. Now Microservices
    are basically dead
    and my talk is
    pointless

    View Slide

  6. Fullfilling Data
    Needs
    between
    Microservices
    Bounded Contexts

    View Slide

  7. What is a "microservice"?
    What are "data needs"?

    View Slide

  8. What is a "microservice"?
    Autonomous
    Specialized
    Develop, Build, Deploy independently
    Data ownership
    Communicate through well-​
    defined API's
    Knowledge isolation
    https://aws.amazon.com/microservices/

    View Slide

  9. What are "data needs"?
    When a service needs
    Data it doesn't own
    To perform a task from another service

    View Slide

  10. Task 1
    Return data you
    don't own

    View Slide

  11. Task 1
    Return data you
    don't own
    Request

    View Slide

  12. Concrete example
    Service Projects is requested
    to return the details of a specific
    project, together with the name
    of the customer

    View Slide

  13. Concrete example
    Projects Customers
    Customer ID
    Name
    Address
    Project ID
    Title
    Customer 443f16e3-​
    e398-4e57-88ea-9fa4972be6a0
    443f16e3-​
    e398-4e57-88ea-9fa4972b
    Bezos Company
    Parkway Drive 666, 90210 Bvrly Hills
    17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0
    Install Solar Panels

    View Slide

  14. Solution
    Projects Customers
    API Gateway
    one of many solutions actually
    Routes the request
    Consolidates responses
    Protocol translation
    {
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "title" : "Install Solar Panels",
      ​
    "customer" : {
      ​
      ​
    "type": "customer",
      ​
      ​
    "id" : "443f16e3-​
    e398-4e57-88ea-9fa4972be6a0"
      ​
    }
    }
    {
      ​
    "id" : "443f16e3-​
    e398-4e57-88ea-9fa4972be6a0",
      ​
    "Name" : "Bezos Company",
      ​
    "address" : "Parkway Drive 666, 90210 Bvrly Hills"
    }
    {
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "title" : "Install Solar Panels",
      ​
    "customer" : {
      ​
      ​
    "id" : "443f16e3-​
    e398-4e57-88ea-9fa4972be6a0",
      ​
      ​
    "Name" : "Bezos Company",
      ​
      ​
    "address" : "Parkway Drive 666, 90210 Bvrly Hills"
      ​
    }
    }
    API Gateway

    View Slide

  15. Task 1
    Return data you
    don't own
    Request
    Solution: introduce a separate
    component
    AKA not a microservice's task

    View Slide

  16. Task 2
    Use data for a
    business process

    View Slide

  17. Concrete example
    A project's initial title is
    constructed from the customer
    name and an incrementing
    number

    View Slide

  18. Projects
    Solution #1
    Customers
    Database sharing
    Just read from their database!

    View Slide

  19. Solution #1Database sharing
    Pro's Cons
    Why even do
    microservices??
    Easy Fast High
    "implementation"
    coupling

    View Slide

  20. What is a "microservice"?
    Autonomous
    Specialized
    Develop, Build, Deploy independently
    Data ownership
    Communicate through well-​
    defined API's
    Knowledge isolation
    https://aws.amazon.com/microservices/

    View Slide

  21. Should not even be
    considered as a
    solution

    View Slide

  22. Solving the
    implementation
    coupling leads to...

    View Slide

  23. Projects
    Solution #2API Calls
    Just use their API!
    Customers

    View Slide

  24. Solution #2API Calls
    Pro's Cons
    Loose
    "implementation"
    coupling
    High
    "temporal"
    coupling
    aka "latency"
    But... seems acceptable
    Very
    microservice'y!
    API Contract

    View Slide

  25. Fast-​
    forward in
    time...

    View Slide

  26. Projects
    Solution #2API Calls
    Customers
    Invoicing
    The
    Internet
    Currency
    conversion

    View Slide

  27. Solution #2API Calls
    Pro's Cons
    Loose
    "implementation"
    coupling
    High
    "temporal"
    coupling
    But... seems
    acceptable
    Very
    microservice'y!

    View Slide

  28. But what about
    caching?

    View Slide

  29. View Slide

  30. Solution #2API Calls
    Sync calls
    Async calls
    Eventually needs caching
    UX Hell

    View Slide

  31. How to do it well?

    View Slide

  32. API Contracts
    Make your API a 1st Class Citizen
    it was implied in the definition of a Microservice
    JSON Schema

    View Slide

  33. Solving the
    temporal coupling
    leads to...

    View Slide

  34. Projects
    Solution #3Event-​
    Carried State Transfer
    Customers
    Service Bus
    event
    event
    Just use events to transfer data!

    View Slide

  35. Solution #3Event-​
    Carried State Transfer
    Customers
    Service Bus
    T
    r
    a
    n
    s
    l
    a
    t
    e
    Domain
    Event
    Customer
    Activated
    Customer
    Updated
    Produce Domain events
    1.
    3. Send out
    integration event
    2. Translate into 1 integration
    event with all entity data
    {
      ​
    "action" : "UPDATE",
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "data" : {
      ​
      ​
    "first_name" : "Josh",
      ​
      ​
    "last_name" : "Holme",
      ​
      ​
    // ...
      ​
    }
    }
    Producing Events

    View Slide

  36. Solution #3Event-​
    Carried State Transfer
    Fat Event
    https://verraes.net/2019/05/patterns-​
    for-​
    decoupling-​
    distsys-​
    segregated-​
    event-​
    layers/
    https://verraes.net/2019/05/patterns-​
    for-​
    decoupling-​
    distsys-​
    fat-​
    event/
    Segregated
    Event Layers
    Rich

    View Slide

  37. Projects
    Solution #3Event-​
    Carried State Transfer
    Service Bus
    {
      ​
    "action" : "UPDATE",
      ​
    "id" : "17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0",
      ​
    "data" : {
      ​
      ​
    "first_name" : "Josh",
      ​
      ​
    "last_name" : "Holme",
      ​
      ​
    // ...
      ​
    }
    }
    Customer
    Updated
    A
    C
    L
    Ingest integration event
    1.
    2. Pick interesting data
    3. Store in a
    read model
    ID | NAME
    17a4b5ac-38f0-4945-​
    b3a5-5aed968ce0b0 | Josh Holme
    Consuming Events

    View Slide

  38. Solution #3Event-​
    Carried State Transfer
    Anti-​
    Corruption Layer
    https://learn.microsoft.com/en-​
    us/azure/architecture/patterns/cqrs
    https://www.youtube.com/watch?​
    v=Dok2Ikcjaro
    Read Model

    View Slide

  39. Solution #3Event-​
    Carried State Transfer
    https://www.youtube.com/watch?​
    v=IzBEbfSg0uY

    View Slide

  40. Solution #3Event-​
    Carried State Transfer
    Pro's Cons
    Loose
    "implementation"
    coupling
    Loose
    "temporal"
    coupling
    Data
    duplication
    Is this really a
    problem these days?
    Eventual
    Consistency

    View Slide

  41. How to do it well?

    View Slide

  42. Event Contracts
    Make your Events 1st Class Citizens

    View Slide

  43. Solving the data
    duplication &
    eventual
    consistency leads
    to...

    View Slide

  44. View Slide

  45. View Slide

  46. https://particular.net/webinars/2023-​
    live-​
    qa-​
    with-​
    udi

    View Slide

  47. View Slide

  48. Projects
    Solution #4Service Composition
    Customers
    C
    C
    Just inject a component!

    View Slide

  49. Solution #4Service Composition
    Component Piece of functionality,
    exposed as a
    reusable package

    View Slide

  50. Solution #4Service Composition
    Component Created & maintained
    where it logically
    belongs
    Inside the knowledge
    boundary or "system"

    View Slide

  51. How to do it well?
    experimental !
    in PHP

    View Slide

  52. Guard the boundaries
    C
    o
    m
    p
    o
    n
    e
    n
    t
    O
    t
    h
    e
    r
    c
    o
    d
    e
    exposed
    hidden
    Cannot depend on each other
    https://qossmic.github.io/deptrac/
    same codebase

    View Slide

  53. e.g. Github Actions
    Mirror to Read-​
    Only repository
    C
    o
    m
    p
    o
    n
    e
    n
    t
    O
    t
    h
    e
    r
    c
    o
    d
    e
    C
    o
    m
    p
    o
    n
    e
    n
    t
    same repository separate repository
    mirror

    View Slide

  54. Re-​
    use
    s
    e
    r
    v
    i
    c
    e
    C
    o
    m
    p
    o
    n
    e
    n
    t
    composer require
    // composer.json
    {
      ​
    "repositories": [
      ​
      ​
    {
      ​
      ​
      ​
    "type": "vcs",
    "no-​
    api": true,
    "url": "only repository>"
      ​
      ​
    }
    }
    (the Read-​
    only repository)

    View Slide

  55. Decouple
    s
    e
    r
    v
    i
    c
    e
    Component
    Interfaces
    depends on
    & uses for
    typehinting
    depends on
    implements
    Dependency-​
    inversion
    principle
    (separate repository)

    View Slide

  56. Just like PSR!

    View Slide

  57. Why?

    View Slide

  58. To avoid cyclic dependencies
    Customers
    C
    L
    i
    b
    r
    a
    r
    y
    depends on
    depends on
    v1
    depends on v2
    conflict!

    View Slide

  59. Some heuristics

    View Slide

  60. Solution #4Service Composition
    Component Should only do reads
    and/or business logic
    No writes in its
    knowledge boundary

    View Slide

  61. Solution #4Service Composition
    Component Should not do writes to
    your database
    Service no longer
    retains data ownership

    View Slide

  62. Solution #4Service Composition
    Component Interface(s) are
    distributed separately
    To break cyclic
    dependencies

    View Slide

  63. Solution #4Service Composition
    Component Caching is not its
    responsibility
    Use decorator
    pattern & cache
    locally

    View Slide

  64. Solution #4Service Composition
    Component Ideally has a single
    purpose
    It's not a library!

    View Slide

  65. Quick Recap

    View Slide

  66. Task 1
    Return data you
    don't own

    View Slide

  67. Solution
    Projects Customers
    API Gateway
    API Gateway

    View Slide

  68. Task 2
    Use data for a
    business process

    View Slide

  69. Projects
    Solution #1
    Customers
    Database sharing
    seriously... don't!

    View Slide

  70. Projects
    Solution #2API Calls
    Customers

    View Slide

  71. Projects
    Solution #3Event-​
    Carried State Transfer
    Customers
    Service Bus
    event
    event

    View Slide

  72. Projects
    Solution #4Service Composition
    Customers
    C
    C

    View Slide

  73. Questions?

    View Slide

  74. THANK YOU!

    View Slide