Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Modeling Pub/Sub system with F# and CQRS (on micro-scale)

Ba6a3fcf8c0d009b8c23c1773a4bb726?s=47 Antya Dev
December 13, 2019

Modeling Pub/Sub system with F# and CQRS (on micro-scale)

Almost all APIs that are built in different companies have a lot in common, but when you have to open several projects, they are very different. Query and error handling, interaction with domain objects and infrastructure layers, etc. shouldn't be so different. All this fake difference only complicates the understanding of the components of the project and their interaction. In this talk, I will share how we build APIs that support different semantic models(Request/Response and Pub/Sub) and various protocols exchange (HTTP/WebSockets/AMQP). I will also try to unlock the potential of the CQRS architectural pattern and explain why we apply it to almost all API projects.

Ba6a3fcf8c0d009b8c23c1773a4bb726?s=128

Antya Dev

December 13, 2019
Tweet

Transcript

  1. Modeling Pub/Sub system with F# and CQRS (on micro scale)

    by @antyadev
  2. The Problem Almost all APIs that are built in different

    companies have a lot in common, but when you have to open several projects, they are very different. Query and error handling, interaction with domain objects and infrastructure layers, etc. shouldn't be so different. All this fake difference only complicates the understanding of the components of the project and their interaction.
  3. @antyadev antyadev@gmail.com

  4. Data Feed WebSites/Terminals

  5. Data Feed F# F# WebSites/Terminals F#

  6. Agenda CQRS in 5 mins F# and CQRS(on micro scale)

    Advantages and disadvantages
  7. CQRS

  8. None
  9. None
  10. None
  11. None
  12. None
  13. CQRS (on micro scale)

  14. Incidents

  15. WebScokets DB

  16. Subscribe

  17. None
  18. { incident }

  19. push { incident }

  20. Command Command

  21. Command Event

  22. Incidents Domain

  23. type IncidentType = Goal | Corner | Penalty | RedCard

    | YellowCard type ParticipantRole = Home | Away type Incident = { Id: IncidentId GameId: GameId Type: IncidentType ParticipantId: ParticipantId ParticipantRole: ParticipantRole option TimeInGame: DateTime GamePart: GamePart option }
  24. type Query = | Incident of IncidentId | IncidentsForGames of

    GameId[] * IncidentType[] * LimitsParams type Command = | AddIncident of IncidentArgs | RemoveIncident of IncidentId | Subscribe of GameId[] * IncidentType[] | Unsubscribe of SubscriptionId type DomainEvent = | IncidentAdded of Incident * subscriptionsToNotify:SubscriptionId[] | IncidentRemoved of IncidentId | SubscriptionCreated of SubscriptionId | SubscriberAdded of SubscriptionId * subscribersCount:uint32 | SubscriptionRemoved of SubscriptionId | SubscriberRemoved of SubscriptionId * subscribersCount:uint32
  25. type ParserError = | GameIdFormat of originValues:string[] | IncidentIdFormat of

    originValues:string[] | IncidentTypeFormat of originValues:string[] type DomainError = | IncidentCantBeCreated of gameId:int * typeId:int | LimitsOutOfRange of top:int * maxTop:int * skip:int | GameNotFound of gameId:int | IncidentNotFound of incidentId:int | IncidentsForGamesNotFound of gameIds:int[] type AppError = | Parser of ParserError | Domain of DomainError static member create(e: ParserError) = e |> Parser static member create(e: DomainError) = e |> Domain static member createResult(e: DomainError) = e |> Domain |> Error static member createResult(e: ParserError) = e |> Parser |> Error
  26. Project development with familiar backbone Domain | Commands | Queries

    | Events CommandHandler | QueryHandler CQRS (on micro scale)
  27. In every project you will see the following structure CQRS

    (on micro scale)
  28. Domain Queries Commands Events Errors Command Handler Query Handler

  29. Command Handler CQRS (on micro scale)

  30. Command Handler Command -> DomainEvent

  31. Command Handler Command -> Result<DomainEvent, AppError>

  32. Command Handler Command -> Async<Result<DomainEvent, AppError>>

  33. Command Handler Command -> Async<Result<DomainEvent, AppError>> TOP LEVEL CONSTRAINT which

    force any new functionality to return result
  34. type Command = | CreateIncident of IncidentArgs | RemoveIncident of

    IncidentId | Subscribe of GameId[] * IncidentType[] | Unsubscribe of SubscriptionId type DomainEvent = | IncidentCreated of Incident * subscriptionsToNotify:SubscriptionId[] | IncidentRemoved of IncidentId | SubscriptionCreated of SubscriptionId | SubscriberAdded of SubscriptionId * subscribersCount:uint32 | SubscriptionRemoved of SubscriptionId | SubscriberRemoved of SubscriptionId * subscribersCount:uint32
  35. let handle (command) = match command with | CreateIncident args

    -> DomainEvent.IncidentCreated | RemoveIncident id -> DomainEvent.IncidentRemoved | Subscribe (gamesIds, incTypes) -> DomainEvent.SubscriptionCreated | Unsubscribe id -> DomainEvent.SubscriptionRemoved type Command = | CreateIncident of IncidentArgs | RemoveIncident of IncidentId | Subscribe of GameId[] * IncidentType[] | Unsubscribe of SubscriptionId type DomainEvent = | IncidentCreated of Incident * subscriptionsToNotify:SubscriptionId[] | IncidentRemoved of IncidentId | SubscriptionCreated of SubscriptionId | SubscriberAdded of SubscriptionId * subscribersCount:uint32 | SubscriptionRemoved of SubscriptionId | SubscriberRemoved of SubscriptionId * subscribersCount:uint32
  36. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = let handle (command) =

    asyncResult { match command with | CreateIncident params -> let! game = storage.FindGame(params.GametId) with match game with | Some game -> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle(command) = handle(command)
  37. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = let handle (command) =

    asyncResult { match command with | CreateIncident params -> let! game = storage.FindGame(params.GametId) with match game with | Some game -> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle
  38. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = let handle (command) =

    asyncResult { match command with | CreateIncident params -> match! storage.FindGame(params.GametId) with | Some game -> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle
  39. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = // handle: Command ->

    Async<Result<DomainEvent, AppError>> let handle (command) = asyncResult { match command with | CreateIncident params -> match! storage.FindGame(params.GametId) with // Task<Option<Game>> | Some game -> // Result<DomainEvent, AppError> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle
  40. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = // handle: Command ->

    Async<Result<DomainEvent, AppError>> let handle (command) = asyncResult { match command with | CreateIncident params -> match! storage.FindGame(params.GametId) with // Task<Option<Game>> | Some game -> // Result<DomainEvent, AppError> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle
  41. module IncidentDomain = let create (params: IncidentArgs, game: IncidentArgs, subscriptions:

    Subscriptions) = if validation(params, game) then let incident = { GameId = game.Id } let subscriptionsToNotify = subscriptions |> Subscriptions.filter incident let event = DomainEvent.IncidentCreated(incident, subscriptionsToNotify) Ok event else Error // domain error
  42. module IncidentDomain = let create (params: IncidentArgs, game: IncidentArgs, subscriptions:

    Subscriptions) = if validation(params, game) then let incident = { GameId = game.Id } let subscriptionsToNotify = subscriptions |> Subscriptions.filter incident Ok DomainEvent.IncidentCreated(incident, subscriptionsToNotify) else Error // domain error
  43. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = // handle: Command ->

    Async<Result<DomainEvent, AppError>> let handle (command) = asyncResult { match command with | CreateIncident params -> match! storage.FindGame(params.GametId) with // Task<Option<Game>> | Some game -> // Result<DomainEvent, AppError> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle
  44. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = let eventStream = new

    Subject<DomainEvent>() let publishEvent (event) = eventStream.OnNext(event) // handle: Command -> Async<Result<DomainEvent, AppError>> let handle (command) = asyncResult { match command with | CreateIncident params -> match! storage.FindGame(params.GametId) with // Task<Option<Game>> | Some game -> // Result<DomainEvent, AppError> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle member x.EventStream = eventStream :> IObservable<DomainEvent>
  45. type CommandHandler(storage: IStorage, subscriptions: Subscriptions) = let eventStream = new

    Subject<DomainEvent>() let publishEvent = eventStream.OnNext // handle: Command -> Async<Result<DomainEvent, AppError>> let handle (command) = asyncResult { match command with | CreateIncident params -> match! storage.FindGame(params.GametId) with // Task<Option<Game>> | Some game -> // Result<DomainEvent, AppError> let! event = IncidentDomain.create(params, game, subscriptions) publishEvent(event) return event | None -> return! GameNotFound(params.GametId) |> AppError.create } member x.Handle = handle member x.EventStream = eventStream :> IObservable<DomainEvent>
  46. Command Handler Handle: Command -> Async<Result<DomainEvent, AppError> EventStream: IObservable<DomainEvent> WebScokets

  47. Command Handler WebSockets Middleware AMQP Middleware connections connections EventStream: IObservable<DomainEvent>

    Dictionary<SubscriptionId, Connections[]> Dictionary<SubscriptionId, Connections[]>
  48. Command Handler IncidentAddedEvent { SubscriptionId = 5,6,7 Incident: { …

    } } EventStream: IObservable<DomainEvent> WebSockets Middleware AMQP Middleware connections connections Dictionary<SubscriptionId, Connections[]> Dictionary<SubscriptionId, Connections[]>
  49. Decoupled components via Pub/Sub messaging (IObservable<T>) publishEvent(event) CQRS (on micro

    scale)
  50. Query Handler CQRS (on micro scale)

  51. Query Handler Query -> Async<Result<Incident[], AppError>>

  52. type QueryHandler(storage: InStorage) = let response (incidents: Incident[]) = incidents

    |> Array.map(Incident.toContractType) let handle (query) = asyncResult { match query with | Incident id -> match! storage.FindIncident(id) with | Some incident -> return response [| incident |] | None -> return response(Array.empty) } member x.Handle = handle
  53. API Response CQRS (on micro scale)

  54. Your Server as a function CQRS (on micro scale)

  55. Your Server as a function HttpRequest -> HttpResponse

  56. Your Server as a function HttpRequest -> Async<HttpResponse>

  57. Your Server as a function HttpRequest -> Query -> QueryHandler

    -> Async<HttpResponse>
  58. Your Server as a function HttpRequest -> Query -> QueryHandler

    -> Async<Result<Incident[], AppError>> -> Async<HttpResponse>
  59. HttpRequest -> Query -> QueryHandler -> Async<Result<Incident[], AppError>> -> Async<HttpResponse>

    WebSocketRequest -> Query -> QueryHandler -> Async<Result<Incident[], AppError>> -> Async<WebSocketResponse> AMQPRequest -> Query -> QueryHandler -> Async<Result<Incident[], AppError>> -> Async<AMQPRequest>
  60. HttpRequest -> Query -> QueryHandler -> Async<Result<Incident[], AppError>> -> Async<HttpResponse>

    WebSocketRequest -> Query -> QueryHandler -> Async<Result<Incident[], AppError>> -> Async<WebSocketResponse> AMQPRequest -> Query -> QueryHandler -> Async<Result<Incident[], AppError>> -> Async<AMQPRequest>
  61. Error responses

  62. type ParserError = | GameIdFormat of originValues:string[] | IncidentIdFormat of

    originValues:string[] | IncidentTypeFormat of originValues:string[] type DomainError = | IncidentCantBeCreated of gameId:int * typeId:int | LimitsOutOfRange of top:int * maxTop:int * skip:int | GameNotFound of gameId:int | IncidentNotFound of incidentId:int | IncidentsForGamesNotFound of gameIds:int[] type AppError = | Parser of ParserError | Domain of DomainError static member create(e: ParserError) = e |> Parser static member create(e: DomainError) = e |> Domain static member createResult(e: DomainError) = e |> Domain |> Error static member createResult(e: ParserError) = e |> Parser |> Error
  63. module HTTPErrorResponse = let map (error: AppError) = ... let

    fromDomainError (error: DomainError) = match error with | IncidentCantBeCreated -> new HttpResponseMessage(HttpStatusCode.BadReqeust, { ... }) let fromParserError (error: ParserError) = match error with | GameIdFormat ids -> new HttpResponseMessage(HttpStatusCode.BadReqeust, { ... }) | IncidentIdFormat ids -> new HttpResponseMessage(HttpStatusCode.BadReqeust, { ... }) match error with | Domain e -> fromDomainError(e) | Parser e -> fromParserError(e)
  64. Testing

  65. Input -> Output Command -> Event

  66. [<Fact>] let ``CommandHandler on AddIncident should return IncidentAdded if related

    game exist`` () = let cmdHandler = TestHelper.createCommandHandler() let game = TestHelper.createGame(id = 777) let incident = TestHelper.createIncident(id = 1, gameId = 777) asyncResult { let! gameEvent = Command.AddGame(game) |> cmdHandler.Handle let! incidentEvent = Command.AddIncident(incident) |> cmdHandler.Handle test <@ match incidentEvent with | IncidentAdded _ -> true | _ -> false @> } |> TestHelper.runTest
  67. What about non-trivial cases with no output?

  68. Testing Log Events

  69. logger.Warning("Test message") logger.Warning("Test message") logger.Warning("Test message") InMemoryLogger .Should() .HaveMessage("Test message")

    .Appearing().Times(3) .WithLevel(LogEventLevel.Information)
  70. [<Fact>] let ``Agent should be able to reconnect automatically and

    join the cluster`` () = async { let coordinatorDep = Dependency.createFor(NodeType.Coordinator) let (agentDep, loggerBuffer) = Dependency.createWithInMemoryLogger(NodeType.Agent) let server = MqttTests.startMqttServer() // we start one agent let! agent = Agent.init(agentDep, [| scenario |], agentSettings) Agent.startListening(agent) |> Async.Start // we stop mqtt server MqttTests.stopMqttServer server // and wait some time do! Async.Sleep(5_000) // spin up the mqtt server again to see that agent will reconnect let server = MqttTests.startMqttServer() let! coordinator = Coordinator.init(coordinatorDep, [| scenario |], coordinatorSettings, [| scenarioSettings |], customSettings) // waiting on reconnect do! Async.Sleep(2_000) let! statsResult = Coordinator.run(coordinator) let allStats = statsResult |> Result.defaultValue Array.empty
  71. Property based testing

  72. Property based testing [<Property>] let ``Subscription filter correctness verification`` (allTypes:

    IncidentType[], typesToSubscribe: IncidentType[]) = type IncidentType = Goal | Corner | Penalty | RedCard | YellowCard
  73. Property based testing allTypes: [Goal; Corner; Penalty], typesToSubscribe: [Corner]) =

    type IncidentType = Goal | Corner | Penalty | RedCard | YellowCard
  74. Property based testing allTypes: [Goal; Corner; Penalty; Corner;], typesToSubscribe: [RedCard;

    YellowCard]) = type IncidentType = Goal | Corner | Penalty | RedCard | YellowCard
  75. [<Property>] let ``Subscription filter corectness verification`` (game: Game, allTypes: IncidentType

    list, typesToSubscribe: IncidentType list) = let cmdHandler = createCmdHandler() let incidents = allTypes |> List.map(fun x -> TestHelper.createIncidents(event) Command.AddGame(game) |> cmdHandler.Handle |> ignore Command.Subscribe([|game.Id|], typesToSubscribe) |> cmdHandler.Handle |> ignore let receivedIncidents = incidents |> List.map(fun x -> Command.AddIncident(x) |> cmdHandler.Handle |> retrieveIncidents |> List.sum let shouldReceive = if List.isEmpty(typesToSubscribe) // subscribed on all types then incidents.Length else let machedTypes = Set.intersect (set allTypes) (set typesToSubscribe) incidents |> List.choose(fun x -> if machedTypes.Contains(enum<IncidentType> x.IncidentTypeId) then Some 1 else None) |> List.sum receivedIncidents = shouldReceive
  76. Part III Advantages and disadvantages

  77. We stopped write frameworks with unlimited extensibility to cover cases

    from future
  78. We stopped write frameworks No abstract factories No factory method

  79. We stopped write frameworks No interfaces per class

  80. We stopped write frameworks No interfaces like ISimpleIdentifiable

  81. We stopped write frameworks Instead, we build our private world

    where we know everything
  82. Simpler reading let publishMsg = Json.serialize >> Zip.compress >> RabbitMq.createMsg

    topicName >> client.Publish
  83. Simpler reading let validate (context: NBomberContext) = context.Scenarios |> checkEmptyName

    >>= checkDuplicateName >>= checkEmptyStepName >>= checkDuration >>= checkConcurrentCopies >>= fun _ -> Ok context |> Result.mapError(AppError.create)
  84. Simpler testing

  85. Type Driven Development

  86. module CardGame = type Suit = Club | Diamond |

    Spade | Heart type Rank = Two | Three | Four | Five | Six | Seven | Eight | Nine | Ten | Jack | Queen | King | Ace type Card = Suit * Rank type Hand = Card list type Deck = Card list type Player = {Name:string; Hand:Hand} type Game = {Deck:Deck; Players: Player list} type Deal = Deck -> (Deck*Card) type PickupCard = (Hand*Card) -> Hand
  87. type DbCommand<'TEntityId,'TEntity,'TChange> = | Insert of 'TEntity | Update of

    'TChange | Delete of 'TEntityId | Clear | CreateMemorySnapshot of TaskCompletionSource<DbSnapshot[]> | Flush | Restore type DbCommandTransaction<'TEntityId,'TEntity,'TChange> = { Commands: DbCommand<'TEntityId,'TEntity,'TChange>[] TaskCompletionSource: TaskCompletionSource<bool> option Version: Version }
  88. Anonymous Types let data = {| X = 1; Y

    = 2 |} let data1 = {| data with Y = 3 |} // {| X:Int; Y:Int |} let data2 = {| data with Z = 3 |} // {| X:Int; Y:Int; Z:Int |}
  89. type User = { Age: int } let user =

    { Age = 1 } let anonymous = {| user with Y=1 |} // {| Age:Int; Y:Int |} Anonymous Types
  90. Expressive API with Explicit Effects

  91. Option<T> - to represent cases when something may exist but

    also may not. Result<TOk, TError> - to represent operations which could be success or fail. Choice<TLeft, TRight> - to model operations which return different values depending on the input Async<T> - to represent asynchronicity as an effect (in addition people quite often use F# Async<T> to model impure operations as alternative to IO monad)
  92. Previously working in C# we were forced to use name

    conventions
  93. // if method starts with Find then in case of

    not found // it returns null public User FindUser(int userId) // if method starts with Get then in case of not found // it throws Exception public User GetUser(int userId)
  94. // if method starts with Try // it means that

    operation can fail and // and exception will be not thrown public bool TryParse(string json, out Config config) // in this case the exception will be thrown public Config Parse(string json)
  95. let findUser (userId): Async<Result<Option<User>, AppError>> let parse (json): Result<Config, ParserError>

  96. Monad Comprehensions

  97. async { } task { } option { } result

    { }
  98. async { } // Async<T> task { } // Task<T>

    option { } // Option<T> result { } // Result<T>
  99. option { let! x = getSomething() let! y = getSomethingElse()

    let! z = andYetGetSomethingElse() // Code will only hit this point if the three // operations above return Some value return x + y + z }
  100. Monad Transformers

  101. asyncResult { } asyncResultOption { } taskResult { } taskResultOption

    { } async { } task { } option { } result { }
  102. type QueryHandler(storage: IStorage) = let handle (query) = asyncResult {

    match query with | Incidents (eventIds, incidentTypes, limits) -> // vLimits: Result<LimitsParams, AppError> let! vLimits = QueryValidation.validateLimits(limits) // incidents: Async<Result<Incident[], AppError>> let! incidents = storage.FindIncidents(eventIds, incidentTypes, vLimits) let data = incidents |> Array.map(Incident.toContractType) return { Data = data } } member x.Handle = handle
  103. Useful Links

  104. FsToolkit.ErrorHandling https://github.com/demystifyfp/FsToolkit.ErrorHandling F# unit test assertions https://github.com/SwensenSoftware/unquote In-memory sink for

    Serilog to use for testing https://github.com/sandermvanvliet/SerilogSinksInMemory
  105. THANKS @antyadev antyadev@gmail.com