Event Sourcing in practice @ Qandidate.com

A talk I gave first at Domcode Conference 2015.

Transcript

1. Event sourcing in production Willem-Jan Zijderveld @willemjanz wjzijderveld@gmail.com joind.in/16227
2. Event sourcing
3. Short introduction into our domain
4. Job Marketing Platform
5. Choosing the best channels to find your candidates
6. Online Job Marketing
7. 1. Create a campaign for your target group
8. 2. You get a recommendation
9. 3. You can customize your campaign
10. Why event sourcing?
11. You are throwing away data!
12. By only saving last known state
13. You don't know the previous state
14. You don't know why something changed
15. How does event sourcing help?
16. Using events to store changes
17. Describe with your events what happened
18. Describe with your events why something happened
19. Use the events to calculate the current state
20. Allows you to go back in time
21. Helps to understand the flow in your application
22. Example: Discarding the customizations
23. CRUD
24. Keep track if channel was part of recommendation
25. Keep track if channel was part of recommendation Remove the non-recommended channels
26. Keep track if channel was part of recommendation How about removed channels?
27. Keep track if channel was part of recommendation Implement soft-delete
28. Keep track if channel was part of recommendation Implement soft-delete for new campaigns
29. Event sourcing
30. Just rewind
31. Just rewind Campaign Created For Target Group Channel Removed From Campaign Channel Added To Campaign Customization Discarded
32. Just rewind Campaign Created For Target Group Channel Removed From Campaign Channel Added To Campaign Customization Discarded
33. Just rewind Campaign Created For Target Group Channel Removed From Campaign Channel Added To Campaign Customization Discarded
34. Just rewind Campaign Created For Target Group Channel Removed From Campaign Channel Added To Campaign Customization Discarded
35. How?
36. With the help of CQRS and Domain Driven Design
37. And the help of Broadway github.com/qandidate-labs/broadway
38. Creating a new campaign
39. Campaign Event Store Create Campaign For Target Group Campaign Created For Target Group
40. Dispatching the command /** @Route("/campaigns", methods={"POST"}) */ public function createCampaignAction(CreateCampaignDto $dto) { $campaignId = Webshop\CampaignId::generate(); $this->commandBus->dispatch(new CreateCampaignForTargetGroup( $campaignId, $this->companyId, $this->accountId, TargetGroup::describe( TargetGroup\JobTitle::fromString($dto->title), TargetGroup\Region::fromId($dto->region), TargetGroup\JobCategory::fromId($dto->jobCategory), TargetGroup\EducationLevel::fromId($dto->educationLevel), TargetGroup\JobLevel::fromId($dto->jobLevel), TargetGroup\Industry::fromId($dto->industry) ) )); return new JsonResponse( ['id' => (string) $campaignId], JsonResponse::HTTP_CREATED ); }
41. Dispatching the command /** @Route("/campaigns", methods={"POST"}) */ public function createCampaignAction(CreateCampaignDto $dto) { $campaignId = Webshop\CampaignId::generate(); $this->commandBus->dispatch(new CreateCampaignForTargetGroup( $campaignId, $this->companyId, $this->accountId, TargetGroup::describe( TargetGroup\JobTitle::fromString($dto->title), TargetGroup\Region::fromId($dto->region), TargetGroup\JobCategory::fromId($dto->jobCategory), TargetGroup\EducationLevel::fromId($dto->educationLevel), TargetGroup\JobLevel::fromId($dto->jobLevel), TargetGroup\Industry::fromId($dto->industry) ) )); return new JsonResponse( ['id' => (string) $campaignId], JsonResponse::HTTP_CREATED ); }
42. Dispatching the command /** @Route("/campaigns", methods={"POST"}) */ public function createCampaignAction(CreateCampaignDto $dto) { $campaignId = Webshop\CampaignId::generate(); $this->commandBus->dispatch(new CreateCampaignForTargetGroup( $campaignId, $this->companyId, $this->accountId, TargetGroup::describe( TargetGroup\JobTitle::fromString($dto->title), TargetGroup\Region::fromId($dto->region), TargetGroup\JobCategory::fromId($dto->jobCategory), TargetGroup\EducationLevel::fromId($dto->educationLevel), TargetGroup\JobLevel::fromId($dto->jobLevel), TargetGroup\Industry::fromId($dto->industry) ) )); return new JsonResponse( ['id' => (string) $campaignId], JsonResponse::HTTP_CREATED ); }
43. Handling the command // CampaignCommandHandler.php public function handleCreateCampaignForTargetGroup( CreateCampaignForTargetGroup $command ) { $campaign = Campaign::createForTargetGroup( $command->getCampaignId(), $command->getCompanyId(), $command->getRecruiterId(), $command->getTargetGroup(), $this->recommender ); $this->campaignRepository->save($campaign); }
44. Handling the command // CampaignCommandHandler.php public function handleCreateCampaignForTargetGroup( CreateCampaignForTargetGroup $command ) { $campaign = Campaign::createForTargetGroup( $command->getCampaignId(), $command->getCompanyId(), $command->getRecruiterId(), $command->getTargetGroup(), $this->recommender ); $this->campaignRepository->save($campaign); }
45. Handling the command // CampaignCommandHandler.php public function handleCreateCampaignForTargetGroup( CreateCampaignForTargetGroup $command ) { $campaign = Campaign::createForTargetGroup( $command->getCampaignId(), $command->getCompanyId(), $command->getRecruiterId(), $command->getTargetGroup(), $this->recommender ); $this->campaignRepository->save($campaign); }
46. Action → Reaction // Campaign.php public static function createForTargetGroup(/* args */) { $campaign = new Campaign(); $campaignChannels = $recommender->recommend(/* args */); $campaign->apply(new CampaignCreatedForTargetGroup( $campaignId, $companyId, $accountId, Channels::fromCollection($campaignChannels), $targetGroup, $campaign->calculateCampaignPrice($campaignChannels), $campaign->calculateEstimatedOutcome($campaignChannels) )); return $campaign; }
47. Action → Reaction // Campaign.php public static function createForTargetGroup(/* args */) { $campaign = new Campaign(); $campaignChannels = $recommender->recommend(/* args */); $campaign->apply(new CampaignCreatedForTargetGroup( $campaignId, $companyId, $accountId, Channels::fromCollection($campaignChannels), $targetGroup, $campaign->calculateCampaignPrice($campaignChannels), $campaign->calculateEstimatedOutcome($campaignChannels) )); return $campaign; }
48. Action → Reaction // Campaign.php public static function createForTargetGroup(/* args */) { $campaign = new Campaign(); $campaignChannels = $recommender->recommend(/* args */); $campaign->apply(new CampaignCreatedForTargetGroup( $campaignId, $companyId, $accountId, Channels::fromCollection($campaignChannels), $targetGroup, $campaign->calculateCampaignPrice($campaignChannels), $campaign->calculateEstimatedOutcome($campaignChannels) )); return $campaign; }
49. Action → Reaction // Campaign.php public static function createForTargetGroup(/* args */) { $campaign = new Campaign(); $campaignChannels = $recommender->recommend(/* args */); $campaign->apply(new CampaignCreatedForTargetGroup( $campaignId, $companyId, $accountId, Channels::fromCollection($campaignChannels), $targetGroup, $campaign->calculateCampaignPrice($campaignChannels), $campaign->calculateEstimatedOutcome($campaignChannels) )); return $campaign; }
50. // Broadway: EventSourcedAggregateRoot.php public function apply($event) { $this->handleRecursively($event); $this->playhead++; $this->uncommittedEvents[] = DomainMessage::recordNow( $this->getAggregateRootId(), $this->playhead, new Metadata(), $event ); }
51. // Broadway: EventSourcedAggregateRoot.php public function apply($event) { $this->handleRecursively($event); $this->playhead++; $this->uncommittedEvents[] = DomainMessage::recordNow( $this->getAggregateRootId(), $this->playhead, new Metadata(), $event ); }
52. Campaign Event Store Create Campaign For Target Group Campaign Created For Target Group
53. // Campaign.php protected function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event ) { $this->id = $event->getCampaignId(); $this->channels = $event->getRecommendedChannels()->toArray(); $this->campaignPrice = $event->getCampaignPrice(); $this->estimatedOutcome = $event->getEstimatedOutcome(); $this->targetGroup = $event->getTargetGroup(); $this->setInitialRecommendation( $this->channels, $this->estimatedOutcome, $this->campaignPrice ); }
54. // Broadway: EventSourcedAggregateRoot.php public function apply($event) { $this->handleRecursively($event); $this->playhead++; $this->uncommittedEvents[] = DomainMessage::recordNow( $this->getAggregateRootId(), $this->playhead, new Metadata(), $event ); }
55. // Broadway: EventSourcedAggregateRoot.php public function apply($event) { $this->handleRecursively($event); $this->playhead++; $this->uncommittedEvents[] = DomainMessage::recordNow( $this->getAggregateRootId(), $this->playhead, new Metadata(), $event ); }
56. // Broadway: EventSourcedAggregateRoot.php public function apply($event) { $this->handleRecursively($event); $this->playhead++; $this->uncommittedEvents[] = DomainMessage::recordNow( $this->getAggregateRootId(), $this->playhead, new Metadata(), $event ); }
57. Handling the command // CampaignCommandHandler.php public function handleCreateCampaignForTargetGroup( CreateCampaignForTargetGroup $command ) { $campaign = Campaign::createForTargetGroup( $command->getCampaignId(), $command->getCompanyId(), $command->getRecruiterId(), $command->getTargetGroup(), $this->recommender, $this->portfolio, $this->aidaMetricsCalculator ); $this->campaignRepository->save($campaign); }
58. Saving the events // Broadway: EventSourcingRepository.php public function save(AggregateRoot $aggregate) { // maybe we can get generics one day.... ;) Assert::isInstanceOf($aggregate, $this->aggregateClass); $eventStream = $aggregate->getUncommittedEvents(); $this->eventStore->append( $aggregate->getAggregateRootId(), $eventStream ); $this->eventBus->publish($eventStream); }
59. Saving the events // Broadway: EventSourcingRepository.php public function save(AggregateRoot $aggregate) { // maybe we can get generics one day.... ;) Assert::isInstanceOf($aggregate, $this->aggregateClass); $eventStream = $aggregate->getUncommittedEvents(); $this->eventStore->append( $aggregate->getAggregateRootId(), $eventStream ); $this->eventBus->publish($eventStream); }
60. Saving the events // Broadway: EventSourcingRepository.php public function save(AggregateRoot $aggregate) { // maybe we can get generics one day.... ;) Assert::isInstanceOf($aggregate, $this->aggregateClass); $eventStream = $aggregate->getUncommittedEvents(); $this->eventStore->append( $aggregate->getAggregateRootId(), $eventStream ); $this->eventBus->publish($eventStream); }
61. It happened in the past
62. Append only
63. Campaign Event Store Create Campaign For Target Group Campaign Created For Target Group
64. How would we load a campaign?
65. Campaign Event Store Campaign Created For Target Group
66. // Broadway: EventSourcingRepository.php public function load($id) { try { $domainEventStream = $this->eventStore->load($id); return $this->aggregateFactory->create( $this->aggregateClass, $domainEventStream ); } catch (EventStreamNotFoundException $e) { throw AggregateNotFoundException::create($id, $e); } } // Broadway: EventSourcedAggregateRoot.php public function initializeState(DomainEventStreamInterface $stream) { foreach ($stream as $message) { $this->playhead++; $this->handleRecursively($message->getPayload()); } }
67. // Broadway: EventSourcingRepository.php public function load($id) { try { $domainEventStream = $this->eventStore->load($id); return $this->aggregateFactory->create( $this->aggregateClass, $domainEventStream ); } catch (EventStreamNotFoundException $e) { throw AggregateNotFoundException::create($id, $e); } } // Broadway: EventSourcedAggregateRoot.php public function initializeState(DomainEventStreamInterface $stream) { foreach ($stream as $message) { $this->playhead++; $this->handleRecursively($message->getPayload()); } }
68. // Broadway: EventSourcingRepository.php public function load($id) { try { $domainEventStream = $this->eventStore->load($id); return $this->aggregateFactory->create( $this->aggregateClass, $domainEventStream ); } catch (EventStreamNotFoundException $e) { throw AggregateNotFoundException::create($id, $e); } } // Broadway: EventSourcedAggregateRoot.php public function initializeState(DomainEventStreamInterface $stream) { foreach ($stream as $message) { $this->playhead++; $this->handleRecursively($message->getPayload()); } }
69. // Broadway: EventSourcingRepository.php public function load($id) { try { $domainEventStream = $this->eventStore->load($id); return $this->aggregateFactory->create( $this->aggregateClass, $domainEventStream ); } catch (EventStreamNotFoundException $e) { throw AggregateNotFoundException::create($id, $e); } } // Broadway: EventSourcedAggregateRoot.php public function initializeState(DomainEventStreamInterface $stream) { foreach ($stream as $message) { $this->playhead++; $this->handleRecursively($message->getPayload()); } }
70. Let's add a channel
71. Campaign Event Store Add Channel To Campaign Channel Added To Campaign
72. // CampaignCommandHandler.php public function handleAddChannelToCampaignCommand( AddChannelToCampaignCommand $command ) { $channel = $this->portfolio->getById($command->getChannelId()); $campaign = $this->campaignRepository->load( $command->getCampaignId() ); $campaign->addChannel($channel, $this->aidaMetricsCalculator); $this->campaignRepository->save($campaign); }
73. // CampaignCommandHandler.php public function handleAddChannelToCampaignCommand( AddChannelToCampaignCommand $command ) { $channel = $this->portfolio->getById($command->getChannelId()); $campaign = $this->campaignRepository->load( $command->getCampaignId() ); $campaign->addChannel($channel, $this->aidaMetricsCalculator); $this->campaignRepository->save($campaign); }
74. // CampaignCommandHandler.php public function handleAddChannelToCampaignCommand( AddChannelToCampaignCommand $command ) { $channel = $this->portfolio->getById($command->getChannelId()); $campaign = $this->campaignRepository->load( $command->getCampaignId() ); $campaign->addChannel($channel, $this->aidaMetricsCalculator); $this->campaignRepository->save($campaign); }
75. // CampaignCommandHandler.php public function handleAddChannelToCampaignCommand( AddChannelToCampaignCommand $command ) { $channel = $this->portfolio->getById($command->getChannelId()); $campaign = $this->campaignRepository->load( $command->getCampaignId() ); $campaign->addChannel($channel, $this->aidaMetricsCalculator); $this->campaignRepository->save($campaign); }
76. // Campaign.php public function addChannel( ChannelInPortfolio $channel, AidaMetricsCalculator $aidaMetricsCalculator ) { $this->guardCannotModifyOrderedCampaign(); $channelInCampaign = new ChannelInCampaign(/* $args */); $channels = array_merge($this->channels, [$channelInCampaign]); $campaignPrice = $this->calculateCampaignPrice($channels); $estimatedOutcome = $this->calculateEstimatedOutcome($channels); $this->apply(new ChannelAddedToCampaign( $this->id, $channelInCampaign, $campaignPrice, $estimatedOutcome )); }
77. // Campaign.php public function addChannel( ChannelInPortfolio $channel, AidaMetricsCalculator $aidaMetricsCalculator ) { $this->guardCannotModifyOrderedCampaign(); $channelInCampaign = new ChannelInCampaign(/* $args */); $channels = array_merge($this->channels, [$channelInCampaign]); $campaignPrice = $this->calculateCampaignPrice($channels); $estimatedOutcome = $this->calculateEstimatedOutcome($channels); $this->apply(new ChannelAddedToCampaign( $this->id, $channelInCampaign, $campaignPrice, $estimatedOutcome )); }
78. // Campaign.php public function addChannel( ChannelInPortfolio $channel, AidaMetricsCalculator $aidaMetricsCalculator ) { $this->guardCannotModifyOrderedCampaign(); $channelInCampaign = new ChannelInCampaign(/* $args */); $channels = array_merge($this->channels, [$channelInCampaign]); $campaignPrice = $this->calculateCampaignPrice($channels); $estimatedOutcome = $this->calculateEstimatedOutcome($channels); $this->apply(new ChannelAddedToCampaign( $this->id, $channelInCampaign, $campaignPrice, $estimatedOutcome )); }
79. // Campaign.php protected function applyChannelAddedToCampaign( ChannelAddedToCampaign $event ) { $this->channels[] = $event->getChannel(); $this->campaignPrice = $event->getCampaignPrice(); $this->estimatedOutcome = $event- >getEstimatedOutcomeOfCampaign(); $this->isCustomized = true; }
80. How do we get a listing of the campaigns?
81. We'll create a read model
82. Campaign Event Store Campaign Created For Target Group
83. Campaign Event Store Campaign Created For Target Group Read Model Store
84. Campaign Event Store Campaign Created For Target Group Read Model Store MySQL Read Model Store Redis
85. Saving the events // Broadway: EventSourcingRepository.php public function save(AggregateRoot $aggregate) { // maybe we can get generics one day.... ;) Assert::isInstanceOf($aggregate, $this->aggregateClass); $eventStream = $aggregate->getUncommittedEvents(); $this->eventStore->append( $aggregate->getAggregateRootId(), $eventStream ); $this->eventBus->publish($eventStream); }
86. Creating read models // CampaignOverviewProjector.php public function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event, DomainMessage $message ) { $this->repository->save(new CampaignOverview( $event->getCampaignId(), $event->getCompanyId(), $event->getTargetGroup()->getJobTitle(), $message->getRecordedOn() )); }
87. Creating read models // CampaignOverviewProjector.php public function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event, DomainMessage $message ) { $this->repository->save(new CampaignOverview( $event->getCampaignId(), $event->getCompanyId(), $event->getTargetGroup()->getJobTitle(), $message->getRecordedOn() )); }
88. Creating read models // CampaignOverviewProjector.php public function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event, DomainMessage $message ) { $this->repository->save(new CampaignOverview( $event->getCampaignId(), $event->getCompanyId(), $event->getTargetGroup()->getJobTitle(), $message->getRecordedOn() )); }
89. Creating read models // CampaignDetailsProjector.php public function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event ) { $campaignDetails = new CampaignDetails( $event->getCampaignId(), $event->getRecruiterId(), $event->getTargetGroup(), $event->getCompanyId(), $event->getRecommendedChannels()->toArray() ); $campaignDetails->updatePrice($event->getCampaignPrice()); $campaignDetails->updateEstimatedOutcome( $event->getEstimatedOutcome() ); $this->repository->save($campaignDetails); }
90. Creating read models // CampaignDetailsProjector.php public function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event ) { $campaignDetails = new CampaignDetails( $event->getCampaignId(), $event->getRecruiterId(), $event->getTargetGroup(), $event->getCompanyId(), $event->getRecommendedChannels()->toArray() ); $campaignDetails->updatePrice($event->getCampaignPrice()); $campaignDetails->updateEstimatedOutcome( $event->getEstimatedOutcome() ); $this->repository->save($campaignDetails); }
91. Creating read models // CampaignDetailsProjector.php public function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event ) { $campaignDetails = new CampaignDetails( $event->getCampaignId(), $event->getRecruiterId(), $event->getTargetGroup(), $event->getCompanyId(), $event->getRecommendedChannels()->toArray() ); $campaignDetails->updatePrice($event->getCampaignPrice()); $campaignDetails->updateEstimatedOutcome( $event->getEstimatedOutcome() ); $this->repository->save($campaignDetails); }
92. Using the history to your advantage
93. Discard Customizations
94. // Campaign.php protected function applyCampaignCreatedForTargetGroup( CampaignCreatedForTargetGroup $event ) { $this->id = $event->getCampaignId(); $this->channels = $event->getRecommendedChannels()->toArray(); $this->campaignPrice = $event->getCampaignPrice(); $this->estimatedOutcome = $event->getEstimatedOutcome(); $this->targetGroup = $event->getTargetGroup(); $this->setInitialRecommendation( $this->channels, $this->estimatedOutcome, $this->campaignPrice ); }
95. // Campaign.php public function discardCustomization() { $this->guardCannotModifyOrderedCampaign(); if ($this->isCustomized === false) { return; } $this->apply(new CustomizationDiscarded( $this->id, $this->initialRecommendation['channels'], $this->initialRecommendation['campaignPrice'], $this->initialRecommendation['estimatedOutcome'] )); }
96. // Campaign.php protected function applyCustomizationDiscarded( CustomizationDiscarded $event ) { $this->channels = $event->getRecommendedChannels(); $this->campaignPrice = $event->getCampaignPrice(); $this->estimatedOutcome = $event->getEstimatedOutcome(); $this->isCustomized = false; }
97. Where are the tests?
98. Scenario testing
99. Given When Then
100. Campaign Created For Target Group Add Channel To Campaign Channel Added To Campaign
101. $channelId = new ChannelId('42'); $this->scenario ->given([ new CampaignCreatedForTargetGroup( $this->campaignId, // $args ) ]) ->when(new AddChannelToCampaignCommand( $this->campaignId, $channelId )) ->then([ new ChannelAddedToCampaign( $this->campaignId, $this->stubbedChannel($channelId), new CampaignPrice(Money::EUR(40000), Money::EUR(0)), new EstimatedOutcome(/* snip */) ) ]);
102. $channelId = new ChannelId('42'); $this->scenario ->given([ new CampaignCreatedForTargetGroup( $this->campaignId, // $args ) ]) ->when(new AddChannelToCampaignCommand( $this->campaignId, $channelId )) ->then([ new ChannelAddedToCampaign( $this->campaignId, $this->stubbedChannel($channelId), new CampaignPrice(Money::EUR(40000), Money::EUR(0)), new EstimatedOutcome(/* snip */) ) ]);
103. $channelId = new ChannelId('42'); $this->scenario ->given([ new CampaignCreatedForTargetGroup( $this->campaignId, // $args ) ]) ->when(new AddChannelToCampaignCommand( $this->campaignId, $channelId )) ->then([ new ChannelAddedToCampaign( $this->campaignId, $this->stubbedChannel($channelId), new CampaignPrice(Money::EUR(40000), Money::EUR(0)), new EstimatedOutcome(/* snip */) ) ]);
104. $channelId = new ChannelId('42'); $this->scenario ->given([ new CampaignCreatedForTargetGroup( $this->campaignId, // $args ) ]) ->when(new AddChannelToCampaignCommand( $this->campaignId, $channelId )) ->then([ new ChannelAddedToCampaign( $this->campaignId, $this->stubbedChannel($channelId), new CampaignPrice(Money::EUR(40000), Money::EUR(0)), new EstimatedOutcome(/* snip */) ) ]);
105. Alternative scenario test $channelId = new ChannelId('42'); $this->scenario ->given([ new CampaignCreatedForTargetGroup( $this->campaignId, // $args ) ]) ->when(function (Campaign $campaign) { $campaign->addChannel($channelId) }) ->then([ new ChannelAddedToCampaign( $this->campaignId, $this->stubbedChannel($channelId), new CampaignPrice(Money::EUR(40000), Money::EUR(0)), new EstimatedOutcome(/* snip */) ) ]);
106. You can do something similar to test your read models
107. Using the history to correct mistakes
108. Replaying events to fix read models
109. Replaying events to create new read models
110. Very powerful
111. Very tricky
112. Replaying events Event Store Read Model Store e v e n t b u s Read Model Store
113. Replaying events Event Store Read Model Store e v e n t b u s Email Notification Processor
114. Find a solution for your specific problem
115. Questions? joind.in/16227 freenode: #qandidate github.com/qandidate-labs/broadway
116. Symfony User Group The Renaissance 10 december http://www.meetup.com/Symfony-User-Group-NL