Slide 1

Slide 1 text

Event Sourcing #PHPTour 2015 @MattKetmo

Slide 2

Slide 2 text

CQRS Event Sourcing Reactive DDD }Patterns to build & structure applications Not new buzzwords, exist for many years How can it help to scale horizontally? Today’s focus

Slide 3

Slide 3 text

Matthieu Moquet @MattKetmo @ BlaBlaCar for ~4 years

Slide 4

Slide 4 text

{Disclaimer Oriented for long lived apps.
 Overkill for RAD / CRUD / TODO apps. Use those patterns in moderation.
 There is no silver bullet. This talk aims to open your mind.
 Think different!

Slide 5

Slide 5 text

I/O

Slide 6

Slide 6 text

90% ViewModels / Flexible Eventually consistent 10% Validation / Business rules Coherence (ACID) Read Write Fast & Scale

Slide 7

Slide 7 text

Use case bank accounts Account #1 ! 1337€ Account #2 ! 42€ Account #3 ! 10€ + Transfer money

Slide 8

Slide 8 text

Account ! id user_id amount User ! id name ...

Slide 9

Slide 9 text

Facade (Controller) Models (ORM) DB

Slide 10

Slide 10 text

//  Open  new  account   $account  =  new  Account();   $account-­‐>setAmount(0,  'EUR');   $em-­‐>persist($account);   $em-­‐>flush();   ! //  Credit  100  EUR   $account-­‐>setAmount(100,  'EUR');   $em-­‐>flush();   ! //  Debit  20  EUR   $account-­‐>setAmount(80,  'EUR');   $em-­‐>flush();

Slide 11

Slide 11 text

is not the solution When user process an operation on his account his doesn’t directly update his total amount of money We need a service that translate our business needs CRUD

Slide 12

Slide 12 text

Facade (Controller) Models (ORM) DB Service Layer (Business)

Slide 13

Slide 13 text

interface  BankAccountService     {          /**  @return  int  */          public  function  open();   !        /**  @return  Account  */          public  function  get($accountId);   !        /**  @return  Account[]  */          public  function  findAll($userId);   !        /**  @return  void  */          public  function  credit($accountId,  $balance);   !        /**  @return  void  */          public  function  debit($accountId,  $balance);   }  

Slide 14

Slide 14 text

CQS Command Query Separation }Each operation is either a Command or a Query, not both

Slide 15

Slide 15 text

Commands Queries Change the state of the system. Do not return anything. Does not change the state of the system. Return data. Write only Read only

Slide 16

Slide 16 text

Commands /**  @return  Account  */   public  function  get($accountId);   ! /**  @return  Account[]  */   public  function  findAll($userId); /**  @return  int  */   public  function  open();   ! /**  @return  void  */   public  function  credit($accountId,  $balance);   ! /**  @return  void  */   public  function  debit($accountId,  $balance); Should be void Queries

Slide 17

Slide 17 text

/**      *  @param  Uuid  $accountId    *    *  @return  void      */   public  function  open(Uuid  $accountId);   Let the client generate the identifier, not the infrastructure (database)

Slide 18

Slide 18 text

CQRS Command Query Responsibility Segregation }Like CQS but separate queries and commands in different classes

Slide 19

Slide 19 text

Command Facade Query Facade Command Service Query Service Models (ORM) DB

Slide 20

Slide 20 text

CQRS challenges the assumption that reading and writing are sharing the same abstractions Databases Models Apps

Slide 21

Slide 21 text

Command Facade Query Facade Command Handler Query Repository Command Models DB Read Models Command DTO Segregation of read and write is a radical form of decoupling

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

The interesting part about CQRS is not the pattern itself. But it allows to challenge established assumptions and opens new architectural options.

Slide 24

Slide 24 text

How to Scale

Slide 25

Slide 25 text

Now that the workflow of Read & Write has been separated. We can optimize each side separately.

Slide 26

Slide 26 text

asynchronously Since Command Handlers do NOT return anything, 
 then do the execution

Slide 27

Slide 27 text

Command Facade Command Handlers Commands Queueing Ack

Slide 28

Slide 28 text

! $accoundId  =  AccountId::generate();   $command  =  new  OpenAccountCommand($accountId);   ! if  (/*  is  command  valid  */)  {          $commandBus-­‐>handle($command);   !        return  new  Response(                  null,                  $async  ?  202  :  201,  //  Accepted  :  Created                  ['Location'  =>  "/accounts/$accountId"]          );     }

Slide 29

Slide 29 text

$command  =  new  DebitCommand(          $accountId,            new  Money(20,  'EUR')   );   ! if  (/*  account  has  enough  money  */)  {          $commandBus-­‐>handle($command);   !        return  new  Response(null,  204);   }

Slide 30

Slide 30 text

Command tracking Scale workers Horizontally by instantiating more workers (subscribers) Generate an identifier per command to let the client tracks its status Rabbitmq, Kafka, Gearman, ……… make your choice on queues.io

Slide 31

Slide 31 text

Projections On the Read side you should prepare you data as it will be requested

Slide 32

Slide 32 text

Having only 1 data store for reads & writes does not scale (well) different usage different needs

Slide 33

Slide 33 text

Denormalize your data UI View models Statistic models Search index models API Read models ... Read models are faster than JOIN

Slide 34

Slide 34 text

• search & aggregations capabilities • horizontal scaling architecture (sharding, replication) • denormalization • fast Secondary data store to read your data

Slide 35

Slide 35 text

{      "user_id":  133742      "from":  "Paris",      "to":  "Luxembourg",      "by":  [          "Reims",            "..."      ],      "date":  "2015-­‐05-­‐11",   } Input (mysql)

Slide 36

Slide 36 text

[{      "trip":  {          "from":  "Paris",          "to":  "Luxembourg",          "date":  "2015-­‐05-­‐11",          "..."      },      "user":  {          "name":  "John  D",          "age":  29,          "grade":  "beginner",          "..."      }   }] Output (Elasticsearch) denormalized searchable

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

SELECT  tweets.*,  users.*      FROM  tweets   ! JOIN  users      ON  users.id  =  tweet.sender_id   ! JOIN  follows      ON  follows.followee_id  =  user.id   ! WHERE  follows.follower_id  =  $userId   ! ORDER  BY  tweets.time  DESC   LIMIT  100

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Tweets stream DB Aggs Timelines {      "user_id":  1234567890      "status":  "Hello  World"      "timestamp":  1430491773   } ! [{      "tweet_id":  1234567890876543,      "username":  "MattKetmo",      "name":  "Matthieu  Moquet",      "timestamp":  1430491773,      "status":  "Hello  World",   },  {      "tweet_id":  1234567890886445,      ...   }]  

Slide 41

Slide 41 text

SELECT  tweets      FROM  timelines    WHERE  timeline_id  =  $userId    LIMIT  500

Slide 42

Slide 42 text

PageViewEvents DB Aggs Increment counters /month Google Analytics /day /hour total {      "eventType":  "PageViewEvent"      "timestamp":  1430491773,      "ipAddress":  "12.34.56.78",      "sessionId":  "abcd1234567890",      "pageUrl":  "/hello-­‐world",      "..."   }

Slide 43

Slide 43 text

Using Cassandra you need to: ‣ know the read requests before creating your data models ‣ create as many tables (ie. KeySpaces) than you have views ‣ denormalize the data (no join allowed) ’s keypoints

Slide 44

Slide 44 text

C* C* C* Primary Data C* Read Data User post message Endpoint

Slide 45

Slide 45 text

Multi Data Center Replication + Sharding + Cache

Slide 46

Slide 46 text

Command Command Handler Primary DB Projections View Models Message Bus Write Read Eventual Consistency {...} {...} {...} {...}

Slide 47

Slide 47 text

Event Sourcing change the way we store our primary data

Slide 48

Slide 48 text

If we know the events of the past, we can reconstitute the present. Event Sourcing

Slide 49

Slide 49 text

Event Sourcing Register only a series of events. Reconstitute the state of current "entity" by reading the past events.

Slide 50

Slide 50 text

the standard way Something happen State A State B Something happen (delta) (dropped) (stored) (delta) Time

Slide 51

Slide 51 text

the event sourcing way Something happen State A State B Something happen (stored) (reconstituted) (reconstituted) (stored) Time

Slide 52

Slide 52 text

[{        "uuid":  "110e8400-­‐e29b-­‐11d4-­‐a716-­‐446655440000",        "type":  "AccountWasOpen",        "recorded_on":  "2015-­‐05-­‐11T13:37:00Z",        "payload":  {}   },  {        "uuid":  "110e8400-­‐e29b-­‐11d4-­‐a716-­‐446655440000",        "type":  "AccountWasCredited",        "recorded_on":  "2015-­‐05-­‐11T14:42:00Z",        "payload":  {  "amount":  100,  "currency":  "EUR"  }   },  {      "..."   }] Event Store

Slide 53

Slide 53 text

Aggregates Domain models which serve the business logic using the ubiquitous language. Work as an isolated object (or graph of objects) which doesn’t reference any others. Can ensure only its own integrity.

Slide 54

Slide 54 text

//  Open  a  new  account   $accountId  =  AccountId::generate();   $account  =  BankAccount::open($accountId);   ! //  Add  some  money   $account-­‐>credit(new  Money(100,  'EUR'));

Slide 55

Slide 55 text

Records events Aggregates class  BankAccount  implements  AggregateRootInterface   {          //  ...   }

Slide 56

Slide 56 text

interface  AggregateRoot   {          /**            *  @return  DomainEventStream            */          public  function  getUncommittedEvents();   !        /**            *  @return  string            */          public  function  getAggregateRootId();   }

Slide 57

Slide 57 text

$accountId  =  AccountId::generate();   $account  =  BankAccount::open($accountId);   $account-­‐>credit(new  Money(100,  'EUR'));   ! //  Retrieve  event  stream   $events  =  $account-­‐>getUncommittedEvents();   //  -­‐  AccountWasOpen   //  -­‐  AccountWasCredited

Slide 58

Slide 58 text

class  BankAccount  extends  EventSourcedAggregateRoot   {          private  $accountId;   !        public  function  getAggregateRootId()          {                  return  $this-­‐>accountId;          }   !        public  static  function  open(AccountId  $accountId)          {                  $account  =  new  self();                  $account-­‐>apply(new  AccountWasOpen($accountId));   !                return  $account;          }   !        public  function  credit(Money  $balance)          {                  $this-­‐>apply(new  AccountWasCredited(                          $this-­‐>accountId,                          $balance-­‐>getAmount(),                          $balance-­‐>getCurrency()                  );          }   }

Slide 59

Slide 59 text

Reconstitute from events Aggregates

Slide 60

Slide 60 text

class  BankAccount  extends  EventSourcedAggregateRoot   {          //  ...              protected  function  applyAccountWasOpen(AccountWasOpen  $event)          {                  $this-­‐>accountId  =  $event-­‐>getAccountId();          }   !        protected  function  applyAccountWasCredited(AccountWasCredited  $event)          {                  $this-­‐>amount  +=  $event-­‐>getBalance();          }   !        protected  function  applyAccountWasDebited(AccountWasDebited  $event)          {                  $this-­‐>amount  -­‐=  $event-­‐>getBalance();          }   }

Slide 61

Slide 61 text

Aggregates Protect invariants

Slide 62

Slide 62 text

class  BankAccount  extends  EventSourcedAggregateRoot   {          //  ...   !        public  function  debit(Money  $balance)          {                  if  ($this-­‐>amount  <  $balance-­‐>getAmount())  {                          throw  new  NotEnoughMoneyException(                                  'Cannot  debit  more  than  current  amount'                          );                  }   !                return  $this-­‐>apply(new  AccountWasCredited(                          $this-­‐>accountId,                          $balance-­‐>getAmount(),                          $balance-­‐>getCurrency()                  ));                        }   }

Slide 63

Slide 63 text

Command + Event Sourcing

Slide 64

Slide 64 text

Command User intent Events Event Store Apply changes on system Primary DB Transactional append

Slide 65

Slide 65 text

class  BankAccountCommandHandler  extends  CommandHandler   {          public  function  handleOpenAccount(OpenAccount  $command)          {                  $account  =  BankAccount::open($command-­‐>getAccountId());   !                $this-­‐>repository-­‐>save($account);          }   !        public  function  handleCreditAccount(CreditAccount  $command)          {                  $account  =  $this-­‐>repository-­‐>load($command-­‐>getAccountId());                  $account-­‐>credit($command-­‐>getBalance());   !                $this-­‐>repository-­‐>save($account);          }   }

Slide 66

Slide 66 text

class  EventSourcingRepository  implements  RepositoryInterface   {          public  function  load($id)          {                  try  {                          $events  =  $this-­‐>eventStore-­‐>load($id);   !                        return  $this-­‐>aggregateFactory-­‐>create($events);                  }  catch  (EventStreamNotFoundException  $e)  {                          throw  AggregateNotFoundException::create($id,  $e);                  }          }   !        public  function  save(AggregateRoot  $aggregate)          {                  $events  =  $aggregate-­‐>getUncommittedEvents();   !                $this-­‐>eventStore-­‐>append(                          $aggregate-­‐>getAggregateRootId(),                            $events                  );          }   }

Slide 67

Slide 67 text

Event Sourcing PROS

Slide 68

Slide 68 text

Append Only Very fast data update Deleting data = new events

Slide 69

Slide 69 text

Immutable Infinite caching

Slide 70

Slide 70 text

Sharding Aggregates are self-reliant and don’t have external references No Joins

Slide 71

Slide 71 text

Complete historical data Being able to replay history is a major benefit both technically and for the business BI team will love it

Slide 72

Slide 72 text

No Database migration when updating the data DBA will love it

Slide 73

Slide 73 text

Event Sourcing CONS Well, not really

Slide 74

Slide 74 text

Ensure consistency How to avoid concurrent updates?

Slide 75

Slide 75 text

Load aggregate from EventStore: 150€ Create AccountWasDebited event Append event in datastore Example Two debits of 100€ must not be accepted if only 150€ left Load aggregate from EventStore: 150€ Create AccountWasDebited event Append event in datastore Event must NOT be appended twice

Slide 76

Slide 76 text

abstract  class  EventSourcedAggregateRoot  implements  AggregateRootInterface   {          private  $uncommittedEvents  =  array();   !        private  $playhead  =  -­‐1;            /**          *  Applies  an  event.            *  The  event  is  added  to  the  list  of  uncommited  events.          */          public  function  apply($event)          {                  $this-­‐>playhead++;   !                $this-­‐>uncommittedEvents[]  =  DomainMessage::recordNow(                          $this-­‐>getAggregateRootId(),                          $this-­‐>playhead,                          new  Metadata(array()),                          $event                  );          }   }

Slide 77

Slide 77 text

DBAL EventStore (MySQL) CREATE  TABLE  EventStore  (   !    `id`                    INT(11)  AUTO_INCREMENT,      `uuid`                BINARY(16),      `playhead`        INT(11),      `type`                TEXT,      `payload`          TEXT,      `metadata`        TEXT,      `recorded_on`  DATETIME,   !    PRIMARY  KEY  (`id`),   !    UNIQUE  KEY  `UNIQ_PLAYHEAD`  (`uuid`,  `playhead`)   );

Slide 78

Slide 78 text

E_TOO_MANY_EVENTS When your Aggregates are "long lived" it may be slow to read the full history eg. BankAccount

Slide 79

Slide 79 text

Snapshotting is a solution

Slide 80

Slide 80 text

Snapshotting Store the state of the aggregate and read only the events created after

Slide 81

Slide 81 text

Snapshotting Snapshot Snapshot Events States

Slide 82

Slide 82 text

TL;DR

Slide 83

Slide 83 text

Command Handler Events Event Store Command Auditing Read Model Event Listener Event Bus Projector Write Read Business rules Historical data Read / search index Side effects User intent View models

Slide 84

Slide 84 text

Frameworks C# / Java nCQRS Fohjin NEventStore LiteCQRS Lokad.CQRS Agr.CQRS Axon Framework JDON PHP Broadway Predaddy EventCentric.Core litecqrs-php ...?

Slide 85

Slide 85 text

qandidate-labs/broadway qandidate-labs/broadway-demo

Slide 86

Slide 86 text

GetEventStore.com Event Sourcing oriented database

Slide 87

Slide 87 text

cqrs.nu #MustRead

Slide 88

Slide 88 text

dddinphp.org Google Groups

Slide 89

Slide 89 text

Thank You! Slides available at moquet.net/talks/phptour-2015 Leave feedbacks @MattKetmo