Event Sourcing

PHPTour 2015

Transcript

1. Event Sourcing
   #PHPTour 2015
   @MattKetmo
   

   View Slide

2. CQRS
   Event Sourcing
   Reactive
   DDD
   }Patterns to build &
   structure applications
   Not new buzzwords,
   exist for many years
   How can it help to
   scale horizontally?
   Today’s focus
   

   View Slide

3. Matthieu Moquet
   @MattKetmo
   @ BlaBlaCar
   for ~4 years
   

   View Slide

4. {Disclaimer
   Oriented for long lived apps.

   Overkill for RAD / CRUD / TODO apps.
   Use those patterns in moderation.

   There is no silver bullet.
   This talk aims to open your mind.

   Think different!
   

   View Slide

5. I/O
   

   View Slide

6. 90%
   ViewModels / Flexible
   Eventually consistent
   10%
   Validation / Business rules
   Coherence (ACID)
   Read Write
   Fast & Scale
   

   View Slide

7. Use case
   bank accounts
   Account #1
   !
   1337€
   Account #2
   !
   42€
   Account #3
   !
   10€
   + Transfer money
   

   View Slide

8. Account
   !
   id
   user_id
   amount
   User
   !
   id
   name
   ...
   

   View Slide

9. Facade
   (Controller)
   Models
   (ORM)
   DB
   

   View Slide

10. //  Open  new  account  
    $account  =  new  Account();  
    $account-­‐>setAmount(0,  'EUR');  
    $em-­‐>persist($account);  
    $em-­‐>flush();  
    !
    //  Credit  100  EUR  
    $account-­‐>setAmount(100,  'EUR');  
    $em-­‐>flush();  
    !
    //  Debit  20  EUR  
    $account-­‐>setAmount(80,  'EUR');  
    $em-­‐>flush();
    

    View Slide

11. is not the solution
    When user process an operation on his account his
    doesn’t directly update his total amount of money
    We need a service that translate our business needs
    CRUD
    

    View Slide

12. Facade
    (Controller)
    Models
    (ORM)
    DB
    Service
    Layer
    (Business)
    

    View Slide

13. interface  BankAccountService    
    {  
           /**  @return  int  */  
           public  function  open();  
    !
           /**  @return  Account  */  
           public  function  get($accountId);  
    !
           /**  @return  Account[]  */  
           public  function  findAll($userId);  
    !
           /**  @return  void  */  
           public  function  credit($accountId,  $balance);  
    !
           /**  @return  void  */  
           public  function  debit($accountId,  $balance);  
    }  
    

    View Slide

14. CQS
    Command
    Query
    Separation
    }Each operation is
    either a Command or
    a Query, not both
    

    View Slide

15. Commands
    Queries
    Change the state of the system.
    Do not return anything.
    Does not change the state of the system.
    Return data.
    Write only
    Read only
    

    View Slide

16. Commands
    /**  @return  Account  */  
    public  function  get($accountId);  
    !
    /**  @return  Account[]  */  
    public  function  findAll($userId);
    /**  @return  int  */  
    public  function  open();  
    !
    /**  @return  void  */  
    public  function  credit($accountId,  $balance);  
    !
    /**  @return  void  */  
    public  function  debit($accountId,  $balance);
    Should be void
    Queries
    

    View Slide

17. /**    
     *  @param  Uuid  $accountId  
     *  
     *  @return  void    
     */  
    public  function  open(Uuid  $accountId);  
    Let the client generate the identifier,
    not the infrastructure (database)
    

    View Slide

18. CQRS
    Command
    Query
    Responsibility
    Segregation
    }Like CQS but
    separate queries and
    commands in
    different classes
    

    View Slide

19. Command Facade
    Query Facade
    Command
    Service
    Query
    Service
    Models
    (ORM)
    DB
    

    View Slide

20. CQRS challenges the assumption that
    reading and writing are sharing the
    same abstractions
    Databases
    Models
    Apps
    

    View Slide

21. Command Facade
    Query Facade
    Command
    Handler
    Query
    Repository
    Command
    Models
    DB
    Read
    Models
    Command
    DTO
    Segregation of read and write is a
    radical form of decoupling
    

    View Slide

22. View Slide

23. The interesting part about CQRS is not
    the pattern itself.
    But it allows to challenge established
    assumptions and opens new architectural
    options.
    

    View Slide

24. How to
    Scale
    

    View Slide

25. Now that the workflow of Read & Write
    has been separated.
    We can optimize each side separately.
    

    View Slide

26. asynchronously
    Since Command Handlers
    do NOT return anything, 

    then do the execution
    

    View Slide

27. Command
    Facade
    Command
    Handlers
    Commands Queueing
    Ack
    

    View Slide

28. !
    $accoundId  =  AccountId::generate();  
    $command  =  new  OpenAccountCommand($accountId);  
    !
    if  (/*  is  command  valid  */)  {  
           $commandBus-­‐>handle($command);  
    !
           return  new  Response(  
                   null,  
                   $async  ?  202  :  201,  //  Accepted  :  Created  
                   ['Location'  =>  "/accounts/$accountId"]  
           );    
    }
    

    View Slide

29. $command  =  new  DebitCommand(  
           $accountId,    
           new  Money(20,  'EUR')  
    );  
    !
    if  (/*  account  has  enough  money  */)  {  
           $commandBus-­‐>handle($command);  
    !
           return  new  Response(null,  204);  
    }
    

    View Slide

30. Command tracking
    Scale workers
    Horizontally by instantiating more workers (subscribers)
    Generate an identifier per command to let the client
    tracks its status
    Rabbitmq, Kafka, Gearman, ………
    make your choice on queues.io
    

    View Slide

31. Projections
    On the Read side you
    should prepare you data
    as it will be requested
    

    View Slide

32. Having only 1 data store
    for reads & writes does not
    scale (well)
    different usage
    different needs
    

    View Slide

33. Denormalize your data
    UI View models
    Statistic models
    Search index models
    API Read models
    ...
    Read models are
    faster than JOIN
    

    View Slide

34. • search & aggregations capabilities
    • horizontal scaling architecture (sharding, replication)
    • denormalization
    • fast
    Secondary data store to read your data
    

    View Slide

35. {  
       "user_id":  133742  
       "from":  "Paris",  
       "to":  "Luxembourg",  
       "by":  [  
           "Reims",    
           "..."  
       ],  
       "date":  "2015-­‐05-­‐11",  
    }
    Input (mysql)
    

    View Slide

36. [{  
       "trip":  {  
           "from":  "Paris",  
           "to":  "Luxembourg",  
           "date":  "2015-­‐05-­‐11",  
           "..."  
       },  
       "user":  {  
           "name":  "John  D",  
           "age":  29,  
           "grade":  "beginner",  
           "..."  
       }  
    }]
    Output (Elasticsearch)
    denormalized
    searchable
    

    View Slide

37. View Slide

38. SELECT  tweets.*,  users.*  
       FROM  tweets  
    !
    JOIN  users  
       ON  users.id  =  tweet.sender_id  
    !
    JOIN  follows  
       ON  follows.followee_id  =  user.id  
    !
    WHERE  follows.follower_id  =  $userId  
    !
    ORDER  BY  tweets.time  DESC  
    LIMIT  100
    

    View Slide

39. View Slide

40. Tweets stream
    DB
    Aggs
    Timelines
    {  
       "user_id":  1234567890  
       "status":  "Hello  World"  
       "timestamp":  1430491773  
    }
    !
    [{  
       "tweet_id":  1234567890876543,  
       "username":  "MattKetmo",  
       "name":  "Matthieu  Moquet",  
       "timestamp":  1430491773,  
       "status":  "Hello  World",  
    },  {  
       "tweet_id":  1234567890886445,  
       ...  
    }]  
    

    View Slide

41. SELECT  tweets  
       FROM  timelines  
     WHERE  timeline_id  =  $userId  
     LIMIT  500
    

    View Slide

42. PageViewEvents
    DB
    Aggs
    Increment
    counters
    /month
    Google Analytics
    /day /hour total
    {  
       "eventType":  "PageViewEvent"  
       "timestamp":  1430491773,  
       "ipAddress":  "12.34.56.78",  
       "sessionId":  "abcd1234567890",  
       "pageUrl":  "/hello-­‐world",  
       "..."  
    }
    

    View Slide

43. Using Cassandra you need to:
    ‣ know the read requests before creating your data models
    ‣ create as many tables (ie. KeySpaces) than you have views
    ‣ denormalize the data (no join allowed)
    ’s keypoints
    

    View Slide

44. C* C* C*
    Primary
    Data
    C*
    Read Data
    User post message
    Endpoint
    

    View Slide

45. Multi Data Center
    Replication + Sharding + Cache
    

    View Slide

46. Command
    Command
    Handler
    Primary
    DB
    Projections
    View
    Models
    Message Bus
    Write
    Read
    Eventual Consistency
    {...}
    {...}
    {...}
    {...}
    

    View Slide

47. Event Sourcing
    change the way we store our primary data
    

    View Slide

48. If we know the events of the past,
    we can reconstitute the present.
    Event Sourcing
    

    View Slide

49. Event Sourcing
    Register only a series of events.
    Reconstitute the state of current "entity"
    by reading the past events.
    

    View Slide

50. the standard way
    Something
    happen State A State B
    Something
    happen
    (delta) (dropped) (stored)
    (delta)
    Time
    

    View Slide

51. the event sourcing way
    Something
    happen State A State B
    Something
    happen
    (stored) (reconstituted) (reconstituted)
    (stored)
    Time
    

    View Slide

52. [{  
         "uuid":  "110e8400-­‐e29b-­‐11d4-­‐a716-­‐446655440000",  
         "type":  "AccountWasOpen",  
         "recorded_on":  "2015-­‐05-­‐11T13:37:00Z",  
         "payload":  {}  
    },  {  
         "uuid":  "110e8400-­‐e29b-­‐11d4-­‐a716-­‐446655440000",  
         "type":  "AccountWasCredited",  
         "recorded_on":  "2015-­‐05-­‐11T14:42:00Z",  
         "payload":  {  "amount":  100,  "currency":  "EUR"  }  
    },  {  
       "..."  
    }]
    Event Store
    

    View Slide

53. Aggregates
    Domain models which serve the business logic
    using the ubiquitous language.
    Work as an isolated object (or graph of objects)
    which doesn’t reference any others.
    Can ensure only its own integrity.
    

    View Slide

54. //  Open  a  new  account  
    $accountId  =  AccountId::generate();  
    $account  =  BankAccount::open($accountId);  
    !
    //  Add  some  money  
    $account-­‐>credit(new  Money(100,  'EUR'));
    

    View Slide

55. Records events
    Aggregates
    class  BankAccount  implements  AggregateRootInterface  
    {  
           //  ...  
    }
    

    View Slide

56. interface  AggregateRoot  
    {  
           /**  
             *  @return  DomainEventStream  
             */  
           public  function  getUncommittedEvents();  
    !
           /**  
             *  @return  string  
             */  
           public  function  getAggregateRootId();  
    }
    

    View Slide

57. $accountId  =  AccountId::generate();  
    $account  =  BankAccount::open($accountId);  
    $account-­‐>credit(new  Money(100,  'EUR'));  
    !
    //  Retrieve  event  stream  
    $events  =  $account-­‐>getUncommittedEvents();  
    //  -­‐  AccountWasOpen  
    //  -­‐  AccountWasCredited
    

    View Slide

58. class  BankAccount  extends  EventSourcedAggregateRoot  
    {  
           private  $accountId;  
    !
           public  function  getAggregateRootId()  
           {  
                   return  $this-­‐>accountId;  
           }  
    !
           public  static  function  open(AccountId  $accountId)  
           {  
                   $account  =  new  self();  
                   $account-­‐>apply(new  AccountWasOpen($accountId));  
    !
                   return  $account;  
           }  
    !
           public  function  credit(Money  $balance)  
           {  
                   $this-­‐>apply(new  AccountWasCredited(  
                           $this-­‐>accountId,  
                           $balance-­‐>getAmount(),  
                           $balance-­‐>getCurrency()  
                   );  
           }  
    }
    

    View Slide

59. Reconstitute from events
    Aggregates
    

    View Slide

60. class  BankAccount  extends  EventSourcedAggregateRoot  
    {  
           //  ...  
       
           protected  function  applyAccountWasOpen(AccountWasOpen  $event)  
           {  
                   $this-­‐>accountId  =  $event-­‐>getAccountId();  
           }  
    !
           protected  function  applyAccountWasCredited(AccountWasCredited  $event)  
           {  
                   $this-­‐>amount  +=  $event-­‐>getBalance();  
           }  
    !
           protected  function  applyAccountWasDebited(AccountWasDebited  $event)  
           {  
                   $this-­‐>amount  -­‐=  $event-­‐>getBalance();  
           }  
    }
    

    View Slide

61. Aggregates
    Protect invariants
    

    View Slide

62. class  BankAccount  extends  EventSourcedAggregateRoot  
    {  
           //  ...  
    !
           public  function  debit(Money  $balance)  
           {  
                   if  ($this-­‐>amount  <  $balance-­‐>getAmount())  {  
                           throw  new  NotEnoughMoneyException(  
                                   'Cannot  debit  more  than  current  amount'  
                           );  
                   }  
    !
                   return  $this-­‐>apply(new  AccountWasCredited(  
                           $this-­‐>accountId,  
                           $balance-­‐>getAmount(),  
                           $balance-­‐>getCurrency()  
                   ));                
           }  
    }
    

    View Slide

63. Command + Event Sourcing
    

    View Slide

64. Command
    User intent
    Events Event Store
    Apply changes
    on system
    Primary DB
    Transactional append
    

    View Slide

65. class  BankAccountCommandHandler  extends  CommandHandler  
    {  
           public  function  handleOpenAccount(OpenAccount  $command)  
           {  
                   $account  =  BankAccount::open($command-­‐>getAccountId());  
    !
                   $this-­‐>repository-­‐>save($account);  
           }  
    !
           public  function  handleCreditAccount(CreditAccount  $command)  
           {  
                   $account  =  $this-­‐>repository-­‐>load($command-­‐>getAccountId());  
                   $account-­‐>credit($command-­‐>getBalance());  
    !
                   $this-­‐>repository-­‐>save($account);  
           }  
    }
    

    View Slide

66. class  EventSourcingRepository  implements  RepositoryInterface  
    {  
           public  function  load($id)  
           {  
                   try  {  
                           $events  =  $this-­‐>eventStore-­‐>load($id);  
    !
                           return  $this-­‐>aggregateFactory-­‐>create($events);  
                   }  catch  (EventStreamNotFoundException  $e)  {  
                           throw  AggregateNotFoundException::create($id,  $e);  
                   }  
           }  
    !
           public  function  save(AggregateRoot  $aggregate)  
           {  
                   $events  =  $aggregate-­‐>getUncommittedEvents();  
    !
                   $this-­‐>eventStore-­‐>append(  
                           $aggregate-­‐>getAggregateRootId(),    
                           $events  
                   );  
           }  
    }
    

    View Slide

67. Event Sourcing
    PROS
    

    View Slide

68. Append Only
    Very fast data update
    Deleting data = new events
    

    View Slide

69. Immutable
    Infinite caching
    

    View Slide

70. Sharding
    Aggregates are self-reliant and
    don’t have external references
    No Joins
    

    View Slide

71. Complete historical data
    Being able to replay history is a major benefit
    both technically and for the business
    BI team will love it
    

    View Slide

72. No Database migration
    when updating the data
    DBA will love it
    

    View Slide

73. Event Sourcing
    CONS
    Well, not really
    

    View Slide

74. Ensure consistency
    How to avoid concurrent updates?
    

    View Slide

75. Load aggregate from EventStore: 150€
    Create AccountWasDebited event
    Append event in datastore
    Example
    Two debits of 100€ must not be accepted
    if only 150€ left
    Load aggregate from EventStore: 150€
    Create AccountWasDebited event
    Append event in datastore
    Event must NOT be appended twice
    

    View Slide

76. abstract  class  EventSourcedAggregateRoot  implements  AggregateRootInterface  
    {  
           private  $uncommittedEvents  =  array();  
    !
           private  $playhead  =  -­‐1;  
       
         /**  
           *  Applies  an  event.    
           *  The  event  is  added  to  the  list  of  uncommited  events.  
           */  
           public  function  apply($event)  
           {  
                   $this-­‐>playhead++;  
    !
                   $this-­‐>uncommittedEvents[]  =  DomainMessage::recordNow(  
                           $this-­‐>getAggregateRootId(),  
                           $this-­‐>playhead,  
                           new  Metadata(array()),  
                           $event  
                   );  
           }  
    }
    

    View Slide

77. DBAL EventStore (MySQL)
    CREATE  TABLE  EventStore  (  
    !
       `id`                    INT(11)  AUTO_INCREMENT,  
       `uuid`                BINARY(16),  
       `playhead`        INT(11),  
       `type`                TEXT,  
       `payload`          TEXT,  
       `metadata`        TEXT,  
       `recorded_on`  DATETIME,  
    !
       PRIMARY  KEY  (`id`),  
    !
       UNIQUE  KEY  `UNIQ_PLAYHEAD`  (`uuid`,  `playhead`)  
    );
    

    View Slide

78. E_TOO_MANY_EVENTS
    When your Aggregates are "long lived" it
    may be slow to read the full history
    eg. BankAccount
    

    View Slide

79. Snapshotting
    is a solution
    

    View Slide

80. Snapshotting
    Store the state of the aggregate and
    read only the events created after
    

    View Slide

81. Snapshotting
    Snapshot Snapshot
    Events
    States
    

    View Slide

82. TL;DR
    

    View Slide

83. Command
    Handler
    Events
    Event
    Store
    Command
    Auditing
    Read
    Model
    Event
    Listener
    Event
    Bus
    Projector
    Write
    Read
    Business rules Historical data
    Read / search index
    Side effects
    User intent
    View models
    

    View Slide

84. Frameworks
    C# / Java
    nCQRS
    Fohjin
    NEventStore
    LiteCQRS
    Lokad.CQRS
    Agr.CQRS
    Axon Framework
    JDON
    PHP
    Broadway
    Predaddy
    EventCentric.Core
    litecqrs-php
    ...?
    

    View Slide

85. qandidate-labs/broadway
    qandidate-labs/broadway-demo
    

    View Slide

86. GetEventStore.com
    Event Sourcing
    oriented database
    

    View Slide

87. cqrs.nu
    #MustRead
    

    View Slide

88. dddinphp.org
    Google Groups
    

    View Slide

89. Thank You!
    Slides available at
    moquet.net/talks/phptour-2015
    Leave feedbacks @MattKetmo
    

    View Slide