$30 off During Our Annual Pro Sale. View Details »

Event Sourcing

Event Sourcing

PHPTour 2015

Matthieu Moquet

May 13, 2015
Tweet

More Decks by Matthieu Moquet

Other Decks in Programming

Transcript

  1. Event Sourcing
    #PHPTour 2015
    @MattKetmo

    View Slide

  2. CQRS
    Event Sourcing
    Reactive
    DDD
    }Patterns to build &
    structure applications
    Not new buzzwords,
    exist for many years
    How can it help to
    scale horizontally?
    Today’s focus

    View Slide

  3. Matthieu Moquet
    @MattKetmo
    @ BlaBlaCar
    for ~4 years

    View Slide

  4. {Disclaimer
    Oriented for long lived apps.

    Overkill for RAD / CRUD / TODO apps.
    Use those patterns in moderation.

    There is no silver bullet.
    This talk aims to open your mind.

    Think different!

    View Slide

  5. I/O

    View Slide

  6. 90%
    ViewModels / Flexible
    Eventually consistent
    10%
    Validation / Business rules
    Coherence (ACID)
    Read Write
    Fast & Scale

    View Slide

  7. Use case
    bank accounts
    Account #1
    !
    1337€
    Account #2
    !
    42€
    Account #3
    !
    10€
    + Transfer money

    View Slide

  8. Account
    !
    id
    user_id
    amount
    User
    !
    id
    name
    ...

    View Slide

  9. Facade
    (Controller)
    Models
    (ORM)
    DB

    View Slide

  10. //  Open  new  account  
    $account  =  new  Account();  
    $account-­‐>setAmount(0,  'EUR');  
    $em-­‐>persist($account);  
    $em-­‐>flush();  
    !
    //  Credit  100  EUR  
    $account-­‐>setAmount(100,  'EUR');  
    $em-­‐>flush();  
    !
    //  Debit  20  EUR  
    $account-­‐>setAmount(80,  'EUR');  
    $em-­‐>flush();

    View Slide

  11. is not the solution
    When user process an operation on his account his
    doesn’t directly update his total amount of money
    We need a service that translate our business needs
    CRUD

    View Slide

  12. Facade
    (Controller)
    Models
    (ORM)
    DB
    Service
    Layer
    (Business)

    View Slide

  13. interface  BankAccountService    
    {  
           /**  @return  int  */  
           public  function  open();  
    !
           /**  @return  Account  */  
           public  function  get($accountId);  
    !
           /**  @return  Account[]  */  
           public  function  findAll($userId);  
    !
           /**  @return  void  */  
           public  function  credit($accountId,  $balance);  
    !
           /**  @return  void  */  
           public  function  debit($accountId,  $balance);  
    }  

    View Slide

  14. CQS
    Command
    Query
    Separation
    }Each operation is
    either a Command or
    a Query, not both

    View Slide

  15. Commands
    Queries
    Change the state of the system.
    Do not return anything.
    Does not change the state of the system.
    Return data.
    Write only
    Read only

    View Slide

  16. Commands
    /**  @return  Account  */  
    public  function  get($accountId);  
    !
    /**  @return  Account[]  */  
    public  function  findAll($userId);
    /**  @return  int  */  
    public  function  open();  
    !
    /**  @return  void  */  
    public  function  credit($accountId,  $balance);  
    !
    /**  @return  void  */  
    public  function  debit($accountId,  $balance);
    Should be void
    Queries

    View Slide

  17. /**    
     *  @param  Uuid  $accountId  
     *  
     *  @return  void    
     */  
    public  function  open(Uuid  $accountId);  
    Let the client generate the identifier,
    not the infrastructure (database)

    View Slide

  18. CQRS
    Command
    Query
    Responsibility
    Segregation
    }Like CQS but
    separate queries and
    commands in
    different classes

    View Slide

  19. Command Facade
    Query Facade
    Command
    Service
    Query
    Service
    Models
    (ORM)
    DB

    View Slide

  20. CQRS challenges the assumption that
    reading and writing are sharing the
    same abstractions
    Databases
    Models
    Apps

    View Slide

  21. Command Facade
    Query Facade
    Command
    Handler
    Query
    Repository
    Command
    Models
    DB
    Read
    Models
    Command
    DTO
    Segregation of read and write is a
    radical form of decoupling

    View Slide

  22. View Slide

  23. The interesting part about CQRS is not
    the pattern itself.
    But it allows to challenge established
    assumptions and opens new architectural
    options.

    View Slide

  24. How to
    Scale

    View Slide

  25. Now that the workflow of Read & Write
    has been separated.
    We can optimize each side separately.

    View Slide

  26. asynchronously
    Since Command Handlers
    do NOT return anything, 

    then do the execution

    View Slide

  27. Command
    Facade
    Command
    Handlers
    Commands Queueing
    Ack

    View Slide

  28. !
    $accoundId  =  AccountId::generate();  
    $command  =  new  OpenAccountCommand($accountId);  
    !
    if  (/*  is  command  valid  */)  {  
           $commandBus-­‐>handle($command);  
    !
           return  new  Response(  
                   null,  
                   $async  ?  202  :  201,  //  Accepted  :  Created  
                   ['Location'  =>  "/accounts/$accountId"]  
           );    
    }

    View Slide

  29. $command  =  new  DebitCommand(  
           $accountId,    
           new  Money(20,  'EUR')  
    );  
    !
    if  (/*  account  has  enough  money  */)  {  
           $commandBus-­‐>handle($command);  
    !
           return  new  Response(null,  204);  
    }

    View Slide

  30. Command tracking
    Scale workers
    Horizontally by instantiating more workers (subscribers)
    Generate an identifier per command to let the client
    tracks its status
    Rabbitmq, Kafka, Gearman, ………
    make your choice on queues.io

    View Slide

  31. Projections
    On the Read side you
    should prepare you data
    as it will be requested

    View Slide

  32. Having only 1 data store
    for reads & writes does not
    scale (well)
    different usage
    different needs

    View Slide

  33. Denormalize your data
    UI View models
    Statistic models
    Search index models
    API Read models
    ...
    Read models are
    faster than JOIN

    View Slide

  34. • search & aggregations capabilities
    • horizontal scaling architecture (sharding, replication)
    • denormalization
    • fast
    Secondary data store to read your data

    View Slide

  35. {  
       "user_id":  133742  
       "from":  "Paris",  
       "to":  "Luxembourg",  
       "by":  [  
           "Reims",    
           "..."  
       ],  
       "date":  "2015-­‐05-­‐11",  
    }
    Input (mysql)

    View Slide

  36. [{  
       "trip":  {  
           "from":  "Paris",  
           "to":  "Luxembourg",  
           "date":  "2015-­‐05-­‐11",  
           "..."  
       },  
       "user":  {  
           "name":  "John  D",  
           "age":  29,  
           "grade":  "beginner",  
           "..."  
       }  
    }]
    Output (Elasticsearch)
    denormalized
    searchable

    View Slide

  37. View Slide

  38. SELECT  tweets.*,  users.*  
       FROM  tweets  
    !
    JOIN  users  
       ON  users.id  =  tweet.sender_id  
    !
    JOIN  follows  
       ON  follows.followee_id  =  user.id  
    !
    WHERE  follows.follower_id  =  $userId  
    !
    ORDER  BY  tweets.time  DESC  
    LIMIT  100

    View Slide

  39. View Slide

  40. Tweets stream
    DB
    Aggs
    Timelines
    {  
       "user_id":  1234567890  
       "status":  "Hello  World"  
       "timestamp":  1430491773  
    }
    !
    [{  
       "tweet_id":  1234567890876543,  
       "username":  "MattKetmo",  
       "name":  "Matthieu  Moquet",  
       "timestamp":  1430491773,  
       "status":  "Hello  World",  
    },  {  
       "tweet_id":  1234567890886445,  
       ...  
    }]  

    View Slide

  41. SELECT  tweets  
       FROM  timelines  
     WHERE  timeline_id  =  $userId  
     LIMIT  500

    View Slide

  42. PageViewEvents
    DB
    Aggs
    Increment
    counters
    /month
    Google Analytics
    /day /hour total
    {  
       "eventType":  "PageViewEvent"  
       "timestamp":  1430491773,  
       "ipAddress":  "12.34.56.78",  
       "sessionId":  "abcd1234567890",  
       "pageUrl":  "/hello-­‐world",  
       "..."  
    }

    View Slide

  43. Using Cassandra you need to:
    ‣ know the read requests before creating your data models
    ‣ create as many tables (ie. KeySpaces) than you have views
    ‣ denormalize the data (no join allowed)
    ’s keypoints

    View Slide

  44. C* C* C*
    Primary
    Data
    C*
    Read Data
    User post message
    Endpoint

    View Slide

  45. Multi Data Center
    Replication + Sharding + Cache

    View Slide

  46. Command
    Command
    Handler
    Primary
    DB
    Projections
    View
    Models
    Message Bus
    Write
    Read
    Eventual Consistency
    {...}
    {...}
    {...}
    {...}

    View Slide

  47. Event Sourcing
    change the way we store our primary data

    View Slide

  48. If we know the events of the past,
    we can reconstitute the present.
    Event Sourcing

    View Slide

  49. Event Sourcing
    Register only a series of events.
    Reconstitute the state of current "entity"
    by reading the past events.

    View Slide

  50. the standard way
    Something
    happen State A State B
    Something
    happen
    (delta) (dropped) (stored)
    (delta)
    Time

    View Slide

  51. the event sourcing way
    Something
    happen State A State B
    Something
    happen
    (stored) (reconstituted) (reconstituted)
    (stored)
    Time

    View Slide

  52. [{  
         "uuid":  "110e8400-­‐e29b-­‐11d4-­‐a716-­‐446655440000",  
         "type":  "AccountWasOpen",  
         "recorded_on":  "2015-­‐05-­‐11T13:37:00Z",  
         "payload":  {}  
    },  {  
         "uuid":  "110e8400-­‐e29b-­‐11d4-­‐a716-­‐446655440000",  
         "type":  "AccountWasCredited",  
         "recorded_on":  "2015-­‐05-­‐11T14:42:00Z",  
         "payload":  {  "amount":  100,  "currency":  "EUR"  }  
    },  {  
       "..."  
    }]
    Event Store

    View Slide

  53. Aggregates
    Domain models which serve the business logic
    using the ubiquitous language.
    Work as an isolated object (or graph of objects)
    which doesn’t reference any others.
    Can ensure only its own integrity.

    View Slide

  54. //  Open  a  new  account  
    $accountId  =  AccountId::generate();  
    $account  =  BankAccount::open($accountId);  
    !
    //  Add  some  money  
    $account-­‐>credit(new  Money(100,  'EUR'));

    View Slide

  55. Records events
    Aggregates
    class  BankAccount  implements  AggregateRootInterface  
    {  
           //  ...  
    }

    View Slide

  56. interface  AggregateRoot  
    {  
           /**  
             *  @return  DomainEventStream  
             */  
           public  function  getUncommittedEvents();  
    !
           /**  
             *  @return  string  
             */  
           public  function  getAggregateRootId();  
    }

    View Slide

  57. $accountId  =  AccountId::generate();  
    $account  =  BankAccount::open($accountId);  
    $account-­‐>credit(new  Money(100,  'EUR'));  
    !
    //  Retrieve  event  stream  
    $events  =  $account-­‐>getUncommittedEvents();  
    //  -­‐  AccountWasOpen  
    //  -­‐  AccountWasCredited

    View Slide

  58. class  BankAccount  extends  EventSourcedAggregateRoot  
    {  
           private  $accountId;  
    !
           public  function  getAggregateRootId()  
           {  
                   return  $this-­‐>accountId;  
           }  
    !
           public  static  function  open(AccountId  $accountId)  
           {  
                   $account  =  new  self();  
                   $account-­‐>apply(new  AccountWasOpen($accountId));  
    !
                   return  $account;  
           }  
    !
           public  function  credit(Money  $balance)  
           {  
                   $this-­‐>apply(new  AccountWasCredited(  
                           $this-­‐>accountId,  
                           $balance-­‐>getAmount(),  
                           $balance-­‐>getCurrency()  
                   );  
           }  
    }

    View Slide

  59. Reconstitute from events
    Aggregates

    View Slide

  60. class  BankAccount  extends  EventSourcedAggregateRoot  
    {  
           //  ...  
       
           protected  function  applyAccountWasOpen(AccountWasOpen  $event)  
           {  
                   $this-­‐>accountId  =  $event-­‐>getAccountId();  
           }  
    !
           protected  function  applyAccountWasCredited(AccountWasCredited  $event)  
           {  
                   $this-­‐>amount  +=  $event-­‐>getBalance();  
           }  
    !
           protected  function  applyAccountWasDebited(AccountWasDebited  $event)  
           {  
                   $this-­‐>amount  -­‐=  $event-­‐>getBalance();  
           }  
    }

    View Slide

  61. Aggregates
    Protect invariants

    View Slide

  62. class  BankAccount  extends  EventSourcedAggregateRoot  
    {  
           //  ...  
    !
           public  function  debit(Money  $balance)  
           {  
                   if  ($this-­‐>amount  <  $balance-­‐>getAmount())  {  
                           throw  new  NotEnoughMoneyException(  
                                   'Cannot  debit  more  than  current  amount'  
                           );  
                   }  
    !
                   return  $this-­‐>apply(new  AccountWasCredited(  
                           $this-­‐>accountId,  
                           $balance-­‐>getAmount(),  
                           $balance-­‐>getCurrency()  
                   ));                
           }  
    }

    View Slide

  63. Command + Event Sourcing

    View Slide

  64. Command
    User intent
    Events Event Store
    Apply changes
    on system
    Primary DB
    Transactional append

    View Slide

  65. class  BankAccountCommandHandler  extends  CommandHandler  
    {  
           public  function  handleOpenAccount(OpenAccount  $command)  
           {  
                   $account  =  BankAccount::open($command-­‐>getAccountId());  
    !
                   $this-­‐>repository-­‐>save($account);  
           }  
    !
           public  function  handleCreditAccount(CreditAccount  $command)  
           {  
                   $account  =  $this-­‐>repository-­‐>load($command-­‐>getAccountId());  
                   $account-­‐>credit($command-­‐>getBalance());  
    !
                   $this-­‐>repository-­‐>save($account);  
           }  
    }

    View Slide

  66. class  EventSourcingRepository  implements  RepositoryInterface  
    {  
           public  function  load($id)  
           {  
                   try  {  
                           $events  =  $this-­‐>eventStore-­‐>load($id);  
    !
                           return  $this-­‐>aggregateFactory-­‐>create($events);  
                   }  catch  (EventStreamNotFoundException  $e)  {  
                           throw  AggregateNotFoundException::create($id,  $e);  
                   }  
           }  
    !
           public  function  save(AggregateRoot  $aggregate)  
           {  
                   $events  =  $aggregate-­‐>getUncommittedEvents();  
    !
                   $this-­‐>eventStore-­‐>append(  
                           $aggregate-­‐>getAggregateRootId(),    
                           $events  
                   );  
           }  
    }

    View Slide

  67. Event Sourcing
    PROS

    View Slide

  68. Append Only
    Very fast data update
    Deleting data = new events

    View Slide

  69. Immutable
    Infinite caching

    View Slide

  70. Sharding
    Aggregates are self-reliant and
    don’t have external references
    No Joins

    View Slide

  71. Complete historical data
    Being able to replay history is a major benefit
    both technically and for the business
    BI team will love it

    View Slide

  72. No Database migration
    when updating the data
    DBA will love it

    View Slide

  73. Event Sourcing
    CONS
    Well, not really

    View Slide

  74. Ensure consistency
    How to avoid concurrent updates?

    View Slide

  75. Load aggregate from EventStore: 150€
    Create AccountWasDebited event
    Append event in datastore
    Example
    Two debits of 100€ must not be accepted
    if only 150€ left
    Load aggregate from EventStore: 150€
    Create AccountWasDebited event
    Append event in datastore
    Event must NOT be appended twice

    View Slide

  76. abstract  class  EventSourcedAggregateRoot  implements  AggregateRootInterface  
    {  
           private  $uncommittedEvents  =  array();  
    !
           private  $playhead  =  -­‐1;  
       
         /**  
           *  Applies  an  event.    
           *  The  event  is  added  to  the  list  of  uncommited  events.  
           */  
           public  function  apply($event)  
           {  
                   $this-­‐>playhead++;  
    !
                   $this-­‐>uncommittedEvents[]  =  DomainMessage::recordNow(  
                           $this-­‐>getAggregateRootId(),  
                           $this-­‐>playhead,  
                           new  Metadata(array()),  
                           $event  
                   );  
           }  
    }

    View Slide

  77. DBAL EventStore (MySQL)
    CREATE  TABLE  EventStore  (  
    !
       `id`                    INT(11)  AUTO_INCREMENT,  
       `uuid`                BINARY(16),  
       `playhead`        INT(11),  
       `type`                TEXT,  
       `payload`          TEXT,  
       `metadata`        TEXT,  
       `recorded_on`  DATETIME,  
    !
       PRIMARY  KEY  (`id`),  
    !
       UNIQUE  KEY  `UNIQ_PLAYHEAD`  (`uuid`,  `playhead`)  
    );

    View Slide

  78. E_TOO_MANY_EVENTS
    When your Aggregates are "long lived" it
    may be slow to read the full history
    eg. BankAccount

    View Slide

  79. Snapshotting
    is a solution

    View Slide

  80. Snapshotting
    Store the state of the aggregate and
    read only the events created after

    View Slide

  81. Snapshotting
    Snapshot Snapshot
    Events
    States

    View Slide

  82. TL;DR

    View Slide

  83. Command
    Handler
    Events
    Event
    Store
    Command
    Auditing
    Read
    Model
    Event
    Listener
    Event
    Bus
    Projector
    Write
    Read
    Business rules Historical data
    Read / search index
    Side effects
    User intent
    View models

    View Slide

  84. Frameworks
    C# / Java
    nCQRS
    Fohjin
    NEventStore
    LiteCQRS
    Lokad.CQRS
    Agr.CQRS
    Axon Framework
    JDON
    PHP
    Broadway
    Predaddy
    EventCentric.Core
    litecqrs-php
    ...?

    View Slide

  85. qandidate-labs/broadway
    qandidate-labs/broadway-demo

    View Slide

  86. GetEventStore.com
    Event Sourcing
    oriented database

    View Slide

  87. cqrs.nu
    #MustRead

    View Slide

  88. dddinphp.org
    Google Groups

    View Slide

  89. Thank You!
    Slides available at
    moquet.net/talks/phptour-2015
    Leave feedbacks @MattKetmo

    View Slide