Event Sourcing & Microservices With Golang

Transcript

1. Event Sourcing & Microservice Architectures With Golang
2. Who am I? • Luke Roberts • Failed Musician • Ruby -> Go • @awsmsrc on the internet • Work for Sqwiggle • Interrupt with questions!
3. Who are you? • Use Golang at work? • Use Microservices at work? • Have heard of Event Sourcing? • Use Event Sourcing at work? • Think these are just fads just like *Javascript everywhere* and can’t wait for the next talk or the pub?
5. What is Sqwiggle? • WebRTC based video collaboration platform for the web and mac • Monolithic architecture built on RoR • 3 apps backed by the same datastore • Modeled via ActiveRecord backed by PostgreSQL
6. So why am I here? • • Side Project • More passive • Full Disclosure - Small Scale Private Beta • Ask for an invite! • Reusable components for the next experiment required
7. Great so we can reuse all the codez??
8. Problems? • New developers need to conceptualize the whole system • Running the full test suite took upwards of 10 minutes • Deployments were closer to 20 minutes • Development boot time was approaching a minute • Even small changes required a deployment of the entire application • Each dependency (redis, memcache) was a dependency for everyone/everything • The haystack was large so tracking down bugs was often difficult • Conflicts between front and backend developers were common • Performance as perceived by the end user • Real time app, doesn't fit classic web development paradigms
9. Our solution • Microservices • EventSourcing • CQRS • Asynchronous Message Broker • I’ll buy you a pint after the talk in exchange for more buzzwords I can take to my boss!
10. What are Microservices • Discrete applications that collaborate towards a common goal • Communication is often via HTTP(S)/REST • Can be deployed on the same or different machines • Have their own data stores • Can be scaled vertically or horizontally dependent on functionality
11. What is Event Sourcing • Store every event that mutates current application state • Workers/Modules/Services subscribe to events and cache what they need • Store snapshots of current state • Always have *Eventual Consistency* in mind
12. Image source: http://ookami86.github.io/event-sourcing-in-practice/#how-eventsourcing-works/04-restoring-objects-5.md
13. Advantages of Event Sourcing • Complete log of every state change ever • Trace/Debug every interaction within the system • Replicate the system at any point in time • Great performance thanks to parallelization • No more ORM’s YAY!!! • Data Mining
14. CQRS as a side effect • Command Query Responsibility Segregation • Often lumped in with event sourcing • Avoid giant model types (less of a problem in Go) • E.G User creation validation rules run in a separate process to the CRM interface shown to the non tech people
15. Issues with Event Sourcing • Side-effects when replaying events - ( Welcome emails ;) ) • Querying the event store for reports and overviews • Changing event formats or payloads • Concurrent writes
16. Events lifecycle • This is our own naive approach YMMV • Present tense is a request or command • Past tense is an event that signals new application state
17. Request - (optionally saved to event store)
18. Response - (Saved to event store)
19. Error Response - (optionally saved to event store)
20. RabbitMQ - The glue that holds it all together • Persistent queue based message broker • Acts as a load balancer • Well supported by all viable languages and platforms • Flexible topologies, ideal for fanning events out through services • Alternatives: Redis / PgSQL pub-sub/ sidekiq / beanstalkd / IronMQ / Amazon SQS • Currently have 3 exchanges: Events, Timber & Services
21. So why not HTTP/REST • HTTP(S) is slow • Transport vs application responsibilities are often poorly defined • Synchronous (Not ideal for soft real time multi media apps)
22. Architecture Overview
23. EXAMPLE: Invite creation • Client: emits invite.create on the events exchange • Karaoke: validates and authorizes • Karaoke: emits invite.created on the events exchange (does not store the invite) • Timber: stores the event and emits invite.created on the timber exchange • Karaoke: receives the invite.created on the timber exchange and updates its private/internal state and sends an email • Auth Service: receives the invite.created on the timber exchange and creates a record of the invite code to allow for account creation
24. Why is Go a great fit? • Simple • Fast • Easy to deploy, runs anywhere • Asynchronous message processing • Micro Services are often I/O heavy/bound
25. Caveats & tradeoffs • Complex • Data aggregation and reporting • More data storage is needed (excluding files and other assets) • Often a different approach to previous experience • Lack of community/blog posts/libraries
26. Development with Docker & Compose • Developers can exactly replicate the environment • Developers can start and stop only the services they need: $: docker-compose start karaoke • Environment is under version control • No need for virtual machines
27. Development with Docker & Compose
28. Deployment & Orchestration • Currently Ansible • Rolling deployments with Docker not solved • Flynn, Dokku, Deis and many others • Docker are rolling their own orchestration • If your using docker in production, lets chat!
29. In Summary • If its a CRUD app, use Rails, Django & break off Go services as necessary • Complexity CAN be a one to one mapping for flexibility • State has bugs, a history of events is concrete even if assumptions on the world are wrong/change • Go is GREAT for micro services
30. You should consider Event Sourcing if…. • Your app doesn't naturally map with the CRUD paradigm • You need version history for audits or ‘undo’ actions • Your application is highly concurrent with long lived operations/actions • You're scaling and growth patterns are unclear
31. All the links • https://www.sqwiggle.com • https://speak.io • https://www.youtube.com/watch?v=jHk5CrCMs0M • http://ookami86.github.io/event-sourcing-in-practice/ #title.md
32. Thanks