Cookpad and Kafka

Here I share how we got started with Kafka at Cookpad.

As seen at South West Ruby, Bristol, UK.

Transcript

1. + @lewispb Kafka
2. Why? @lewispb
3. Feed @lewispb
4. Activities @lewispb
5. class Recipe < ApplicationRecord after_commit :generate_feed_items, on: %i(create update) after_unpublish :delete_feed_items after_unapprove :delete_feed_items def generate_feed_items FeedBuilder.new(self).build_all end end @lewispb Callbacks
6. @lewispb Activity
7. Architecture @lewispb
8. Monolith @lewispb
9. @lewispb web / api search
10. @lewispb “There are patterns that are less about the code and more about how the code is being written, by whom, and within which organization.” - DHH
11. @lewispb web / api search
12. @lewispb web / api search Web team Search team
13. @lewispb web / api search feeds Kafka
14. Messages activated cooksnap approved recipe bookmarked recipe deleted cooksnap deleted recipe dismissed cooksnap followed user liked recipe published cooksnap published recipe recipe visited unapproved recipe unbookmarked recipe unfollowed user unliked recipe unpublished recipe @lewispb
15. Topics @lewispb Recipes activated cooksnap approved recipe bookmarked recipe deleted cooksnap deleted recipe dismissed cooksnap liked recipe published cooksnap published recipe unapproved recipe unbookmarked recipe unliked recipe unpublished recipe Users followed user unfollowed user Recipe Visits recipe visited
16. Messages @lewispb { "key": "recipe 4946334", "type": "published recipe", "event time": “2018-06-01T15:54:39Z", "body": { "author id": 100, "recipe id": 4946334, "title": "My Awesome Recipe", "created at": "2018-06-01T15:54:39Z" } }
17. @lewispb web / api search feeds Kafka
18. @lewispb Apache Kafka® as a Service Confluent Cloud™
19. @lewispb Getting started
20. @lewispb Rails producer cookpad/streamy zendesk/ruby-kafka
21. @lewispb cookpad/streamy
22. @lewispb Rails producer # app/events/published_recipe_event.rb class PublishedRecipeEvent < RecipeEvent private def event_time recipe.published_at end end # app/events/application_event.rb class RecipeEvent < ApplicationEvent topic_name :recipes def initialize(recipe) @recipe = recipe end private ... end
23. @lewispb Rails consumer
24. @lewispb Environments global-web Production Beta mysql kafka global-feeds postgres redis global-web global-feeds postgres redis
25. @lewispb Follows users initial users
26. @lewispb Consumers users feeds feeds Consumer Group
27. @lewispb Consumers users feeds feeds Consumer Group recipes
28. @lewispb Another use case
29. @lewispb GDPR user_signed_up {name: “Lewis”} user:1 user_changed_details {name: “Lewis”} user:1 user_changed_details {name: “Lewis”} user:1 user_closed_account {} user:1
30. @lewispb Monitoring • New Relic • Sentry • Prometheus / Grafana
31. @lewispb Thanks! Follow me on twitter @lewispb