Cookpad and Kafka - Speaker Deck

Slide 1

Slide 1 text

+ @lewispb Kafka

Slide 2

Slide 2 text

Why? @lewispb

Slide 3

Slide 3 text

Feed @lewispb

Slide 4

Slide 4 text

Activities @lewispb

Slide 5

Slide 5 text

class Recipe < ApplicationRecord after_commit :generate_feed_items, on: %i(create update) after_unpublish :delete_feed_items after_unapprove :delete_feed_items def generate_feed_items FeedBuilder.new(self).build_all end end @lewispb Callbacks

Slide 6

Slide 6 text

@lewispb Activity

Slide 7

Slide 7 text

Architecture @lewispb

Slide 8

Slide 8 text

Monolith @lewispb

Slide 9

Slide 9 text

@lewispb web / api search

Slide 10

Slide 10 text

@lewispb “There are patterns that are less about the code and more about how the code is being written, by whom, and within which organization.” - DHH

Slide 11

Slide 11 text

@lewispb web / api search

Slide 12

Slide 12 text

@lewispb web / api search Web team Search team

Slide 13

Slide 13 text

@lewispb web / api search feeds Kafka

Slide 14

Slide 14 text

Messages activated cooksnap approved recipe bookmarked recipe deleted cooksnap deleted recipe dismissed cooksnap followed user liked recipe published cooksnap published recipe recipe visited unapproved recipe unbookmarked recipe unfollowed user unliked recipe unpublished recipe @lewispb

Slide 15

Slide 15 text

Topics @lewispb Recipes activated cooksnap approved recipe bookmarked recipe deleted cooksnap deleted recipe dismissed cooksnap liked recipe published cooksnap published recipe unapproved recipe unbookmarked recipe unliked recipe unpublished recipe Users followed user unfollowed user Recipe Visits recipe visited

Slide 16

Slide 16 text

Messages @lewispb { "key": "recipe 4946334", "type": "published recipe", "event time": “2018-06-01T15:54:39Z", "body": { "author id": 100, "recipe id": 4946334, "title": "My Awesome Recipe", "created at": "2018-06-01T15:54:39Z" } }

Slide 17

Slide 17 text

@lewispb web / api search feeds Kafka

Slide 18

Slide 18 text

@lewispb Apache Kafka® as a Service Conﬂuent Cloud™

Slide 19

Slide 19 text

@lewispb Getting started

Slide 20

Slide 20 text

@lewispb Rails producer cookpad/streamy zendesk/ruby-kafka

Slide 21

Slide 21 text

@lewispb cookpad/streamy

Slide 22

Slide 22 text

@lewispb Rails producer # app/events/published_recipe_event.rb class PublishedRecipeEvent < RecipeEvent private def event_time recipe.published_at end end # app/events/application_event.rb class RecipeEvent < ApplicationEvent topic_name :recipes def initialize(recipe) @recipe = recipe end private ... end

Slide 23

Slide 23 text

@lewispb Rails consumer

Slide 24

Slide 24 text

@lewispb Environments global-web Production Beta mysql kafka global-feeds postgres redis global-web global-feeds postgres redis

Slide 25

Slide 25 text

@lewispb Follows users initial users

Slide 26

Slide 26 text

@lewispb Consumers users feeds feeds Consumer Group

Slide 27

Slide 27 text

@lewispb Consumers users feeds feeds Consumer Group recipes

Slide 28

Slide 28 text

@lewispb Another use case

Slide 29

Slide 29 text

@lewispb GDPR user_signed_up {name: “Lewis”} user:1 user_changed_details {name: “Lewis”} user:1 user_changed_details {name: “Lewis”} user:1 user_closed_account {} user:1