So You’ve Got Yourself a Kafka: Event-Powered Rails Services

Slide 1

Slide 1 text

So You’ve Got Yourself a Kafka

Slide 2

Slide 2 text

stella cotton | @practice_cactus

Slide 3

Slide 3 text

stella cotton | @practice_cactus

Slide 4

Slide 4 text

stella cotton | @practice_cactus

Slide 5

Slide 5 text

stella cotton | @practice_cactus • What is Kafka • Kafka Powered Services • Practical Considerations and Challenges

Slide 6

Slide 6 text

stella cotton | @practice_cactus What is Kafka?

Slide 7

Slide 7 text

stella cotton | @practice_cactus Distributed Streaming Platform

Slide 8

Slide 8 text

stella cotton | @practice_cactus

Slide 9

Slide 9 text

stella cotton | @practice_cactus     Append Only Log

Slide 10

Slide 10 text

stella cotton | @practice_cactus 010-09-16T15:13:46.677020+00:00 app[web.1]: Processing PostController#list … 2010-09-16T15:13:46.677023+00:00 app[web.1]: Rendering template within layouts/application 2010-09-16T15:13:46.677902+00:00 app[web.1]: Rendering post/list 2010-09-16T15:13:46.678990+00:00 app[web.1]: Rendered includes/_header (0.1ms) 2010-09-16T15:13:46.698234+00:00 app[web.1]: Completed in 74ms (View: 31, DB: 40) | 200 OK 2010-09-16T15:13:46.723498+00:00 heroku[router]: at=info method=GET path=“/posts”… 2010-09-16T15:13:47.893472+00:00 app[worker.1]: 2 jobs processed at 16.6761 j/s, 0 failed ...

Slide 11

Slide 11 text

stella cotton | @practice_cactus     Producers

Slide 12

Slide 12 text

stella cotton | @practice_cactus

Slide 13

Slide 13 text

stella cotton | @practice_cactus

Slide 14

Slide 14 text

stella cotton | @practice_cactus     Consumers

Slide 15

Slide 15 text

stella cotton | @practice_cactus

Slide 16

Slide 16 text

stella cotton | @practice_cactus

Slide 17

Slide 17 text

stella cotton | @practice_cactus     How is this different from Sidekiq or Resque?

Slide 18

Slide 18 text

stella cotton | @practice_cactus     user_event Topic

Slide 19

Slide 19 text

stella cotton | @practice_cactus   { "some_event": "payload", "published_at": "2016-03-16T16:35:04Z" }

Slide 20

Slide 20 text

stella cotton | @practice_cactus

Slide 21

Slide 21 text

stella cotton | @practice_cactus • zendesk/ruby-kafka • DeliveryBoy + Racecar • Phobos

Slide 22

Slide 22 text

stella cotton | @practice_cactus • Karafka • WaterDrop

Slide 23

Slide 23 text

stella cotton | @practice_cactus • zendesk/ruby-kafka • DeliveryBoy + Racecar • Phobos

Slide 24

Slide 24 text

stella cotton | @practice_cactus gem 'delivery_boy' $ bundle $ bundle exec rails generate delivery_boy:install

Slide 25

Slide 25 text

stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<: *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092

Slide 26

Slide 26 text

stella cotton | @practice_cactus class OrdersController < ApplicationController def create @comment = Order.create!(params) DeliveryBoy.deliver_async(order.to_json, topic: "user_event") end end

Slide 27

Slide 27 text

stella cotton | @practice_cactus

Slide 28

Slide 28 text

stella cotton | @practice_cactus

Slide 29

Slide 29 text

stella cotton | @practice_cactus

Slide 30

Slide 30 text

stella cotton | @practice_cactus class OrdersController < ApplicationController def create @comment = Order.create!(params) DeliveryBoy.deliver_async( order.to_json, topic: "user_event", partition_key: user.id ) end end

Slide 31

Slide 31 text

stella cotton | @practice_cactus gem 'racecar' $ bundle $ bundle exec rails generate racecar:install

Slide 32

Slide 32 text

Slide 33

Slide 33 text

stella cotton | @practice_cactus class UserEventConsumer < Racecar::Consumer subscribes_to "user_event" def process(message) data = JSON.parse(message.value) puts data end end

Slide 34

Slide 34 text

stella cotton | @practice_cactus     bundle exec racecar UserEventConsumer

Slide 35

Slide 35 text

stella cotton | @practice_cactus

Slide 36

Slide 36 text

stella cotton | @practice_cactus

Slide 37

Slide 37 text

stella cotton | @practice_cactus     Offset

Slide 38

Slide 38 text

stella cotton | @practice_cactus

Slide 39

Slide 39 text

stella cotton | @practice_cactus Why Kafka?

Slide 40

Slide 40 text

stella cotton | @practice_cactus     High Throughput

Slide 41

Slide 41 text

stella cotton | @practice_cactus     Tracking Consumer Position

Slide 42

Slide 42 text

stella cotton | @practice_cactus     Consumer Agreement is Tricky

Slide 43

Slide 43 text

stella cotton | @practice_cactus     Reading and Writing is O(1)

Slide 44

Slide 44 text

stella cotton | @practice_cactus     Scalable and Fault Tolerant

Slide 45

Slide 45 text

stella cotton | @practice_cactus     Thorough Introduction to Apache Kafka https://hackernoon.com/thorough-introduction-to-apache-kafka-6fbf2989bbc1  Stanislav Kozlovski

Slide 46

Slide 46 text

stella cotton | @practice_cactus     1,000,000,000,000

Slide 47

Slide 47 text

stella cotton | @practice_cactus Kafka Powered Services

Slide 48

Slide 48 text

stella cotton | @practice_cactus Fault Tolerant Replacement for RPC

Slide 49

Slide 49 text

stella cotton | @practice_cactus def create_order create_order_record charge_credit_card send_confirmation_email end

Slide 50

Slide 50 text

stella cotton | @practice_cactus

Slide 51

Slide 51 text

stella cotton | @practice_cactus   Challenge 1:  Upstream service is responsible for downstream availability

Slide 52

Slide 52 text

stella cotton | @practice_cactus

Slide 53

Slide 53 text

stella cotton | @practice_cactus   Challenge 2:  New downstream services require changes to the   upstream service

Slide 54

Slide 54 text

stella cotton | @practice_cactus

Slide 55

Slide 55 text

stella cotton | @practice_cactus     There’s No Such Thing   as a Free Lunch

Slide 56

Slide 56 text

stella cotton | @practice_cactus   New Challenge:  Dependencies   are no longer explicit

Slide 57

Slide 57 text

stella cotton | @practice_cactus Architectural Patterns

Slide 58

Slide 58 text

stella cotton | @practice_cactus   The Many Meanings of   Event-Driven Architecture    Martin Fowler - GOTO Chicago  https://www.youtube.com/watch?v=STKCRSUsyP0  https://martinfowler.com/articles/201701-event-driven.html 

Slide 59

Slide 59 text

stella cotton | @practice_cactus     Event Notification

Slide 60

Slide 60 text

stella cotton | @practice_cactus

Slide 61

Slide 61 text

stella cotton | @practice_cactus

Slide 62

Slide 62 text

stella cotton | @practice_cactus     Event Carried State Transfer

Slide 63

Slide 63 text

stella cotton | @practice_cactus

Slide 64

Slide 64 text

stella cotton | @practice_cactus

Slide 65

Slide 65 text

stella cotton | @practice_cactus

Slide 66

Slide 66 text

stella cotton | @practice_cactus     Event-Sourced Architecture

Slide 67

Slide 67 text

stella cotton | @practice_cactus     Challenge 1:   Code Changes

Slide 68

Slide 68 text

stella cotton | @practice_cactus     Challenge 2:   Third-Party Integrations

Slide 69

Slide 69 text

stella cotton | @practice_cactus     Command Query   Responsibility Segregation (CQRS)

Slide 70

Slide 70 text

stella cotton | @practice_cactus     CRUD vs CQRS

Slide 71

Slide 71 text

stella cotton | @practice_cactus

Slide 72

Slide 72 text

stella cotton | @practice_cactus

Slide 73

Slide 73 text

stella cotton | @practice_cactus     When to use it?

Slide 74

Slide 74 text

stella cotton | @practice_cactus   When to Use   (and Not to Use) CQRS    Grzegorz Sitkowski  http://blog.objectivity.co.uk/when-to-use-and-not-to-use-cqrs/

Slide 75

Slide 75 text

Slide 76

Slide 76 text

stella cotton | @practice_cactus Practical Considerations

Slide 77

Slide 77 text

stella cotton | @practice_cactus Slow Consumers

Slide 78

Slide 78 text

stella cotton | @practice_cactus     Increase Consumers in   the Consumer Group

Slide 79

Slide 79 text

stella cotton | @practice_cactus     process-user-events: bundle exec racecar UserEventConsumer

Slide 80

Slide 80 text

stella cotton | @practice_cactus     process-user-events-1: bundle exec racecar UserEventConsumer  process-user-events-2: bundle exec racecar UserEventConsumer  process-user-events-3: bundle exec racecar UserEventConsumer

Slide 81

Slide 81 text

stella cotton | @practice_cactus

Slide 82

Slide 82 text

stella cotton | @practice_cactus     Sidekiq

Slide 83

Slide 83 text

stella cotton | @practice_cactus     Monitor and Alert on   Consumer Lag

Slide 84

Slide 84 text

stella cotton | @practice_cactus https://github.com/zendesk/ ruby-kafka#what-to-monitor

Slide 85

Slide 85 text

stella cotton | @practice_cactus     Gameday Scenarios

Slide 86

Slide 86 text

stella cotton | @practice_cactus     “exactly once” vs “at least once” https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Slide 87

Slide 87 text

stella cotton | @practice_cactus     Design Consumers for Duplicated Events

Slide 88

Slide 88 text

stella cotton | @practice_cactus     1. Rely on UPSERT  2. Unique identifiers for events

Slide 89

Slide 89 text

stella cotton | @practice_cactus Schemas and Data Serialization

Slide 90

Slide 90 text

stella cotton | @practice_cactus     Downside to Data Flexibility

Slide 91

Slide 91 text

stella cotton | @practice_cactus     Schema Registries

Slide 92

Slide 92 text

stella cotton | @practice_cactus     JSON

Slide 93

Slide 93 text

stella cotton | @practice_cactus Pros: • Human readable • Universal language support

Slide 94

Slide 94 text

stella cotton | @practice_cactus Cons: • Payload can be large due to repeated keys • No built-in documentation, aliases

Slide 95

Slide 95 text

stella cotton | @practice_cactus     Avro

Slide 96

Slide 96 text

stella cotton | @practice_cactus Pros: • Robust schema support with types • Very compact and very fast • Built in documentation • Helps you evolve data over time

Slide 97

Slide 97 text

stella cotton | @practice_cactus     Avro::Builder https://www.salsify.com/blog/engineering/adventures-in-avro

Slide 98

Slide 98 text

stella cotton | @practice_cactus Illustrations by Freepik Beyond 50,000 Partitions: How Heroku Operates and Pushes the Limits of Kafka at Scale  http://www.dataengconf.com/beyond-50k- partitions-how-heroku-pushes-the-limits-of-kafka-at- scale    Dogfooding Kafka: How We Built Heroku's Real- Time Platform Event Stream  https://www.salesforce.com/video/302281/   

Slide 99

Slide 99 text

stella cotton | @practice_cactus Illustrations by Freepik Postgres 10, Performance, and You  Gabe Enslein  Room 306-307   

Slide 100

Slide 100 text

stella cotton | @practice_cactus Illustrations by Freepik https://sfdc.co/herokujobs

Slide 101

Slide 101 text

stella cotton | @practice_cactus Thank You!