So You’ve Got Yourself a Kafka: Event-Powered Rails Services

So You’ve Got Yourself a Kafka: Event-Powered Rails Services

It’s easier than ever to integrate a distributed commit log system like Kafka into your stack. But once you have it, what do you do with it? Come learn the basics about Kafka and event-driven systems and how you can use them to power your Rails services. We’ll also venture beyond the theoretical to talk about practical considerations and unusual operational challenges that your event-driven Rails services might face, like monitoring, queue processing, and scaling slow consumers.

9c3bc4fea06c0e0bafab417be0bbdb74?s=128

stellacotton

April 18, 2018
Tweet

Transcript

  1. 5.

    stella cotton | @practice_cactus • What is Kafka • Kafka

    Powered Services • Practical Considerations and Challenges
  2. 10.

    stella cotton | @practice_cactus 010-09-16T15:13:46.677020+00:00 app[web.1]: Processing PostController#list … 2010-09-16T15:13:46.677023+00:00

    app[web.1]: Rendering template within layouts/application 2010-09-16T15:13:46.677902+00:00 app[web.1]: Rendering post/list 2010-09-16T15:13:46.678990+00:00 app[web.1]: Rendered includes/_header (0.1ms) 2010-09-16T15:13:46.698234+00:00 app[web.1]: Completed in 74ms (View: 31, DB: 40) | 200 OK 2010-09-16T15:13:46.723498+00:00 heroku[router]: at=info method=GET path=“/posts”… 2010-09-16T15:13:47.893472+00:00 app[worker.1]: 2 jobs processed at 16.6761 j/s, 0 failed ...
  3. 24.
  4. 25.

    stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<:

    *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092
  5. 26.

    stella cotton | @practice_cactus class OrdersController < ApplicationController def create

    @comment = Order.create!(params) DeliveryBoy.deliver_async(order.to_json, topic: "user_event") end end
  6. 30.

    stella cotton | @practice_cactus class OrdersController < ApplicationController def create

    @comment = Order.create!(params) DeliveryBoy.deliver_async( order.to_json, topic: "user_event", partition_key: user.id ) end end
  7. 32.

    stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<:

    *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092
  8. 33.

    stella cotton | @practice_cactus class UserEventConsumer < Racecar::Consumer subscribes_to "user_event"

    def process(message) data = JSON.parse(message.value) puts data end end
  9. 45.

    stella cotton | @practice_cactus 
 
 Thorough Introduction to Apache

    Kafka https://hackernoon.com/thorough-introduction-to-apache-kafka-6fbf2989bbc1
 Stanislav Kozlovski
  10. 58.

    stella cotton | @practice_cactus 
 The Many Meanings of 


    Event-Driven Architecture
 
 Martin Fowler - GOTO Chicago
 https://www.youtube.com/watch?v=STKCRSUsyP0
 https://martinfowler.com/articles/201701-event-driven.html

  11. 74.

    stella cotton | @practice_cactus 
 When to Use 
 (and

    Not to Use) CQRS
 
 Grzegorz Sitkowski
 http://blog.objectivity.co.uk/when-to-use-and-not-to-use-cqrs/
  12. 75.

    stella cotton | @practice_cactus 
 The Many Meanings of 


    Event-Driven Architecture
 
 Martin Fowler - GOTO Chicago
 https://www.youtube.com/watch?v=STKCRSUsyP0
 https://martinfowler.com/articles/201701-event-driven.html

  13. 80.

    stella cotton | @practice_cactus 
 
 process-user-events-1: bundle exec racecar

    UserEventConsumer
 process-user-events-2: bundle exec racecar UserEventConsumer
 process-user-events-3: bundle exec racecar UserEventConsumer
  14. 86.

    stella cotton | @practice_cactus 
 
 “exactly once” vs “at

    least once” https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
  15. 94.

    stella cotton | @practice_cactus Cons: • Payload can be large

    due to repeated keys • No built-in documentation, aliases
  16. 96.

    stella cotton | @practice_cactus Pros: • Robust schema support with

    types • Very compact and very fast • Built in documentation • Helps you evolve data over time
  17. 98.

    stella cotton | @practice_cactus Illustrations by Freepik Beyond 50,000 Partitions:

    How Heroku Operates and Pushes the Limits of Kafka at Scale
 http://www.dataengconf.com/beyond-50k- partitions-how-heroku-pushes-the-limits-of-kafka-at- scale
 
 Dogfooding Kafka: How We Built Heroku's Real- Time Platform Event Stream
 https://www.salesforce.com/video/302281/