So You’ve Got Yourself a Kafka: Event-Powered Rails Services

So You’ve Got Yourself a Kafka: Event-Powered Rails Services

It’s easier than ever to integrate a distributed commit log system like Kafka into your stack. But once you have it, what do you do with it? Come learn the basics about Kafka and event-driven systems and how you can use them to power your Rails services. We’ll also venture beyond the theoretical to talk about practical considerations and unusual operational challenges that your event-driven Rails services might face, like monitoring, queue processing, and scaling slow consumers.

9c3bc4fea06c0e0bafab417be0bbdb74?s=128

stellacotton

April 18, 2018
Tweet

Transcript

  1. So You’ve Got Yourself a Kafka

  2. stella cotton | @practice_cactus

  3. stella cotton | @practice_cactus

  4. stella cotton | @practice_cactus

  5. stella cotton | @practice_cactus • What is Kafka • Kafka

    Powered Services • Practical Considerations and Challenges
  6. stella cotton | @practice_cactus What is Kafka?

  7. stella cotton | @practice_cactus Distributed Streaming Platform

  8. stella cotton | @practice_cactus

  9. stella cotton | @practice_cactus 
 
 Append Only Log

  10. stella cotton | @practice_cactus 010-09-16T15:13:46.677020+00:00 app[web.1]: Processing PostController#list … 2010-09-16T15:13:46.677023+00:00

    app[web.1]: Rendering template within layouts/application 2010-09-16T15:13:46.677902+00:00 app[web.1]: Rendering post/list 2010-09-16T15:13:46.678990+00:00 app[web.1]: Rendered includes/_header (0.1ms) 2010-09-16T15:13:46.698234+00:00 app[web.1]: Completed in 74ms (View: 31, DB: 40) | 200 OK 2010-09-16T15:13:46.723498+00:00 heroku[router]: at=info method=GET path=“/posts”… 2010-09-16T15:13:47.893472+00:00 app[worker.1]: 2 jobs processed at 16.6761 j/s, 0 failed ...
  11. stella cotton | @practice_cactus 
 
 Producers

  12. stella cotton | @practice_cactus

  13. stella cotton | @practice_cactus

  14. stella cotton | @practice_cactus 
 
 Consumers

  15. stella cotton | @practice_cactus

  16. stella cotton | @practice_cactus

  17. stella cotton | @practice_cactus 
 
 How is this different

    from Sidekiq or Resque?
  18. stella cotton | @practice_cactus 
 
 user_event Topic

  19. stella cotton | @practice_cactus 
 { "some_event": "payload", "published_at": "2016-03-16T16:35:04Z"

    }
  20. stella cotton | @practice_cactus

  21. stella cotton | @practice_cactus • zendesk/ruby-kafka • DeliveryBoy + Racecar

    • Phobos
  22. stella cotton | @practice_cactus • Karafka • WaterDrop

  23. stella cotton | @practice_cactus • zendesk/ruby-kafka • DeliveryBoy + Racecar

    • Phobos
  24. stella cotton | @practice_cactus gem 'delivery_boy' $ bundle $ bundle

    exec rails generate delivery_boy:install
  25. stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<:

    *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092
  26. stella cotton | @practice_cactus class OrdersController < ApplicationController def create

    @comment = Order.create!(params) DeliveryBoy.deliver_async(order.to_json, topic: "user_event") end end
  27. stella cotton | @practice_cactus

  28. stella cotton | @practice_cactus

  29. stella cotton | @practice_cactus

  30. stella cotton | @practice_cactus class OrdersController < ApplicationController def create

    @comment = Order.create!(params) DeliveryBoy.deliver_async( order.to_json, topic: "user_event", partition_key: user.id ) end end
  31. stella cotton | @practice_cactus gem 'racecar' $ bundle $ bundle

    exec rails generate racecar:install
  32. stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<:

    *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092
  33. stella cotton | @practice_cactus class UserEventConsumer < Racecar::Consumer subscribes_to "user_event"

    def process(message) data = JSON.parse(message.value) puts data end end
  34. stella cotton | @practice_cactus 
 
 bundle exec racecar UserEventConsumer

  35. stella cotton | @practice_cactus

  36. stella cotton | @practice_cactus

  37. stella cotton | @practice_cactus 
 
 Offset

  38. stella cotton | @practice_cactus

  39. stella cotton | @practice_cactus Why Kafka?

  40. stella cotton | @practice_cactus 
 
 High Throughput

  41. stella cotton | @practice_cactus 
 
 Tracking Consumer Position

  42. stella cotton | @practice_cactus 
 
 Consumer Agreement is Tricky

  43. stella cotton | @practice_cactus 
 
 Reading and Writing is

    O(1)
  44. stella cotton | @practice_cactus 
 
 Scalable and Fault Tolerant

  45. stella cotton | @practice_cactus 
 
 Thorough Introduction to Apache

    Kafka https://hackernoon.com/thorough-introduction-to-apache-kafka-6fbf2989bbc1
 Stanislav Kozlovski
  46. stella cotton | @practice_cactus 
 
 1,000,000,000,000

  47. stella cotton | @practice_cactus Kafka Powered Services

  48. stella cotton | @practice_cactus Fault Tolerant Replacement for RPC

  49. stella cotton | @practice_cactus def create_order create_order_record charge_credit_card send_confirmation_email end

  50. stella cotton | @practice_cactus

  51. stella cotton | @practice_cactus 
 Challenge 1:
 Upstream service is

    responsible for downstream availability
  52. stella cotton | @practice_cactus

  53. stella cotton | @practice_cactus 
 Challenge 2:
 New downstream services

    require changes to the 
 upstream service
  54. stella cotton | @practice_cactus

  55. stella cotton | @practice_cactus 
 
 There’s No Such Thing

    
 as a Free Lunch
  56. stella cotton | @practice_cactus 
 New Challenge:
 Dependencies 
 are

    no longer explicit
  57. stella cotton | @practice_cactus Architectural Patterns

  58. stella cotton | @practice_cactus 
 The Many Meanings of 


    Event-Driven Architecture
 
 Martin Fowler - GOTO Chicago
 https://www.youtube.com/watch?v=STKCRSUsyP0
 https://martinfowler.com/articles/201701-event-driven.html

  59. stella cotton | @practice_cactus 
 
 Event Notification

  60. stella cotton | @practice_cactus

  61. stella cotton | @practice_cactus

  62. stella cotton | @practice_cactus 
 
 Event Carried State Transfer

  63. stella cotton | @practice_cactus

  64. stella cotton | @practice_cactus

  65. stella cotton | @practice_cactus

  66. stella cotton | @practice_cactus 
 
 Event-Sourced Architecture

  67. stella cotton | @practice_cactus 
 
 Challenge 1: 
 Code

    Changes
  68. stella cotton | @practice_cactus 
 
 Challenge 2: 
 Third-Party

    Integrations
  69. stella cotton | @practice_cactus 
 
 Command Query 
 Responsibility

    Segregation (CQRS)
  70. stella cotton | @practice_cactus 
 
 CRUD vs CQRS

  71. stella cotton | @practice_cactus

  72. stella cotton | @practice_cactus

  73. stella cotton | @practice_cactus 
 
 When to use it?

  74. stella cotton | @practice_cactus 
 When to Use 
 (and

    Not to Use) CQRS
 
 Grzegorz Sitkowski
 http://blog.objectivity.co.uk/when-to-use-and-not-to-use-cqrs/
  75. stella cotton | @practice_cactus 
 The Many Meanings of 


    Event-Driven Architecture
 
 Martin Fowler - GOTO Chicago
 https://www.youtube.com/watch?v=STKCRSUsyP0
 https://martinfowler.com/articles/201701-event-driven.html

  76. stella cotton | @practice_cactus Practical Considerations

  77. stella cotton | @practice_cactus Slow Consumers

  78. stella cotton | @practice_cactus 
 
 Increase Consumers in 


    the Consumer Group
  79. stella cotton | @practice_cactus 
 
 process-user-events: bundle exec racecar

    UserEventConsumer
  80. stella cotton | @practice_cactus 
 
 process-user-events-1: bundle exec racecar

    UserEventConsumer
 process-user-events-2: bundle exec racecar UserEventConsumer
 process-user-events-3: bundle exec racecar UserEventConsumer
  81. stella cotton | @practice_cactus

  82. stella cotton | @practice_cactus 
 
 Sidekiq

  83. stella cotton | @practice_cactus 
 
 Monitor and Alert on

    
 Consumer Lag
  84. stella cotton | @practice_cactus https://github.com/zendesk/ ruby-kafka#what-to-monitor

  85. stella cotton | @practice_cactus 
 
 Gameday Scenarios

  86. stella cotton | @practice_cactus 
 
 “exactly once” vs “at

    least once” https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
  87. stella cotton | @practice_cactus 
 
 Design Consumers for Duplicated

    Events
  88. stella cotton | @practice_cactus 
 
 1. Rely on UPSERT


    2. Unique identifiers for events
  89. stella cotton | @practice_cactus Schemas and Data Serialization

  90. stella cotton | @practice_cactus 
 
 Downside to Data Flexibility

  91. stella cotton | @practice_cactus 
 
 Schema Registries

  92. stella cotton | @practice_cactus 
 
 JSON

  93. stella cotton | @practice_cactus Pros: • Human readable • Universal

    language support
  94. stella cotton | @practice_cactus Cons: • Payload can be large

    due to repeated keys • No built-in documentation, aliases
  95. stella cotton | @practice_cactus 
 
 Avro

  96. stella cotton | @practice_cactus Pros: • Robust schema support with

    types • Very compact and very fast • Built in documentation • Helps you evolve data over time
  97. stella cotton | @practice_cactus 
 
 Avro::Builder https://www.salsify.com/blog/engineering/adventures-in-avro

  98. stella cotton | @practice_cactus Illustrations by Freepik Beyond 50,000 Partitions:

    How Heroku Operates and Pushes the Limits of Kafka at Scale
 http://www.dataengconf.com/beyond-50k- partitions-how-heroku-pushes-the-limits-of-kafka-at- scale
 
 Dogfooding Kafka: How We Built Heroku's Real- Time Platform Event Stream
 https://www.salesforce.com/video/302281/
 

  99. stella cotton | @practice_cactus Illustrations by Freepik Postgres 10, Performance,

    and You
 Gabe Enslein
 Room 306-307
 

  100. stella cotton | @practice_cactus Illustrations by Freepik https://sfdc.co/herokujobs

  101. stella cotton | @practice_cactus Thank You!