So You’ve Got Yourself a Kafka: Event-Powered Rails Services

It’s easier than ever to integrate a distributed commit log system like Kafka into your stack. But once you have it, what do you do with it? Come learn the basics about Kafka and event-driven systems and how you can use them to power your Rails services. We’ll also venture beyond the theoretical to talk about practical considerations and unusual operational challenges that your event-driven Rails services might face, like monitoring, queue processing, and scaling slow consumers.

Transcript

1. So You’ve Got Yourself a Kafka
2. stella cotton | @practice_cactus
3. stella cotton | @practice_cactus
4. stella cotton | @practice_cactus
5. stella cotton | @practice_cactus • What is Kafka • Kafka Powered Services • Practical Considerations and Challenges
6. stella cotton | @practice_cactus What is Kafka?
7. stella cotton | @practice_cactus Distributed Streaming Platform
8. stella cotton | @practice_cactus
9. stella cotton | @practice_cactus 
 
 Append Only Log
10. stella cotton | @practice_cactus 010-09-16T15:13:46.677020+00:00 app[web.1]: Processing PostController#list … 2010-09-16T15:13:46.677023+00:00 app[web.1]: Rendering template within layouts/application 2010-09-16T15:13:46.677902+00:00 app[web.1]: Rendering post/list 2010-09-16T15:13:46.678990+00:00 app[web.1]: Rendered includes/_header (0.1ms) 2010-09-16T15:13:46.698234+00:00 app[web.1]: Completed in 74ms (View: 31, DB: 40) | 200 OK 2010-09-16T15:13:46.723498+00:00 heroku[router]: at=info method=GET path=“/posts”… 2010-09-16T15:13:47.893472+00:00 app[worker.1]: 2 jobs processed at 16.6761 j/s, 0 failed ...
11. stella cotton | @practice_cactus 
 
 Producers
12. stella cotton | @practice_cactus
13. stella cotton | @practice_cactus
14. stella cotton | @practice_cactus 
 
 Consumers
15. stella cotton | @practice_cactus
16. stella cotton | @practice_cactus
17. stella cotton | @practice_cactus 
 
 How is this different from Sidekiq or Resque?
18. stella cotton | @practice_cactus 
 
 user_event Topic
19. stella cotton | @practice_cactus 
 { "some_event": "payload", "published_at": "2016-03-16T16:35:04Z" }
20. stella cotton | @practice_cactus
21. stella cotton | @practice_cactus • zendesk/ruby-kafka • DeliveryBoy + Racecar • Phobos
22. stella cotton | @practice_cactus • Karafka • WaterDrop
23. stella cotton | @practice_cactus • zendesk/ruby-kafka • DeliveryBoy + Racecar • Phobos
24. stella cotton | @practice_cactus gem 'delivery_boy' $ bundle $ bundle exec rails generate delivery_boy:install
25. stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<: *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092
26. stella cotton | @practice_cactus class OrdersController < ApplicationController def create @comment = Order.create!(params) DeliveryBoy.deliver_async(order.to_json, topic: "user_event") end end
27. stella cotton | @practice_cactus
28. stella cotton | @practice_cactus
29. stella cotton | @practice_cactus
30. stella cotton | @practice_cactus class OrdersController < ApplicationController def create @comment = Order.create!(params) DeliveryBoy.deliver_async( order.to_json, topic: "user_event", partition_key: user.id ) end end
31. stella cotton | @practice_cactus gem 'racecar' $ bundle $ bundle exec rails generate racecar:install
32. stella cotton | @practice_cactus common: &common client_id: "my_app" development: <<: *common brokers: - localhost:9092 test: <<: *common brokers: - localhost:9092 production: <<: *common brokers: - kafka1.myapp.com:9092 - kafka2.myapp.com:9092 - kafka3.myapp.com:9092
33. stella cotton | @practice_cactus class UserEventConsumer < Racecar::Consumer subscribes_to "user_event" def process(message) data = JSON.parse(message.value) puts data end end
34. stella cotton | @practice_cactus 
 
 bundle exec racecar UserEventConsumer
35. stella cotton | @practice_cactus
36. stella cotton | @practice_cactus
37. stella cotton | @practice_cactus 
 
 Offset
38. stella cotton | @practice_cactus
39. stella cotton | @practice_cactus Why Kafka?
40. stella cotton | @practice_cactus 
 
 High Throughput
41. stella cotton | @practice_cactus 
 
 Tracking Consumer Position
42. stella cotton | @practice_cactus 
 
 Consumer Agreement is Tricky
43. stella cotton | @practice_cactus 
 
 Reading and Writing is O(1)
44. stella cotton | @practice_cactus 
 
 Scalable and Fault Tolerant
45. stella cotton | @practice_cactus 
 
 Thorough Introduction to Apache Kafka https://hackernoon.com/thorough-introduction-to-apache-kafka-6fbf2989bbc1
 Stanislav Kozlovski
46. stella cotton | @practice_cactus 
 
 1,000,000,000,000
47. stella cotton | @practice_cactus Kafka Powered Services
48. stella cotton | @practice_cactus Fault Tolerant Replacement for RPC
49. stella cotton | @practice_cactus def create_order create_order_record charge_credit_card send_confirmation_email end
50. stella cotton | @practice_cactus
51. stella cotton | @practice_cactus 
 Challenge 1:
 Upstream service is responsible for downstream availability
52. stella cotton | @practice_cactus
53. stella cotton | @practice_cactus 
 Challenge 2:
 New downstream services require changes to the 
 upstream service
54. stella cotton | @practice_cactus
55. stella cotton | @practice_cactus 
 
 There’s No Such Thing 
 as a Free Lunch
56. stella cotton | @practice_cactus 
 New Challenge:
 Dependencies 
 are no longer explicit
57. stella cotton | @practice_cactus Architectural Patterns
58. stella cotton | @practice_cactus 
 The Many Meanings of 
 Event-Driven Architecture
 
 Martin Fowler - GOTO Chicago
 https://www.youtube.com/watch?v=STKCRSUsyP0
 https://martinfowler.com/articles/201701-event-driven.html

59. stella cotton | @practice_cactus 
 
 Event Notification
60. stella cotton | @practice_cactus
61. stella cotton | @practice_cactus
62. stella cotton | @practice_cactus 
 
 Event Carried State Transfer
63. stella cotton | @practice_cactus
64. stella cotton | @practice_cactus
65. stella cotton | @practice_cactus
66. stella cotton | @practice_cactus 
 
 Event-Sourced Architecture
67. stella cotton | @practice_cactus 
 
 Challenge 1: 
 Code Changes
68. stella cotton | @practice_cactus 
 
 Challenge 2: 
 Third-Party Integrations
69. stella cotton | @practice_cactus 
 
 Command Query 
 Responsibility Segregation (CQRS)
70. stella cotton | @practice_cactus 
 
 CRUD vs CQRS
71. stella cotton | @practice_cactus
72. stella cotton | @practice_cactus
73. stella cotton | @practice_cactus 
 
 When to use it?
74. stella cotton | @practice_cactus 
 When to Use 
 (and Not to Use) CQRS
 
 Grzegorz Sitkowski
 http://blog.objectivity.co.uk/when-to-use-and-not-to-use-cqrs/
75. stella cotton | @practice_cactus 
 The Many Meanings of 
 Event-Driven Architecture
 
 Martin Fowler - GOTO Chicago
 https://www.youtube.com/watch?v=STKCRSUsyP0
 https://martinfowler.com/articles/201701-event-driven.html

76. stella cotton | @practice_cactus Practical Considerations
77. stella cotton | @practice_cactus Slow Consumers
78. stella cotton | @practice_cactus 
 
 Increase Consumers in 
 the Consumer Group
79. stella cotton | @practice_cactus 
 
 process-user-events: bundle exec racecar UserEventConsumer
80. stella cotton | @practice_cactus 
 
 process-user-events-1: bundle exec racecar UserEventConsumer
 process-user-events-2: bundle exec racecar UserEventConsumer
 process-user-events-3: bundle exec racecar UserEventConsumer
81. stella cotton | @practice_cactus
82. stella cotton | @practice_cactus 
 
 Sidekiq
83. stella cotton | @practice_cactus 
 
 Monitor and Alert on 
 Consumer Lag
84. stella cotton | @practice_cactus https://github.com/zendesk/ ruby-kafka#what-to-monitor
85. stella cotton | @practice_cactus 
 
 Gameday Scenarios
86. stella cotton | @practice_cactus 
 
 “exactly once” vs “at least once” https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
87. stella cotton | @practice_cactus 
 
 Design Consumers for Duplicated Events
88. stella cotton | @practice_cactus 
 
 1. Rely on UPSERT
 2. Unique identifiers for events
89. stella cotton | @practice_cactus Schemas and Data Serialization
90. stella cotton | @practice_cactus 
 
 Downside to Data Flexibility
91. stella cotton | @practice_cactus 
 
 Schema Registries
92. stella cotton | @practice_cactus 
 
 JSON
93. stella cotton | @practice_cactus Pros: • Human readable • Universal language support
94. stella cotton | @practice_cactus Cons: • Payload can be large due to repeated keys • No built-in documentation, aliases
95. stella cotton | @practice_cactus 
 
 Avro
96. stella cotton | @practice_cactus Pros: • Robust schema support with types • Very compact and very fast • Built in documentation • Helps you evolve data over time
97. stella cotton | @practice_cactus 
 
 Avro::Builder https://www.salsify.com/blog/engineering/adventures-in-avro
98. stella cotton | @practice_cactus Illustrations by Freepik Beyond 50,000 Partitions: How Heroku Operates and Pushes the Limits of Kafka at Scale
 http://www.dataengconf.com/beyond-50k- partitions-how-heroku-pushes-the-limits-of-kafka-at- scale
 
 Dogfooding Kafka: How We Built Heroku's Real- Time Platform Event Stream
 https://www.salesforce.com/video/302281/
 

99. stella cotton | @practice_cactus Illustrations by Freepik Postgres 10, Performance, and You
 Gabe Enslein
 Room 306-307
 

100. stella cotton | @practice_cactus Illustrations by Freepik https://sfdc.co/herokujobs
101. stella cotton | @practice_cactus Thank You!