From Chaos to Clarity: Structured Event Reporting in Rails

Events in Rails applications are like the heartbeat of your code - whether it’s a log, a telemetry signal, or a business event, they tell us when something interesting is happening. To truly harness he power of these events for observability and data analysis, we need high-quality, contextualized data. The human-readable lines that Rails.logger provides is great for manual inspection but falls short in production and analytics contexts.

At Shopify, we recognized the need for a unified approach to events. After years of managing various in-house solutions for structured logging, we built support for structured events into the Rails framework itself. This talk will unveil the Structured Event Reporter, Rails’ new approach to structured events, and showcase how we’re using it to power events in our monolith.

Transcript

1. Adrianna Chang - Rails World 2025 Structured Event Reporting in Rails From Chaos to Clarity
3. Hi, I ’ m Adrianna! • Ruby on R a ils since 2016 • R a ils Infr a structure • St aff Engineer a t Shopify
4. 📷 @tenderlove
6. Structured Events Av a il a ble in R a ils 8.1!
7. Structured Events 1. Wh a t a re Events? 2. A Brief History of Events a t Shopify 3. Structured Event Reporter in R a ils 4.The Future of Structured Events in R a ils Agenda
8. What is an event? Contextu a l — “wh a t h a ppened”, “when”, a nd often “where” a nd “why”. A n a med record of something th a t h a ppened in a softw a re a pplic a tion.
9. What is an event? Sem a ntic — e a sily p a rsed to extr a ct common context. A n a med record of something th a t h a ppened in a softw a re a pplic a tion. OpenTelemetry Sem a ntic Conventions
10. What is an event? A n a med record of something th a t h a ppened in a softw a re a pplic a tion. Log: Noteworthy occurrence
11. What is an event? Metric A n a med record of something th a t h a ppened in a softw a re a pplic a tion. Cre a teOrder Span User Action: “Pl a ce Order” Cre a teOrder T R A C E Trace
12. What is an event? A n a med record of something th a t h a ppened in a softw a re a pplic a tion. Cre a teOrder User Action: “Pl a ce Order” Cre a teOrder T R A C E
13. “Business Events” What is an event? A n a med record of something th a t h a ppened in a softw a re a pplic a tion.
14. “Business Events” What is an event?
15. Unstructured vs Structured Events
16. Unstructured vs Structured Events
17. Unstructured vs Structured Events
18. A (brief) History REDACTED REDACTED REDACTED
19. REDA | rex "Processing by (?<controller>[^#]+)#(?<action>\S+) as" | rex "Completed (?<http_status>\d+) (?<http_status_message>.+) in (? <duration_ms>\d+)ms" | eval controller_action = controller + "#" + action | rex "\"X-Platform-Api-Id\"=>\"(?<platform_api>\d+)\"" | eval platform_api = if(isnull(platform_api), 0, platform_api) | search platform_api=$platform_api_id$ | search controller=$controller$ | search action=$action$ | search http_status=$http_status$ | where http_status >= $http_status_threshold$ | chart count over controller by http_status A (brief) History
20. A (brief) History Request Log
21. A (brief) History Shopify Monolith Rails.logger.info("[ClassA] A really cool log key=value”) Rails.logger.info("[ClassB] Another log key=value”) Rails.logger.info("[ClassC] Wow I’m also a log key=value”)
22. A (brief) History Shopify Monolith Rails.logger.info("[ClassA] A really cool log key=value”) Rails.logger.info("[ClassB] Another log key=value”) Rails.logger.info("[ClassC] Wow I’m also a log key=value”)
23. A (brief) History “Business Events” Monorail
24. schem a s/merch a nt_protection_shop_close/2.2.yml A (brief) History Monorail
25. A (brief) History schem a s/merch a nt_protection_shop_close/2.2.yml
26. Shopify Monolith A (brief) History
27. Events at Shopify • Develop a uni f ied API for a ll event d a t a in R a ils. • M a ke it f a ster for developers to f ind wh a t they need. • Improve qu a lity of event d a t a . Enforce structure a nd consistency. Goals
28. Events in Rails • API supports di ff erent requirements for how a n event’s d a t a structure is de f ined + enforced. • Support for implicit a nd explicit schem a s Criteria Explicit schem a Implicit schem a
29. Events in Rails • Flexible enough to work with a ny observ a bility st a ck or d a t a pipeline. No a ssumptions a bout how d a t a should be encoded or exported. Criteria • API supports di ff erent requirements for how a n event’s d a t a structure is de f ined + enforced. • Support for implicit a nd explicit schem a s
30. Events in Rails • M a ke it e a sy to a nnot a te events with a ll relev a nt context. Criteria • API supports di ff erent requirements for how a n event’s d a t a structure is de f ined + enforced. • Support for implicit a nd explicit schem a s • Flexible enough to work with a ny observ a bility st a ck or d a t a pipeline. No a ssumptions a bout how d a t a should be encoded or exported.
31. Structured Event Reporter
32. Structured Event Reporter “Provides an interface for reporting structured events to subscribers.”
33. How is this different from Active Support Noti fi cations?
34. Structured Event Reporter Rails.event Structured Event Reporter
35. Structured Event Reporter Rails.event.subscribe Structured Event Reporter Subscribing
36. Structured Event Reporter Subscribing class LogSubscriber def emit(event) # Process event end end Log Subscriber log_subscriber = LogSubscriber.new Rails.event.subscribe(log_subscriber)
37. Structured Event Reporter Subscribing Rails.event.subscribe(subscriber) { |event| event[:payload].is_a?(LogEvent) } Audit Event Subscriber Rails.event.subscribe(subscriber) { |event| event[:name].start_with?(“audit_event.”) } Log Subscriber
38. Structured Event Reporter Rails.event.notify( “user.created”, id: 123) Structured Event Reporter Notifying
39. Structured Event Reporter Rails.event.notify( “user.created”, id: 123) Structured Event Reporter Notifying Event Hash subscriber.emit(event)
40. Structured Event Reporter Subscribers Log Subscriber
41. Structured Event Reporter • V a lid a te the d a t a , encode a nd export it • Aggreg a te events into a request log • B a tch events to export them out-of-b a nd • H a ndle encoding f a ilures a nd send events to a De a d- Letter-Queue Subscribers
42. Structured Event Reporter Subscribers Flexible enough to work with any observability stack or data pipeline. • V a • Aggreg a • B a • H a
43. Structured Event Reporter Event Objects Support for implicit and explicit event schemas.
44. Structured Event Reporter Event Objects Support for implicit and explicit event schemas. def emit(event) event_object = event[:payload] event_data = event_object.serialize end
45. Structured Event Reporter proto/shopify/merch a nt_ a ctivity/shop_pl a n_ch a nge.proto Event Objects at Shopify
46. Structured Event Reporter proto/shopify/merch a nt_ a ctivity/shop_pl a n_ch a nge.proto Rails.event.notify( Proto::Shopify::MerchantActivity:: ShopPlanChangeEvent.new(…)) Event Objects at Shopify
47. Structured Event Reporter Rails.event.notify( Proto::Shopify::MerchantActivity:: ShopPlanChangeEvent.new(…)) Event Objects at Shopify Structured Event Reporter
48. Structured Event Reporter Contextual Events Make it easy to annotate events with all relevant context. tag1: a tag4: d tag2: b tag3: c Tags Context Store
49. Structured Event Reporter Contextual Events: Tags
50. Structured Event Reporter Contextual Events: Tags
51. Structured Event Reporter Contextual Events: Tags
52. Structured Event Reporter Contextual Events: Tags Tags can have explicit schemas too
53. Structured Event Reporter Contextual Events: Context Store
54. Structured Event Reporter Contextual Events: Context Store
55. Structured Event Reporter Contextual Events: Context Store
56. Structured Event Reporter Contextual Events: Context Store
57. Structured Event Reporter Summary • Subscribing (Rails.event.subscribe) • Reporting Events (Rails.event.notify) • Subscriber interf a ce (#emit) a nd ex a mple subscriber • Event objects • Att a ching context to events • T a gs (Rails.event.tagged) • Context Store (Rails.event.set_context)
58. • Develop a uni f ied API for a ll event d a t a in R a ils. • M a ke it f a ster for developers to f ind wh a t they need. • Improve qu a lity of event d a t a . Enforce structure a nd consistency. Structured Event Reporter Summary
59. The Future of Events in Rails
60. The Future of Events in Rails
61. The Future of Events in Rails
62. The Future of Events in Rails ActiveRecord::LogSubscriber “sql.active_record” noti fi cation Rails.logger.info “#{event.payload[:name]} (#{event.duration}) #{event.payload[:sql]}” ActiveRecord::EventSubscriber Rails.event.notify( event.payload[:name], duration: event.duration, sql: event.payload[:sql] )
63. The Future of Events in Rails PaaS examples: Heroku, fl y.io, Render
64. The Future of Events in Rails “Rails Logger” Subscriber json_event = JSON.dump(event) Rails.logger.info(json_event) # => "{\"name\":\"user.created\",\"payload\":{\"id\":123},\"tags\":{},\"context\":{}, \"timestamp\":1756145476586165000,\"source_location\":{\"filepath\":\"app/controllers/ users_controller.rb\",\"lineno\":20,\"label\":\"UsersController#create\"}}"
65. The Future of Events in Rails human_readable_log = format(event) Rails.logger.info(human_readable_log) “Development” Subscriber # => 2024-01-15T10:35:22.128901 [EVENT] user.created id=123 @ users_controller.rb:20
66. 8.1
68. https://x.com/ a dri a nn a kch a ng https://r a ils a tsc a le.com/
69. Attributions <a href="https://www. fl aticon.com/free-icons/login" title="login icons">Login icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/ecommerce" title="ecommerce icons">Ecommerce icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/engineer" title="engineer icons">Engineer icons created by smashingstocks - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/timestamp" title="timestamp icons">Timestamp icons created by Icon home - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/database" title="database icons">Database icons created by phatplus - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/log" title="log icons">Log icons created by juicy_ fi sh - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/application" title="application icons">Application icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/semantic-web" title="semantic web icons">Semantic web icons created by Vectors Tank - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/monorail" title="monorail icons">Monorail icons created by surang - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/ab-testing" title="ab testing icons">Ab testing icons created by PixelVerse - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/development" title="development icons">Development icons created by Uniconlabs - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/document" title="document icons">Document icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/byte" title="byte icons">Byte icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/price" title="price icons">Price icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/http" title="http icons">Http icons created by kerismaker - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/pipeline" title="pipeline icons">Pipeline icons created by Uniconlabs - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/thank-you" title="thank-you icons">Thank-you icons created by rsetiawan - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/tweet" title="tweet icons">Tweet icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/blog" title="blog icons">Blog icons created by Freepik - Flaticon</a> <a href="https://www. fl aticon.com/free-icons/server" title="server icons">Server icons created by smashingstocks - Flaticon</a>