Fix Production Bugs 20x Faster - Wroclave.rb - 2025

Production bugs aren’t just annoying — they’re often invisible.

This talk tells the true story of how structured logging in Rails exposed a flood of invisible problems — and led to an 83% drop in 500 errors and 98% less downtime.

You’ll learn:
* What to log (and how) using semantic_logger
* How to spot job queue issues instantly
* Why plain text logs just aren’t enough

If you're tired of being blind in prod, come see how better logging can translate to faster fixes.

Transcript

1. Fix Production Bugs 20x Faster The Power Of Structured Logging In Rails
3. Based on a true story.
4. Based on a true story. * Names have been changed to protect the innocent
29. Two Months Later
39. Vicious Cycle
41. How often do you encounter issues where you lack visibility into your application's performance or errors?
42. How often do you encounter issues where you lack visibility into your application's performance or errors?
48. The problem isn't you. You're not alone.
52. You Want Answered Question
53. Why are password reset emails not being sent? You Want Answered Question
54. Failing? Timed Out? Delayed? You Want Answered Question
55. Failing? Timed Out? Delayed? You Want Answered Question
56. Failing? Timed Out? Delayed? You Want Answered Question
57. You Want Answered Question
58. You Want Answered Question
59. Which jobs are taking time in within_five_minutes queue? You Want Answered Question
60. Data To Collect Decide
61. Event Filter By Group By Values Of Data To Collect Decide
62. Event Filter By Group By Values Of Job Performed Data To Collect Decide
63. Event Filter By Group By Values Of Job Performed Job Queue Data To Collect Decide
64. Event Filter By Group By Values Of Job Performed Job Queue Job Class Data To Collect Decide
65. Event Filter By Group By Values Of Job Performed Job Queue Job Class Duration Data To Collect Decide
66. Data To Collect Decide
67. Instrumentation Build
68. Instrumentation Build 1. Data type (Traces, Logs, Metrics) 2. Plain Text vs Structured 3. Logging Library 4. Customise
69. Instrumentation Build Traces vs Logs vs Metrics
70. Instrumentation Build Traces vs Logs vs Metrics
71. Instrumentation Build Plain Text Logs vs Structured Logs
72. Instrumentation Build Plain Text Logs vs Structured Logs
73. Plain Text Logs in Code Instrumentation Build
74. Plain Text Logs in UI Instrumentation Build
75. Plain Text Logs in UI Instrumentation Build
76. Plain Text Logs in UI Instrumentation Build ❌
77. Structured Logs in Code Instrumentation Build
78. Structured Logs in UI Instrumentation Build
79. Structured Logs in UI Instrumentation Build
80. Structured Logs in UI Instrumentation Build ✅
81. Instrumentation Build Logging Library
82. Instrumentation Build • Log structured payload • Integrates with Rails • Great documentation • Mature Criteria
83. Instrumentation Build Library Payload Rails Docs Mature fluentd lograge logging contextual_logger semantic_logger dry-logger Thank you to all authors of these gems for your hard work!
84. Instrumentation Build Library Payload Rails Docs Mature fluentd ❌ ✅ ❌ ✅ lograge ❌ ✅ ❌ ✅ logging ❌ ❌ ❌ ❌ contextual_logger ✅ ✅ ❌ ❌ semantic_logger ✅ ✅ ✅ ✅ dry-logger ✅ ❌ ✅ ❌ Thank you to all authors of these gems for your hard work!
85. Instrumentation Build Library Payload Rails Docs Mature fluentd ❌ ✅ ❌ ✅ lograge ❌ ✅ ❌ ✅ logging ❌ ❌ ❌ ❌ contextual_logger ✅ ✅ ❌ ❌ semantic_logger ✅ ✅ ✅ ✅ dry-logger ✅ ❌ ✅ ❌ Thank you to all authors of these gems for your hard work!
86. Instrumentation Build Semantic Logger
87. Instrumentation Build Format As JSON
88. Instrumentation Build Automatic Logging
89. Instrumentation Build • ActionController • ActionMailer • ActionView • ActiveRecord • ActiveJob Supported Libraries
90. Instrumentation Build ActiveJob
91. Instrumentation Build ActiveJob
92. Instrumentation Build ActiveJob
93. Instrumentation Build ActiveJob
94. Instrumentation Build
95. 1. No conventions 2. Missing attributes 3. Missing API requests Many more... Instrumentation Build Weaknesses
96. Instrumentation Build 1. No Conventions
97. Instrumentation Build job_class job.class active_job_class message.class job_class_name job.class.name
98. Instrumentation Build OpenTelemetry
99. Instrumentation Build https://go.joyfulprogramming.com/otel-msg
100. Instrumentation Build https://go.joyfulprogramming.com/otel-msg
101. Instrumentation Build
102. Instrumentation Build
103. Instrumentation Build
104. Instrumentation Build
105. Instrumentation Build
106. Instrumentation Build
107. Instrumentation Build
108. Instrumentation Build 2. Missing Attributes
109. Request Headers Body ID User Agent Instrumentation Build Response Headers Body
110. Request Headers Body ID User Agent Instrumentation Build Response Headers Body
111. Instrumentation Build
112. Instrumentation Build
113. Instrumentation Build
114. Instrumentation Build
115. Instrumentation Build
116. Instrumentation Build 3. Missing API requests
117. Instrumentation Build
118. Instrumentation Build
119. Instrumentation Build
120. Instrumentation Build
121. 1. No conventions 2. Missing attributes 3. Missing API requests Instrumentation Build Weaknesses ✅ ✅ ✅
122. Instrumentation Build Send To Observability Tool
123. Instrumentation Build
125. Instrumentation Build Phew!
126. Graphs Use
127. Graphs Use Event Filter By Group By Plot Values Of Job Performed Job Queue Job Class Duration
128. Graphs Use
129. Graphs Use
130. Graphs Use
131. Graphs Use
132. Graphs Use
133. Improve
134. Improve Criticise. Show. Learn.
135. Improve Job Latency? Request ID? IP?
141. Question Which jobs are taking the most time?
149. Answer Analytics::UpdateUserVisitsJob
150. Question What's enqueuing Analytics::UpdateUserVisitsJob?
158. Answer profiles#show
159. Question What IP is hitting this action?
167. Answer 15.145.1.3
168. Root Cause Scraper at 15.145.1.3 Hits profiles#show Floods queue with Analytics job Delays password reset emails
169. Fix Block IP in infrastructure
170. Check Is it fixed?
173. Done!
178. Results
179. Downtime
180. 98% reduction Downtime
181. 500 Errors
182. 83% reduction 500 Errors
184. Fix Bugs 20x Faster
185. Extra Resources https://linktr.ee/joyfulprogramming Playbook (Free) Newsletter (Free) Sidekiq Course ($1)
186. John Gallagher Thank you! biggerpockets.com/careers joyfulprogramming.com
187. https://linktr.ee/joyfulprogramming biggerpockets.com/careers joyfulprogramming.com