Fix Production Bugs Quickly - Warsaw Ruby Conference 2024

In the presentation "Fix Production Bugs Quickly - The Power of Structured Logging in Rails," John Gallagher discusses enhancing observability in Rails applications to expedite the resolution of production bugs. The talk highlights the challenges engineers face when troubleshooting issues, driven by an anecdote of a critical production incident involving password reset emails at his company, BiggerPockets.

John underscores the importance of structured logging in Rails 8, which will significantly improve logging capabilities compared to traditional text-based logs. He outlines five critical steps for implementing structured logging:

- **Identify the Problem**: Defining what data needs to be collected to address specific questions, such as identifying delays in job processing.
- **Data Collection Criteria**: Gallagher emphasizes the importance of determining the type of data (e.g., job duration, event types), stressing that visibility into application performance is crucial.
- **Implementing Structured Logging**: He compares traditional logging to structured logging, advocating for structured formats that facilitate better searching and filtering.
- **Choosing a Logging Library**: Gallagher recommends using Semantic Logger due to its robust features and compatibility with Rails, providing logged data in a structured JSON format.
- **Sending Data to Monitoring Tools**: He details the process of configuring logs to integrate with observability tools like Rollbar and Honeybadger, illustrating how to utilize logs for proactive error tracking.

Throughout the talk, Gallagher reflects on the emotional aspects of engineering—frustration, embarrassment, and self-doubt—that arise from unresolved bugs and system outages. He encourages developers to view their struggles with observability as common challenges rather than personal failures. Conclusively, the session advocates not only for technical solutions to improve logging but also for a mindset shift towards proactive observability that can prevent future issues before they impact users.

Transcript

1. Fix Production Bugs Quickly The Power Of Structured Logging In Rails
3. Based on a true story.
4. Based on a true story. * Names have been changed to protect the innocent
26. Two Months Later
36. Vicious Cycle
38. How often do you encounter issues where you lack visibility into your application's performance or errors?
39. How often do you encounter issues where you lack visibility into your application's performance or errors?
45. The problem isn't you. You're not alone.
49. You Want Answered Question
50. Why are password reset emails not being sent? You Want Answered Question
51. Failing? Timed Out? Delayed? You Want Answered Question
52. Failing? Timed Out? Delayed? You Want Answered Question
53. Failing? Timed Out? Delayed? You Want Answered Question
54. Which jobs are taking time in within_five_minutes queue? You Want Answered Question
55. Data To Collect Decide
56. Event Filter By Group By Values Of Data To Collect Decide
57. Event Filter By Group By Values Of Job Performed Data To Collect Decide
58. Event Filter By Group By Values Of Job Performed Job Queue Data To Collect Decide
59. Event Filter By Group By Values Of Job Performed Job Queue Job Class Data To Collect Decide
60. Event Filter By Group By Values Of Job Performed Job Queue Job Class Duration Data To Collect Decide
61. Instrumentation Build
62. Instrumentation Build 1. Data type (Traces, Logs, Metrics) 2. Plain Text vs Structured 3. Logging Library 4. Customise
63. Instrumentation Build Traces vs Logs vs Metrics
64. Instrumentation Build Traces vs Logs vs Metrics
65. Instrumentation Build Plain Text Logs vs Structured Logs
66. Instrumentation Build Plain Text Logs vs Structured Logs
67. Plain Text Logs in Code Instrumentation Build
68. Plain Text Logs in UI Instrumentation Build
69. Plain Text Logs in UI Instrumentation Build
70. Plain Text Logs in UI Instrumentation Build ❌
71. Structured Logs in Code Instrumentation Build
72. Structured Logs in UI Instrumentation Build
73. Structured Logs in UI Instrumentation Build
74. Structured Logs in UI Instrumentation Build ✅
75. Instrumentation Build Logging Library
76. Instrumentation Build • Log structured payload • Integrates with Rails • Great documentation • Mature Criteria
77. Instrumentation Build Library Payload Rails Docs Mature fluentd lograge logging contextual_logger semantic_logger dry-logger Thank you to all authors of these gems for your hard work!
78. Instrumentation Build Library Payload Rails Docs Mature fluentd ❌ ✅ ❌ ✅ lograge ❌ ✅ ❌ ✅ logging ❌ ❌ ❌ ❌ contextual_logger ✅ ✅ ❌ ❌ semantic_logger ✅ ✅ ✅ ✅ dry-logger ✅ ❌ ✅ ❌ Thank you to all authors of these gems for your hard work!
79. Instrumentation Build Library Payload Rails Docs Mature fluentd ❌ ✅ ❌ ✅ lograge ❌ ✅ ❌ ✅ logging ❌ ❌ ❌ ❌ contextual_logger ✅ ✅ ❌ ❌ semantic_logger ✅ ✅ ✅ ✅ dry-logger ✅ ❌ ✅ ❌ Thank you to all authors of these gems for your hard work!
80. Instrumentation Build Semantic Logger
81. Instrumentation Build Format As JSON
82. Instrumentation Build Automatic Logging
83. Instrumentation Build • ActionController • ActionMailer • ActionView • ActiveRecord • ActiveJob Supported Libraries
84. Instrumentation Build ActionController
85. Instrumentation Build ActionController
86. Instrumentation Build ActionController
87. Instrumentation Build ActionController
88. Instrumentation Build ActionMailer
89. Instrumentation Build ActionMailer
90. Instrumentation Build ActionMailer
91. Instrumentation Build ActionView
92. Instrumentation Build ActiveRecord
93. Instrumentation Build ActiveJob
94. Instrumentation Build ActiveJob
95. Instrumentation Build ActiveJob
96. Instrumentation Build Customise
97. 1. No conventions 2. Missing attributes 3. Missing API requests Instrumentation Build Weaknesses
98. Instrumentation Build 1. No Conventions
99. Instrumentation Build job_class job.class active_job_class message.class job_class_name job.class.name
100. Instrumentation Build Open Telemetry
101. Instrumentation Build https://go.joyfulprogramming.com/otel-msg
102. Instrumentation Build https://go.joyfulprogramming.com/otel-msg
103. Instrumentation Build
104. Instrumentation Build
105. Instrumentation Build
106. Instrumentation Build
107. Instrumentation Build
108. Instrumentation Build
109. Instrumentation Build
110. Instrumentation Build
111. Instrumentation Build 2. Missing Attributes
112. Request Headers Body ID User Agent Instrumentation Build Response Headers Body
113. Request Headers Body ID User Agent Instrumentation Build Response Headers Body
114. Instrumentation Build http://go.joyfulprogramming.com/otel-http
115. Instrumentation Build https://go.joyfulprogramming.com/otel-useragent
116. Instrumentation Build
117. Instrumentation Build
118. Instrumentation Build
119. Instrumentation Build
120. Instrumentation Build
121. Instrumentation Build 3. Missing API requests
122. Instrumentation Build
123. Instrumentation Build
124. Instrumentation Build
125. Instrumentation Build
126. 1. No conventions 2. Missing attributes 3. Missing API requests Instrumentation Build Weaknesses ✅ ✅ ✅
127. Instrumentation Build Send To Observability Tool
128. Instrumentation Build
130. Instrumentation Build
132. Instrumentation Build
134. Instrumentation Build Phew!
135. Graphs Use
136. Graphs Use Event Filter By Group By Plot Values Of Job Performed Job Queue Job Class Duration
137. Graphs Use
138. Graphs Use
139. Graphs Use
140. Graphs Use
141. Graphs Use
142. Graphs Use
143. Graphs Use • Check time range • Avoid freeform search • Go wide then narrow down Tips
144. Improve
145. Improve Criticise. Show. Learn.
146. Improve Job Latency? Request ID? IP?
152. Question Which jobs are taking the most time?
160. Answer Analytics::UpdateUserVisitsJob
161. Question What's enqueuing Analytics::UpdateUserVisitsJob?
169. Answer profiles#show
170. Question What IP is hitting this action?
178. Answer 15.145.1.3
179. Root Cause Scraper at 15.145.1.3 Hits profiles#show Floods queue with Analytics job Delays password reset emails
180. Fix Block IP in infrastructure
181. Check Is it fixed?
184. Done!
189. Results
190. Downtime
191. 98% reduction Downtime
192. 500 Errors
193. 83% reduction 500 Errors
195. Fix Bugs 20x Faster
196. Extra Resources https://linktr.ee/joyfulprogramming Playbook Sidekiq Course Newsletter
197. Extra Resources https://linktr.ee/joyfulprogramming Workshop - $150 $15 WARSAWRUBY Expires at 9pm
198. John Gallagher Thank you! biggerpockets.com/careers joyfulprogramming.com
199. Questions? biggerpockets.com/careers joyfulprogramming.com