Squash Production Defects Quickly - The Power Of Structured Logging - EuRuKo 2024

Rails apps can be a black box. Ever seen a defect where you just can't figure out what's going on?This talk will give you practical steps to improve the observability of your Rails app, taking the time to understand and fix defects from hours or days to minutes.

Rails 8 brings an exciting new feature: built-in structured logging. This talk will delve into the transformative impact of structured logging on application observability and fixing defects.

Structured logging, as a cornerstone of observability, offers a more organized and query-able way to handle logs compared to traditional text-based logs. This session will guide you through the nuances of structured logging in Rails, demonstrating how it can be leveraged to gain better insights into your application's behavior.

This talk will be a deep technical dive into how to make structured logging work with an existing Rails app.

Transcript

1. Squash Production Defects Quickly The Power Of Structured Logging In Rails
3. Based on a true story.
4. Based on a true story. * Names have been changed to protect the innocent
29. Two Months Later
39. Vicious Cycle
41. How often do you encounter issues where you lack visibility into your application's performance or errors?
42. How often do you encounter issues where you lack visibility into your application's performance or errors?
48. The problem isn't you. You're not alone.
52. You Want Answered Question
53. Why are password reset emails not being sent? You Want Answered Question
54. You Want Answered Question
55. Hypotheses You Want Answered Question
56. Why are the icons missing? You Want Answered Question
57. Are all the icons on the page broken? You Want Answered Question
58. Build pipeline broke You Want Answered Question
59. Build pipeline broke Commit that broke icons You Want Answered Question
60. Which icons are broken vs fixed? You Want Answered Question
61. Build pipeline broke Commit that broke icons You Want Answered Question
62. Build pipeline broke Commit that broke icons You Want Answered Question
63. Build pipeline broke Commit that broke icons You Want Answered Question
64. Immediately after deploy old icons don't work You Want Answered Question
65. Immediately after deploy old icons don't work You Want Answered Question
66. How are icon failures distributed over time? You Want Answered Question
67. Why are password reset emails not being sent? You Want Answered Question
68. Failing? Timed Out? Delayed? You Want Answered Question
69. Failing? Timed Out? Delayed? You Want Answered Question
70. Failing? Timed Out? Delayed? You Want Answered Question
71. You Want Answered Question
72. You Want Answered Question
73. Which jobs are taking time in within_five_minutes queue? You Want Answered Question
74. Data To Collect Decide
75. Event Filter By Group By Values Of Data To Collect Decide
76. Event Filter By Group By Values Of Job Performed Data To Collect Decide
77. Event Filter By Group By Values Of Job Performed Job Queue Data To Collect Decide
78. Event Filter By Group By Values Of Job Performed Job Queue Job Class Data To Collect Decide
79. Event Filter By Group By Values Of Job Performed Job Queue Job Class Duration Data To Collect Decide
80. Instrumentation Build
81. Instrumentation Build 1. Data type (Traces, Logs, Metrics) 2. Plain Text vs Structured 3. Logging Library 4. Customise
82. Instrumentation Build Traces vs Logs vs Metrics
83. Instrumentation Build Traces vs Logs vs Metrics
84. Instrumentation Build Plain Text Logs vs Structured Logs
85. Instrumentation Build Plain Text Logs vs Structured Logs
86. Plain Text Logs in Code Instrumentation Build
87. Plain Text Logs in UI Instrumentation Build
88. Plain Text Logs in UI Instrumentation Build
89. Plain Text Logs in UI Instrumentation Build ❌
90. Structured Logs in Code Instrumentation Build
91. Structured Logs in UI Instrumentation Build
92. Structured Logs in UI Instrumentation Build
93. Structured Logs in UI Instrumentation Build ✅
94. Instrumentation Build Logging Library
95. Instrumentation Build • Log structured payload • Integrates with Rails • Great documentation • Mature Criteria
96. Instrumentation Build Library Payload Rails Docs Mature fluentd lograge logging contextual_logger semantic_logger dry-logger Thank you to all authors of these gems for your hard work!
97. Instrumentation Build Library Payload Rails Docs Mature fluentd ❌ ✅ ❌ ✅ lograge ❌ ✅ ❌ ✅ logging ❌ ❌ ❌ ❌ contextual_logger ✅ ✅ ❌ ❌ semantic_logger ✅ ✅ ✅ ✅ dry-logger ✅ ❌ ✅ ❌ Thank you to all authors of these gems for your hard work!
98. Instrumentation Build Library Payload Rails Docs Mature fluentd ❌ ✅ ❌ ✅ lograge ❌ ✅ ❌ ✅ logging ❌ ❌ ❌ ❌ contextual_logger ✅ ✅ ❌ ❌ semantic_logger ✅ ✅ ✅ ✅ dry-logger ✅ ❌ ✅ ❌ Thank you to all authors of these gems for your hard work!
99. Instrumentation Build Semantic Logger
100. Instrumentation Build Format As JSON
101. Instrumentation Build Automatic Logging
102. Instrumentation Build • ActionController • ActionMailer • ActionView • ActiveRecord • ActiveJob Supported Libraries
103. Instrumentation Build ActiveJob
104. Instrumentation Build ActiveJob
105. Instrumentation Build ActiveJob
106. Instrumentation Build ActiveJob
107. Instrumentation Build
108. 1. No conventions 2. Missing attributes 3. Missing API requests Many more... Instrumentation Build Weaknesses
109. Instrumentation Build 1. No Conventions
110. Instrumentation Build job_class job.class active_job_class message.class job_class_name job.class.name
111. Instrumentation Build OpenTelemetry
112. Instrumentation Build https://go.joyfulprogramming.com/otel-msg
113. Instrumentation Build https://go.joyfulprogramming.com/otel-msg
114. Instrumentation Build
115. Instrumentation Build
116. Instrumentation Build
117. Instrumentation Build
118. Instrumentation Build
119. Instrumentation Build
120. Instrumentation Build
121. Instrumentation Build 2. Missing Attributes
122. Request Headers Body ID User Agent Instrumentation Build Response Headers Body
123. Request Headers Body ID User Agent Instrumentation Build Response Headers Body
124. Instrumentation Build
125. Instrumentation Build
126. Instrumentation Build
127. Instrumentation Build
128. Instrumentation Build
129. Instrumentation Build 3. Missing API requests
130. Instrumentation Build
131. Instrumentation Build
132. Instrumentation Build
133. Instrumentation Build
134. 1. No conventions 2. Missing attributes 3. Missing API requests Instrumentation Build Weaknesses ✅ ✅ ✅
135. Instrumentation Build Send To Observability Tool
136. Instrumentation Build
138. Instrumentation Build Phew!
139. Graphs Use
140. Graphs Use Event Filter By Group By Plot Values Of Job Performed Job Queue Job Class Duration
141. Graphs Use
142. Graphs Use
143. Graphs Use
144. Graphs Use
145. Graphs Use
146. Improve
147. Improve Criticise. Show. Learn.
148. Improve Job Latency? Request ID? IP?
154. Question Which jobs are taking the most time?
162. Answer Analytics::UpdateUserVisitsJob
163. Question What's enqueuing Analytics::UpdateUserVisitsJob?
171. Answer profiles#show
172. Question What IP is hitting this action?
180. Answer 15.145.1.3
181. Root Cause Scraper at 15.145.1.3 Hits profiles#show Floods queue with Analytics job Delays password reset emails
182. Fix Block IP in infrastructure
183. Check Is it fixed?
186. Done!
191. Results
192. Downtime
193. 98% reduction Downtime
194. 500 Errors
195. 83% reduction 500 Errors
197. Fix Bugs 20x Faster
198. Extra Resources https://linktr.ee/joyfulprogramming Playbook (Free) Newsletter (Free) Sidekiq Course ($1)
199. 60% Off Workshops https://linktr.ee/joyfulprogramming Team $2000 $800 Solo $150 $60 EURUKO Expires today at 9pm!
200. John Gallagher Thank you! biggerpockets.com/careers joyfulprogramming.com
201. https://linktr.ee/joyfulprogramming biggerpockets.com/careers joyfulprogramming.com