Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Too Big To Fail

Too Big To Fail

RailsConf 2014 talk on handling failure at ZenPayroll and in life.

Avatar for Chris Maddox

Chris Maddox

April 22, 2014
Tweet

Other Decks in Technology

Transcript

  1. Problems - User prevented from fixing or even seeing the

    problem - Errors cascade to other processes - update_column - update_attributes(validate: false) Tuesday, April 22, 14
  2. Solution - Every three hours, validate every model - Pull

    (sanitized) production data and validate against merged but undeployed code Tuesday, April 22, 14
  3. Try Before You Buy - Run important tasks in staging

    environment on sanitized data - Catch errors before they bring production down - Give yourself time to make the right fix Tuesday, April 22, 14
  4. “Failure happens all the time. It happens every day in

    practice. What makes you better is how you react to it.” – Mia Hamm Tuesday, April 22, 14
  5. maintenance.rb 123 if FtpAch.send_nacha_file(nacha_file) 124 nacha_files << nacha_file 125 end

    126 127 128 nacha_file = NachaFile. 129 generate_nacha_file(Date. 130 today, credit_batches) 131 if FtpAch.send_nacha_file(nacha_file) 132 nacha_files << nacha_file 133 end 134 135 f.puts "" 136 f.puts "Generating and uploading bank 137 verification nacha files" 138 f.puts "" 139 140 verification_credit_batches, 141 verification_debit_batches = Company. 142 process_bank_account_verification 143 144 verification_credit_batches += Employee. 145 process_b 146 ank_accou 147 nt_verifi 148 cation 149 nacha_file = NachaFile. 150 generate_nacha_file(Date. 151 today, 152 verification_credit_batches 153 ) 154 if FtpAch.send_nacha_file(nacha_file) 155 nacha_files << nacha_file 156 end 157 158 # we need to get our money back for the 159 credit batches for company 160 verification 161 nacha_file = NachaFile. 162 generate_nacha_file(Date. 163 today, 164 verification_debit_batches) 165 if FtpAch.send_nacha_file(nacha_file) 166 nacha_files << nacha_file 167 end 168 169 f.puts "" 170 f.puts "Generating and uploading fixed 171 nacha files" 172 f.puts "" 173 174 fixed_debit_batches, 175 fixed_credit_batches = NachaFile. 176 process_fixed_batches 177 178 nacha_file = NachaFile. 179 generate_nacha_file(Date. 180 today, fixed_debit_batches) 181 if FtpAch.send_nacha_file(nacha_file) 182 nacha_files << nacha_file 183 end 1 class Maintenance 2 3 def self.run_maintenance! 4 dates = MaintenanceLog.missing_run_dates 5 dates.each do |date| 6 Maintenance.run_maintenance_for_date!(date 7 ) 8 end 9 end 10 11 def self.run_maintenance_for_date!(date = 12 Date.today) 13 14 log_file = "log/maintenance/" + date. 15 strftime("%Y%m%d") + ".log" 16 17 File.open(log_file, 'w') do |f| 18 f.puts "Maintenance for " + date.to_s 19 f.puts "" 20 f.puts "" 21 f.puts "Start time: " + Time.now.to_s 22 f.puts "" 23 24 begin 25 26 f.puts "" 27 f.puts "Processing nacha response 28 files" 29 f.puts "" 30 31 FtpAch.process_response_files 32 33 f.puts "" 34 f.puts "Processing company maintenance" 35 f.puts "" 36 37 debit_batches = [] 38 credit_batches = [] 39 deposits = [] 40 41 Company.find_each do |company| 42 begin 43 f.puts "Maintenance for company " + 44 company.id.to_s + " (" + company. 45 name + ")" 46 f.puts "" 47 #TODO: Don't save the agent_payment 48 unless the payment has actually 49 been sent 50 company_actions, 51 company_debit_batches, 52 company_credit_batches, 53 company_deposits = company. 54 run_maintenance!(date) 55 company_actions.each do |action| 56 f.puts action 57 end 58 59 debit_batches += company_debit_batch 60 es 61 62 credit_batches += company_credit_bat 63 ches 64 65 deposits += company_deposits 66 67 rescue Exception => e 68 f.puts "THERE WAS AN EXCEPTION" 69 f.puts "" 70 f.puts e.inspect 71 f.puts "" 72 f.puts e.backtrace 73 end 74 75 f.puts "" 76 end 77 78 f.puts "" 79 f.puts "Processing deposits" 80 f.puts "" 81 82 #TODO: write a test to make sure that 83 the payments are processed in the 84 correct order 85 hashed_deposits = {} 86 GOVERNMENT_DEPOSITS.each { |pay_to| 87 hashed_deposi 88 ts[pay_to] = 89 [] } 90 deposits.each do |deposit| 91 hashed_deposits[deposit[:pay_to]] << 92 deposit[:custom_data] 93 end 94 95 hashed_deposits.each_pair do |pay_to, 96 custom_data| 97 if custom_data.length > 0 98 deposit_actions = ("TaxPayment" + 99 pay_to). 100 constantize. 101 submit!( 102 custom_data) 103 # If deposit_actions is non-empty 104 # send a notification email to 105 [email protected] 106 AdminMailer.delay. 107 eftps_deposit_email(deposit_actions) 108 deposit_actions.each { |action| f. 109 puts action } 110 end 111 end 112 113 f.puts "" 114 f.puts "Generating and uploading 115 company maintenance nacha files" 116 f.puts "" 117 118 nacha_files = [] 119 120 nacha_file = NachaFile. 121 generate_nacha_file(Date. 122 today, debit_batches) 184 185 nacha_file = NachaFile. 186 generate_nacha_file(Date. 187 today, 188 fixed_credit_batches) 189 if FtpAch.send_nacha_file(nacha_file) 190 nacha_files << nacha_file 191 end 192 193 f.puts "" 194 f.puts "Sending ACH control email" 195 f.puts "" 196 197 if nacha_files.length > 0 198 if Rails.env.production? 199 AchMailer.control_email(nacha_files) 200 .deliver 201 end 202 end 203 204 f.puts "Sending Faxes" 205 f.puts "" 206 207 # Fax out signed, unfiled 8655's 208 signed_forms = Form.where( 209 "filing_status = ? AND 210 name = ? AND 211 signatory_id IS NOT 212 NULL", 213 FORM_FILING_STATUS_NOT_FI 214 LED, FORM_NAME_8655) 215 if signed_forms 216 f.puts "Faxing out signed 8655's" 217 f.puts "" 218 219 signed_forms.each do |form| 220 if form.fax! 221 f.puts "* Faxed form #{form.id} 222 for #{form. 223 company.id} 224 (#{form. 225 company. 226 name})" 227 f.puts "" 228 end 229 end 230 end 231 232 # Fax out signed, unfiled 8821's 233 signed_forms = Form.where( 234 "filing_status = ? AND 235 name = ? AND 236 signatory_id IS NOT 237 NULL", 238 FORM_FILING_STATUS_NOT_FI 239 LED, FORM_NAME_8821) 240 if signed_forms 241 f.puts "Faxing out signed 8821's" 242 f.puts "" 243 244 signed_forms.each do |form| 245 if form.fax! 246 f.puts "* Faxed form #{form.id} 247 for #{form. 248 company.id} 249 (#{form. 250 company. 251 name})" 252 f.puts "" 253 end 254 end 255 end 256 257 f.puts "" 258 f.puts "Enrolling companies who just 259 filed their 8655s into EFTPS" 260 f.puts "" 261 262 enrollment_actions = Company. 263 process_eftps_enrol 264 lments 265 unless enrollment_actions.empty? 266 # Send email to [email protected] 267 AdminMailer.delay. 268 eftps_enrollment_email( 269 enrollment_actions) 270 end 271 enrollment_actions.each do |action| 272 f.puts action 273 end 274 275 f.puts "" 276 f.puts "Sending sysops notification 277 emails" 278 f.puts "" 279 280 EddTransmission. 281 notify_sysops_of_pending_transmissions 282 283 if date.month == 12 and date.day == 15 284 f.puts "Sending year-end compliance 285 emails" 286 f.puts "" 287 288 PayrollAdmin.find_each do 289 |payroll_admin| 290 PayrollAdminMailer.delay. 291 year_end_compliance_email( 292 payroll_admin) 293 end 294 end 295 296 297 rescue Exception => e2 298 299 f.puts "THERE WAS A TOP LEVEL 300 EXCEPTION" 301 f.puts "" 302 f.puts e2.inspect 303 f.puts "" 304 f.puts e2.backtrace 305 306 end 307 308 f.puts "" 309 f.puts "End time: " + Time.now.to_s 310 f.puts "" 311 end 312 313 # Sync payroll administrators with the 314 mailchimp list 315 if Rails.env.production? || Rails.env. 316 staging? 317 MailChimp::API.sync('payroll_admins', { 318 is_payroll_admin: 319 true}) 320 end 321 322 AdminMailer.delay.maintenance_email(date, 323 log_file 324 ) 325 MaintenanceLog.create({ :ran_at => date }) 326 end 327 end Tuesday, April 22, 14
  6. maintenance.rb 123 if FtpAch.send_nacha_file(nacha_file) 124 nacha_files << nacha_file 125 end

    126 127 128 nacha_file = NachaFile. 129 generate_nacha_file(Date. 130 today, credit_batches) 131 if FtpAch.send_nacha_file(nacha_file) 132 nacha_files << nacha_file 133 end 134 135 f.puts "" 136 f.puts "Generating and uploading bank 137 verification nacha files" 138 f.puts "" 139 140 verification_credit_batches, 141 verification_debit_batches = Company. 142 process_bank_account_verification 143 144 verification_credit_batches += Employee. 145 process_b 146 ank_accou 147 nt_verifi 148 cation 149 nacha_file = NachaFile. 150 generate_nacha_file(Date. 151 today, 152 verification_credit_batches 153 ) 154 if FtpAch.send_nacha_file(nacha_file) 155 nacha_files << nacha_file 156 end 157 158 # we need to get our money back for the 159 credit batches for company 160 verification 161 nacha_file = NachaFile. 162 generate_nacha_file(Date. 163 today, 164 verification_debit_batches) 165 if FtpAch.send_nacha_file(nacha_file) 166 nacha_files << nacha_file 167 end 168 169 f.puts "" 170 f.puts "Generating and uploading fixed 171 nacha files" 172 f.puts "" 173 174 fixed_debit_batches, 175 fixed_credit_batches = NachaFile. 176 process_fixed_batches 177 178 nacha_file = NachaFile. 179 generate_nacha_file(Date. 180 today, fixed_debit_batches) 181 if FtpAch.send_nacha_file(nacha_file) 182 nacha_files << nacha_file 183 end 1 class Maintenance 2 3 def self.run_maintenance! 4 dates = MaintenanceLog.missing_run_dates 5 dates.each do |date| 6 Maintenance.run_maintenance_for_date!(date 7 ) 8 end 9 end 10 11 def self.run_maintenance_for_date!(date = 12 Date.today) 13 14 log_file = "log/maintenance/" + date. 15 strftime("%Y%m%d") + ".log" 16 17 File.open(log_file, 'w') do |f| 18 f.puts "Maintenance for " + date.to_s 19 f.puts "" 20 f.puts "" 21 f.puts "Start time: " + Time.now.to_s 22 f.puts "" 23 24 begin 25 26 f.puts "" 27 f.puts "Processing nacha response 28 files" 29 f.puts "" 30 31 FtpAch.process_response_files 32 33 f.puts "" 34 f.puts "Processing company maintenance" 35 f.puts "" 36 37 debit_batches = [] 38 credit_batches = [] 39 deposits = [] 40 41 Company.find_each do |company| 42 begin 43 f.puts "Maintenance for company " + 44 company.id.to_s + " (" + company. 45 name + ")" 46 f.puts "" 47 #TODO: Don't save the agent_payment 48 unless the payment has actually 49 been sent 50 company_actions, 51 company_debit_batches, 52 company_credit_batches, 53 company_deposits = company. 54 run_maintenance!(date) 55 company_actions.each do |action| 56 f.puts action 57 end 58 59 debit_batches += company_debit_batch 60 es 61 62 credit_batches += company_credit_bat 63 ches 64 65 deposits += company_deposits 66 67 rescue Exception => e 68 f.puts "THERE WAS AN EXCEPTION" 69 f.puts "" 70 f.puts e.inspect 71 f.puts "" 72 f.puts e.backtrace 73 end 74 75 f.puts "" 76 end 77 78 f.puts "" 79 f.puts "Processing deposits" 80 f.puts "" 81 82 #TODO: write a test to make sure that 83 the payments are processed in the 84 correct order 85 hashed_deposits = {} 86 GOVERNMENT_DEPOSITS.each { |pay_to| 87 hashed_deposi 88 ts[pay_to] = 89 [] } 90 deposits.each do |deposit| 91 hashed_deposits[deposit[:pay_to]] << 92 deposit[:custom_data] 93 end 94 95 hashed_deposits.each_pair do |pay_to, 96 custom_data| 97 if custom_data.length > 0 98 deposit_actions = ("TaxPayment" + 99 pay_to). 100 constantize. 101 submit!( 102 custom_data) 103 # If deposit_actions is non-empty 104 # send a notification email to 105 [email protected] 106 AdminMailer.delay. 107 eftps_deposit_email(deposit_actions) 108 deposit_actions.each { |action| f. 109 puts action } 110 end 111 end 112 113 f.puts "" 114 f.puts "Generating and uploading 115 company maintenance nacha files" 116 f.puts "" 117 118 nacha_files = [] 119 120 nacha_file = NachaFile. 121 generate_nacha_file(Date. 122 today, debit_batches) 184 185 nacha_file = NachaFile. 186 generate_nacha_file(Date. 187 today, 188 fixed_credit_batches) 189 if FtpAch.send_nacha_file(nacha_file) 190 nacha_files << nacha_file 191 end 192 193 f.puts "" 194 f.puts "Sending ACH control email" 195 f.puts "" 196 197 if nacha_files.length > 0 198 if Rails.env.production? 199 AchMailer.control_email(nacha_files) 200 .deliver 201 end 202 end 203 204 f.puts "Sending Faxes" 205 f.puts "" 206 207 # Fax out signed, unfiled 8655's 208 signed_forms = Form.where( 209 "filing_status = ? AND 210 name = ? AND 211 signatory_id IS NOT 212 NULL", 213 FORM_FILING_STATUS_NOT_FI 214 LED, FORM_NAME_8655) 215 if signed_forms 216 f.puts "Faxing out signed 8655's" 217 f.puts "" 218 219 signed_forms.each do |form| 220 if form.fax! 221 f.puts "* Faxed form #{form.id} 222 for #{form. 223 company.id} 224 (#{form. 225 company. 226 name})" 227 f.puts "" 228 end 229 end 230 end 231 232 # Fax out signed, unfiled 8821's 233 signed_forms = Form.where( 234 "filing_status = ? AND 235 name = ? AND 236 signatory_id IS NOT 237 NULL", 238 FORM_FILING_STATUS_NOT_FI 239 LED, FORM_NAME_8821) 240 if signed_forms 241 f.puts "Faxing out signed 8821's" 242 f.puts "" 243 244 signed_forms.each do |form| 245 if form.fax! 246 f.puts "* Faxed form #{form.id} 247 for #{form. 248 company.id} 249 (#{form. 250 company. 251 name})" 252 f.puts "" 253 end 254 end 255 end 256 257 f.puts "" 258 f.puts "Enrolling companies who just 259 filed their 8655s into EFTPS" 260 f.puts "" 261 262 enrollment_actions = Company. 263 process_eftps_enrol 264 lments 265 unless enrollment_actions.empty? 266 # Send email to [email protected] 267 AdminMailer.delay. 268 eftps_enrollment_email( 269 enrollment_actions) 270 end 271 enrollment_actions.each do |action| 272 f.puts action 273 end 274 275 f.puts "" 276 f.puts "Sending sysops notification 277 emails" 278 f.puts "" 279 280 EddTransmission. 281 notify_sysops_of_pending_transmissions 282 283 if date.month == 12 and date.day == 15 284 f.puts "Sending year-end compliance 285 emails" 286 f.puts "" 287 288 PayrollAdmin.find_each do 289 |payroll_admin| 290 PayrollAdminMailer.delay. 291 year_end_compliance_email( 292 payroll_admin) 293 end 294 end 295 296 297 rescue Exception => e2 298 299 f.puts "THERE WAS A TOP LEVEL 300 EXCEPTION" 301 f.puts "" 302 f.puts e2.inspect 303 f.puts "" 304 f.puts e2.backtrace 305 306 end 307 308 f.puts "" 309 f.puts "End time: " + Time.now.to_s 310 f.puts "" 311 end 312 313 # Sync payroll administrators with the 314 mailchimp list 315 if Rails.env.production? || Rails.env. 316 staging? 317 MailChimp::API.sync('payroll_admins', { 318 is_payroll_admin: 319 true}) 320 end 321 322 AdminMailer.delay.maintenance_email(date, 323 log_file 324 ) 325 MaintenanceLog.create({ :ran_at => date }) 326 end 327 end NoMethodError: undefined method `<<' for nil:NilClass Tuesday, April 22, 14
  7. Step One: Break it Down • Group conceptually similar pieces

    • Localize failures (minimize their impact) • Domains are still huge • Midpoint failures in Secondary Maintenance Pros Cons Tuesday, April 22, 14
  8. Lifecycle 1. before_run callbacks 2. run 1. on_error callbacks 2.

    reset 3. on_reset callbacks 3. after_run callbacks Tuesday, April 22, 14
  9. “You're afraid of making mistakes. Don't be. Mistakes can be

    profited by. Man, when I was young I shoved my ignorance in people's faces. They beat me with sticks. By the time I was forty my blunt instrument had been honed to a fine cutting point for me. If you hide your ignorance, no one will hit you and you'll never learn.” – Ray Bradbury Tuesday, April 22, 14
  10. “Good people are not those who lack flaws, the brave

    are not those who feel no fear, and the generous are not those who never feel selfish. Extraordinary people are not extraordinary because they are invulnerable to unconscious biases. They are extraordinary because they choose to do something about it.” – Shankar Vedantam Tuesday, April 22, 14