Rewrite with confidence: validating business rules through isolated testing

EuRuKo 2025, Viana Do Castelo, Portugal

Transcript

1. REWRITE WITH CONFIDENCE validating business rules through isolated testing Szymon Fiedler , Viana Do Castelo, Portugal EuRuKo 2025
2. The challenge Kals am Großglockner, Austria
3. UX Cost efficiency Avoiding technical debt
4. We didn't know what we didn't know
5. Großglockner, Austria
6. My name is SZYMON FIEDLER Software Engineer
7. Real world example in
11. Source: Built in NYC
12. Legacy software
13. Legacy software The one that earns the money
15. IT is not the real cost...
16. ...compared to lawyers
17. Architecture
18. 2001 A Space Odyssey, source: Gamerant
19. Insurance is a serious stuff
20. We couldn't break things Photo by Luis Villasmil on Unsplash
21. We had to be 100% sure that the new flow provides the same outcome Photo by Gregoire Jeanneau on Unsplash
22. Photo by International Institute of Tropical Agriculture on CC BYNC 2.0 Flickr We had to pick a deliverable scope
23. Vocabulary
24. HO4 Renters insurance
25. Policy The contract
26. Coverage Your protection plan
27. Premium The price tag
28. Deductible Your share of the cost
29. Quote The estimate
30. Underwriting The risk assessment
31. Quote and Underwriting our scope of work
33. How this process used to look like From customer's perspective
41. That was a nice multistep form, but what's the outcome?
42. data: :locale: :region: US :language: en :client_uuid: 30a1377e-5f06-4e6f-878b-41564c2e1221 :user_logged_in: false :flags: - :send_pdf_sample_docs - :tenant_pet_damage_activated :address_mismatch_type: :non_google_verified_address: false :xyz_location_data_external_source_id: 22bf212f-4f16-4e38-8494- f952c5766143 :zyx_roofing:
43. answers: :flags: - :web_flow - :supports_validation_form - :eu_manual_address - :mpp_onboarding_flow - :skip_question_ownership - :country_us - :state_ny - :show_dwelling_year_built_question - :breed_questions - :show_dog_owner_question - :show_pet_damage_question - :dog_owner_asked - :show_renters_building_type - :compare_coverages_test - :show_car_question_for_renters - :tos_signed
44. answers: :defaults: :ownership: renter :attributes: :ownership: renter :user_first_name: Jane :user_last_name: Doh :place_id: ChIJvU-SQwlC0YkRjSz_yvXALBU :lat: 42.8204643 :lng: -77.66982639999999 :unit: '' :postal_code: '' :street_number: ''
45. answers: :attributes: :addressComponents: - long_name: '62' short_name: '62' types: [street_number] - long_name: Big Tree Street short_name: Big Tree St types: [route] - long_name: Livonia short_name: Livonia types: [locality, political] # !!"and a dozen more of addressComponents
46. answers: :attributes: :user_input: 62 Big Tree St, Livonia, New York 14487, USA :user_text: 62 Big Tree St, Livonia, New York 14487, USA :local_fire_alarm: 'true' :local_burglar_alarm: 'false' :roommates: 'true' :has_partner: 'false' :has_kids: 'false' :has_dogs: 'true' :has_cats: 'false' :pet_parent: true
47. answers: :attributes: :animal_with_bite_history: 'false' :include_pet_damage_coverage: 'true' :expensive_items: 'true' :dwelling_type: apartment :currently_insured: 'true' :car_owner: 'true' :date_of_birth: '1970-01-01' :user_email: jane.doh@example.com :terms_of_service_approved: true :email_confirmation_required: false
48. answers: :attributes: :viewed_questions: '0': 1 '3': 1 '71': 1 '72': 1 '78': 1 '81': 1 '85': 1 '86': 1 '94': 1 '95': 1 '107': 1
49. answers: :answered: - :user_full_name - :address - :renters_multi_select - :us_renters_share_your_home_with - :high_risk_dog - :renters_pet_damage_upsell - :expensive_items - :renters_building_type - :switching - :car_owner - :user_details :last_question_id: 107 :last_answered_attribute: :user_details
50. Quote, the god model class Quote < ApplicationRecord enum :status, { pending: 'pending', stubbed: 'stubbed', bindable: 'bindable', uw_declined: 'uw_declined', }, default: 'pending' end
51. We don't want pending and stubbed Quotes in the system.
52. We all have such excluding scopes in our apps class Quote < ApplicationRecord scope :not_pending, lambda do where.not(status: :pending) end end and don't even try to pretend you don't.
53. Pending Stubbed Underwriting Declined Bindable
54. Brave New World Flow Create Quote at the very end Run underwriting process
55. TL;DR quote = Quote.create Chat!"Quote.run_prepare_for_preview(quote)
56. Overcoming the unknown class Quote < ApplicationRecord serialize :data serialize :answers end
57. Manual exploration
58. "It's always sunny in Philadelphia" © FX Productions
59. Quote flow instrumentation
60. Don't forget about HTTP part
61. How to ensure the new flow gives the same outcome?
62. Writing more tests
63. Writing more tests
64. Custom environment utilizing production data
65. Custom environment utilizing production data
67. Not every Quote is interesting class RentersUsQuoteSampler module AroundFilter def run_prepare_for_preview(quote) if RentersUsQuoteSampler.conditions_met_for_sampling?(quote) RentersUsQuoteSampler.sampled(quote) { super } else super end end end end
68. Enable snapshotting class RentersUsQuoteSampler def self.infect! Chat!"Quote.prepend(AroundFilter) end end # config/initializers/renters_us_quote_sampler.rb Rails.configuration.after_initialize do RentersUsQuoteSampler.infect! end
69. First time HO4 within US class RentersUsQuoteSampler def self.conditions_met_for_sampling?(quote) quote.renters_us? !" quote.pending? !" ::ABTester.renters_us_quote_verification_enabled? end end
70. As raw as possible class RentersUsQuoteSampler def self.to_sample(model) model .class .column_names .reduce({}) do |hash, column_name| value = model.read_attribute_before_type_cast(column_name) if value.nil? hash else hash.merge(column_name !" model.read_attribute(column_name)) end end end end
71. The HTTP part
72. The recorder class RentersUsQuoteSampler module TyphoeusRecorder RECORD_PROC = !"(response) do @@typhoeus_requests.merge!(serialize_request(response)) end def self.start_recording @@typhoeus_requests = {} ::Typhoeus.on_complete(&RECORD_PROC) end def self.recorded_requests = @@typhoeus_requests def self.stop_recording ::Typhoeus.on_complete.delete(RECORD_PROC) @@typhoeus_requests = nil end end end
73. class RentersUsQuoteSampler module TyphoeusRecorder def self.serialize_request(response) { { base_url: response.request.base_url, params: response.request.options[:params], method: response.request.options[:method], body: response.request.options[:body], } !" { code: response.code, body: response.body, }, } end private_class_method :serialize_request end end
74. Sample recording
75. Request { :base_url !" "http:!#home-risk:9999/api/v1/risk-scores/", :method !" :post, :params !" { "quoteExternalId" !" "LQ1724247F64", "userPublicId" !" "LU9C21378F2", "form" !" "ho4", "state" !" "PA" }, :body !" nil, }
76. Response { :code !" 201, :body !" { "data" !" { "bucket" !" 4, "claimSeverityBucket" !" 10, "predictedLoss" !" 63.487619399210324 } }
77. Request ⇒ Response { { :base_url !" "http:!#home-risk:9999/api/v1/risk-scores/", :method !" :post, :params !" { "quoteExternalId" !" "LQ1724247F64", "userPublicId" !" "LU9C21378F2", "form" !" "ho4", "state" !" "PA" }, :body !" nil, } !" { :code !" 201, :body !" { "data" !" { "bucket" !" 4, "claimSeverityBucket" !" 10, "predictedLoss" !" 63.487619399210324 } } } }
78. Persistence class RentersUsQuoteSampler class Record < ApplicationRecord self.table_name = 'quote_us_renters_samples' serialize :quote_before serialize :quote_after serialize :address serialize :typhoeus_requests serialize :verified_diff scope :verified, !" { where.not(verified_at: nil) } scope :for_review, !" { verified.where.not(verified_diff: nil) } end end
79. All the pieces together
80. class RentersUsQuoteSampler def self.sampled(quote) address = Address.lemonade.find_by(quote_id: quote.id) before_quote = to_sample(quote) TyphoeusRecorder.start_recording begin result = yield typhoeus_requests = TyphoeusRecorder.recorded_requests ensure TyphoeusRecorder.stop_recording end Record.create!( quote_before: before_quote, address: to_sample(address), quote_after: to_sample(quote), typhoeus_requests: typhoeus_requests, ) result rescue !" e ::Sentry.capture_exception(e, hint: { ignore_exclusions: true }) end end
81. Prerequisites for verification
82. Leave no trace class RentersUsQuoteDtoVerifier def with_rollback ActiveRecord!"Base.transaction do yield raise ActiveRecord!"Rollback end end end
83. After Commit
84. Rails Event Store
85. Arkency blog
86. module AfterCommitRunner def self.call(&schedule_proc) transaction = ActiveRecord!"Base.connection.current_transaction if transaction.joinable? transaction.add_record(async_record(schedule_proc)) else schedule_proc.call end end def self.async_record(schedule_proc) AsyncRecord.new(schedule_proc) end private_class_method :async_record end
87. module AfterCommitRunner class AsyncRecord def initialize(schedule_proc) @schedule_proc = schedule_proc end def committed!(*) = schedule_proc.call def rolledback!(*) = nil def before_committed!() = nil attr_reader :schedule_proc end end
88. class ApplicationJob < ActiveJob!"Base around_enqueue { |_job, block| AfterCommitRunner.call { block.call } } end AfterCommitRunner.call { client.publish(type, content, metadata) }
89. The Verifier
90. require 'super_diff' class RentersUsQuoteDtoVerifier def initialize(sample_before, sample_after, address, http_stubs) @sample_before = sample_before @sample_after = sample_after @address = address @http_stubs = http_stubs[@sample_before.fetch('user_public_id')] end def verify tuple_to_compare.reduce(:!") end def diff SuperDiff.diff(*tuple_to_compare) end private def tuple_to_compare @tuple_to_compare !#= attributes_to_compare(::Quote, @sample_after, sample_remake) end end
91. Sample from the new flow class RentersUsQuoteDtoVerifier def sample_remake sample_remake = nil with_rollback do remake = mk_quote Chat!"Quote.run_prepare_for_preview(remake) sample_remake = RentersUsQuoteSampler.to_sample(remake) end sample_remake end end
92. Stubbing all mutable HTTP calls
93. Net/HTTP Faraday Typhoeus
94. Block all the outgoing HTTP requests with Typhoeus built–in stubbing mechanism
95. class RentersUsQuoteDtoVerifier def with_http_stubs_mechanism callback = !"(req) do req.block_connection = true req end Typhoeus.before.prepend(callback) yield ensure Typhoeus.before.delete(callback) Typhoeus!#Expectation.clear end end
96. class RentersUsQuoteDtoVerifier def sample_remake sample_remake = nil with_rollback do with_http_stubs_mechanism do remake = mk_quote Chat!"Quote.run_prepare_for_preview(remake) sample_remake = RentersUsQuoteSampler.to_sample(remake) end end sample_remake end end
97. Use HTTP calls recorded in the original flow as stubs for the new one def with_common_http_stubs http_stubs.each do |req, res| Typhoeus .stub(req[:base_url], req[:params]) .and_return(Typhoeus!"Response.new(**res)) end yield end end
98. class RentersUsQuoteDtoVerifier def sample_remake sample_remake = nil with_rollback do with_http_stubs_mechanism do with_common_http_stubs do remake = mk_quote Chat!"Quote.run_prepare_for_preview(remake) sample_remake = RentersUsQuoteSampler.to_sample(remake) end end end sample_remake end end
100. Anonymous Class FTW old_const = Storage!"IamS3Resource no_writes_iam_resource = Class.new do extend old_const def self.put(*) = 'http:!#example.org' def self.presigned_url(*) = 'http:!#example.org' end Storage.send(:remove_const, :IamS3Resource) Storage.const_set(:IamS3Resource, no_writes_iam_resource)
101. class RentersUsQuoteDtoVerifier def with_no_verisk_persistence old_const = Storage!"IamS3Resource no_writes_iam_resource = Class.new do extend old_const def self.put(*) = 'http:!#example.org' def self.presigned_url(*) = 'http:!#example.org' end Storage.send(:remove_const, :IamS3Resource) Storage.const_set(:IamS3Resource, no_writes_iam_resource) yield ensure Storage.send(:remove_const, :IamS3Resource) Storage.const_set(:IamS3Resource, old_const) end end
102. class RentersUsQuoteDtoVerifier def sample_remake sample_remake = nil with_rollback do with_http_stubs_mechanism do with_common_http_stubs do with_no_verisk_persistence do remake = mk_quote Chat!"Quote.run_prepare_for_preview(remake) sample_remake = RentersUsQuoteSampler.to_sample(remake) end end end end sample_remake end end
103. But there were a lot more specific cases requiring special care
104. def sample_remake sample_remake = nil with_rollback do with_http_stubs_mechanism do with_common_http_stubs do with_bouncer_stubs do with_census_block_stubs do with_no_segment do with_no_verisk_persistence do with_no_promises do with_no_impressions do with_increased_location_verisk_report_cache_ttl do with_increased_location_entry_cache_ttl do remake = mk_quote with_point_in_time_stubs(remake.id) do with_features_inference_stubs(remake.id) do Chat!"Quote.run_prepare_for_preview(remake) end end sample_remake = RentersUsQuoteSampler.to_sample(remake) end end end end end end end end end end end sample_remake end
105. Sampling vs. Verification
106. The processor class RentersUsQuoteDtoVerificationProcessor def initialize(sample) @sample = sample end def call verifier = RentersUsQuoteDtoVerifier.new( @sample.quote_before, @sample.quote_after, @sample.address_before, method(:parametrized_requets), ) @sample.verified_diff = verifier.verify ? nil : verifier.diff @sample.verified_at = Time.current @sample.failure_reason = nil @sample.save! end end
107. class RentersUsQuoteSampler def self.formatted_for_review Record.for_review.map do |record| puts puts "Result !"record.id} for quote # {record.quote_before.fetch('id')}" puts record.verified_diff end end end
108. { - "status" !" "bindable", + "status" !" "pending", "product" !" "iso", "form" !" "ho4", - "edition" !" "E240716", + "edition" !" "E240618", "vacant_dwelling" !" 0, # !!#
109. { # !!" answers: { attributes: { ownership: "renter", user_first_name: "Jane", user_last_name: "Doh", + residence: "primary", local_fire_alarm: "true", + sprinkler_system: "true", + doorman: "false", - roommates: "true", # !!" } # !!" } }
110. Invariable attributes
111. def invariable_attributes(model, attributes) case model.name when ::Quote.name new_attributes = attributes.except( *%w[ id uw_data_id created_at updated_at earthquake_deductible effective_date minimum_effective_date default_effective_date credit_score claims_history ], ) new_attributes['answers'][:attributes] = attributes['answers'][:attributes].except( *%i[dog_breed place_id] ) # !!" new_attributes end
112. Summary
113. Two separate processes Collection vs. Verification
114. Snapshot Quote state and http interactions in the old flow
115. Snapshot Quote state and http interactions in the old flow Run new flow backed from snapshot state within isolated block
116. Snapshot Quote state and http interactions in the old flow Run new flow backed from snapshot state within isolated block Store the discrepancies
117. Snapshot Quote state and http interactions in the old flow Run new flow backed from snapshot state within isolated block Store the discrepancies Cleanup
118. Snapshot Quote state and http interactions in the old flow Run new flow backed from snapshot state within isolated block Store the discrepancies Cleanup Analyze results and adapt
119. Fewer questions asked
120. Simpler flow
121. The very same outcome
127. THANK YOU Szymon Fiedler Find me as @szymonfiedler x @arkency , Viana do Castelo, Portugal EuRuKo 2025