Dirty Magic for Resilient API Dependencies

Imagine that you want to build a system which depends on external service, e.g., logistics, payments or notifications service. Those systems have its life-cycle which you have to be in sync with. I’ll share how to treat issues you could face, using the examples of DHL, UPS, Russian Post integrations.

Transcript

1. Dirty magic for resilient API dependencies Sergey Dolganov @ Evil Martians
9. 7 Distributed system with lots of API integrations and calls
10. 7 Distributed system with lots of API integrations and calls
26. 15 Contract is:
27. 15 Contract is: Pre-conditions (request)
28. 15 Contract is: Pre-conditions (request) Post-conditions (response)
29. 15 Contract is: Pre-conditions (request) Post-conditions (response) Invariants (state)
30. 16 Contract Pre-conditions Post-conditions Invariants State function (input) { … } # => Input Output
48. 22 Evolution
49. 22 Evolution Input/Output Control
50. 22 Evolution Input/Output Control State Control
51. 22 Evolution Input/Output Control State Control Homogeneous Design
52. 23 Problem solving
53. 24 API Client Problem
54. 24 API Client Problem
55. 24 API Client Problem
56. 25 Standalone Policy Step 1
57. 26 Layout (for DHL) eBay Orders, eBaymag Parcels Lots of validations HTTP Client Confusing content
58. 27 module DHL module API def get_tariff(mapped_request) sampler = TariffSampler.new sampler.register_request(mapped_request) response = http_client.get("/GetQuote", mapped_request.to_xml) sampler.register_response(response) sampler.tag! response end
59. 28 class TariffSampler < BaseSampler def tag! tag_success! tag_dhl_address_ambiguity_warning! tag_dhl_composite_call_warning! tag_data_invalid_input_error! tag_destination_invalid_for_dhl! tag_unexpected_behavior! if tags.empty? end # ... end Response data “Contract”
60. 29 module DHL class Calculator < ::BaseService attr_reader :errors param :parcel def call policy = DHL::RegionalPolicy[parcel] return if policy.invalid? policy = DHL::Requests::TariffPolicy[parcel] return if policy.invalid? response = DHL::API.get_tariff(Requests::TariffMapper[parcel]) mapped_response = Responses::TariffMapper[response] policy = DHL::Responses::TariffPolicy[response, mapped_response] return if policy.invalid? parcel.merge_update!(dhl: mapped_response) ensure @errors = policy&.errors end end end
61. 30 module DHL module Responses class TariffPolicy < Tram::Policy param :response param :mapped_response validate :no_response_errors, stop_on_failure: true validate :delivery_cost_presence validate :delivery_currency_presence validate :delivery_date_presence def no_response_errors return if response.errors.empty? errors.add :dhl_responded_with_error, errors: response.errors end # implementation of other methods end end
62. 31 module DHL class Calculator < ::BaseService attr_reader :errors param :parcel def call policy = DHL::RegionalPolicy[parcel] return if policy.invalid? policy = DHL::Requests::TariffSchemaPolicy[parcel] return if policy.invalid? response = DHL::API.get_tariff(Requests::TariffMapper[parcel]) mapped_response = Responses::TariffMapper[response] policy = DHL::Responses::TariffPolicy[response, mapped_response] return if policy.invalid? parcel.merge_update!(dhl: mapped_response) ensure @errors = policy&.errors end end end
63. 32 class DHL::Requests::TariffMapper extend Dry::Initializer option :parcel, optional: true option :origin, default: -> { default_origin } option :destination, default: -> { default_destination } option :reference_id, default: -> { default_reference_id } option :value, default: -> { default_value } option :weight, default: -> { default_weight } # other parameters def call [ route_attributes, size_weight_attributes, package_attributes, ].reduce(:merge) end end
64. 33 if (service = DHL::Calculator.new(parcel)).call render json: { tariff: parcel.reload.dhl["tariff"] } else render json: { errors: service.errors.to_h } end
65. 34 Dry::Initializer Tram::Policy API Sampling + +
66. 35 Order Order
67. 36 Sagas Step 2
68. 37 Sagas. Orchestration approach 1. Wrap each mutation in a Command 2.Chain Command 3.Execute chain 4.If one fails run `undo`
69. 38 Saga Orchestrator Parcel Parcel* Validate Validate Print Print … … Success Failure
70. 39 Validate PrintLabel … Success Validate#call PrintLabel#call
71. 40 Validate PrintLabel … Failure Validate#undo PrintLabel#undo
72. 41 Aaaand that’s it?
73. 42 Let’s implement sagas!
74. 42 Let’s implement sagas!
75. 43 eBaymag Sagas
76. 44 Let’s extract sagas to a gem!
77. 44 Let’s extract sagas to a gem!
78. 44 Let’s extract sagas to a gem!
79. 45 DirtyPipeline (Sagas)
80. 45 DirtyPipeline (Sagas)
82. 47 Homogeneous Design Step 3
83. 48 …2 years of fighting API issues…
84. 49 Functional way to validation
85. 50 Composition Errors Processing +
86. 51 Algebraic Data Types module UPS module Rates # types product, # Request passes both RegionalParcel and Schema contracts RequestPolicy = RegionalPolicy.and_then(SchemaPolicy) # types sum, # Error passes either RecoverableInputError or # InvalidRequest or BasicError contract ErrorPolicy = RecoverableInputErrorPolicy .or(InvalidRequestPolicy) .or(BasicErrorPolicy) # combine ResponsePolicy = JSONResponsePolicy.and_then(TariffPolicy.or(ErrorPolicy)) end end
87. 52 Refinement Types — Type (ADT) — Predicate
88. 52 Refinement Types — Type (ADT) — Predicate { n :: Integer | n > 5 }
93. 57 Context: Why did validation fail? Contract Failure
94. 58 Refinement Type irb:1> Types::JSONResponse.match(HTTParty.get(“…”)) # => #<Types::JSONResponse …> # OR #<ContractFailure …> irb:2> _.unpack # => {…}
95. 59 Refinement Type module Types class JSONResponse < BaseType alias :response :value def match context[:body] = response.body context[:parsed] = ::JSON.parse(response.body) self rescue StandardError => error ContractFailure.new(error, context) end def mapped context[:parsed] end end end
96. 60 Refinement Type module UPS module Shipping class InputSchemaType < BaseType extract :origin extract :destination extract :weight def match extract! data = context.slice(*self.class.extractables) return self if ParcelPolicy[**data].valid? PolicyFailure.new(policy.errors, context) end def mapped context.slice(*self.class.extractables) end end end end Advanced
97. 61 class UPS::RatesContract < BaseContract # sampling by types def match(*input) input_match = Rates::RequestType.match(*input) return input_match if input_match.invalid? result = yield(input_match) Rates::ResponseType.match(result, input_match.context) end end
98. 62 module UPS module Rates # types product, # Request passes both RegionalPolicy and SchemaInput RequestType = RegionalPolicyType.and_then(InputSchemaType) # types sum, # Error passes either RecoverableInputError or # InvalidRequest or BasicError contract ErrorType = RecoverableInputErrorType .or(InvalidRequestType) .or(BasicErrorType) # combine ResponseType = Types::JSONResponse.and_then(Types::LogisticsTariff.or(ErrorType)) end end Contract Definiton
99. 63 class UPS::API def rates_request(parcel) RatesContract.match(parcel) do |mapped_parcel| http_client.get('/1.0/Rates', mapped_parcel.unpack) end # returns a refined "type" end end
100. 64 case (match = UPS::API.rate_request(parcel)) when LogisticsTariff render json: { tariff: match.unpack } when PolicyFailure render json: { errors: match.errors } when InvalidRequest Honeybadger.notify "Invalid UPS request", context: match.context render json: { errors: "Unprocessable now, try later" } when ContractFailure Honeybadger.notify "Unexpected UPS behavior", context: match.context render json: { errors: "Unprocessable now, try later" } else raise "Invalid contract processing" end
101. 65 Monitoring module BC class YabedaInstrument def call(session) valid_marker = session.valid? ? "V" : "I" result = "[#{valid_marker}] #{session.result_type_name}" Yabeda.api_contract_matches.increment(result: result) end end end BloodContracts::Instrumentation.configure do |cfg| # Attach to every BC::Refined ancestor matching UPS.*Contract cfg.instrument /UPS.*Contract/, BC::YabedaInstrument.new # Uses Sniffer to record Requests and Responses cfg.instrument /UPS.*Contract/, BC::SnifferInstrument.new end
102. 65 Monitoring module BC class YabedaInstrument def call(session) valid_marker = session.valid? ? "V" : "I" result = "[#{valid_marker}] #{session.result_type_name}" Yabeda.api_contract_matches.increment(result: result) end end end BloodContracts::Instrumentation.configure do |cfg| # Attach to every BC::Refined ancestor matching UPS.*Contract cfg.instrument /UPS.*Contract/, BC::YabedaInstrument.new # Uses Sniffer to record Requests and Responses cfg.instrument /UPS.*Contract/, BC::SnifferInstrument.new end
103. 66 Yabeda & Sniffer Tram::Policy BloodContracts + + (on Refinement Types)
104. 67 New Horizons Outcome
105. 68 Evolution Input/Output Control State Control Homogeneous Design
106. 69 Future Steps — Tram::Events — DirtyPipeline — BloodContracts docs
107. 70 Extra Reading — Category Theory for Programmers — Maybe Haskell — Chaos Engineering
108. 71 Thanks! @ss_dolganov @sergey.dolganov Contracts: sclinede/blood_contracts
109. 71 Thanks! @ss_dolganov @sergey.dolganov Contracts: sclinede/blood_contracts Our blog: evilmartians.com/ chronicles