Slide 1
Slide 1 text
Dirty magic for resilient
API dependencies
Sergey Dolganov
@ Evil Martians
Slide 2
Slide 2 text
2
Slide 3
Slide 3 text
2
Slide 4
Slide 4 text
2
Slide 5
Slide 5 text
3
Slide 6
Slide 6 text
4
Slide 7
Slide 7 text
5
Slide 8
Slide 8 text
6
Slide 9
Slide 9 text
7
Distributed system with lots of API integrations and calls
Slide 10
Slide 10 text
7
Distributed system with lots of API integrations and calls
Slide 11
Slide 11 text
8
Slide 12
Slide 12 text
9
Slide 13
Slide 13 text
9
Slide 14
Slide 14 text
9
Slide 15
Slide 15 text
10
Slide 16
Slide 16 text
10
Slide 17
Slide 17 text
10
Slide 18
Slide 18 text
11
Slide 19
Slide 19 text
11
Slide 20
Slide 20 text
11
Slide 21
Slide 21 text
12
Slide 22
Slide 22 text
12
Slide 23
Slide 23 text
13
Slide 24
Slide 24 text
13
Slide 25
Slide 25 text
14
Slide 26
Slide 26 text
15
Contract is:
Slide 27
Slide 27 text
15
Contract is:
Pre-conditions
(request)
Slide 28
Slide 28 text
15
Contract is:
Pre-conditions
(request)
Post-conditions
(response)
Slide 29
Slide 29 text
15
Contract is:
Pre-conditions
(request)
Post-conditions
(response)
Invariants
(state)
Slide 30
Slide 30 text
16
Contract
Pre-conditions Post-conditions
Invariants
State
function (input) { … } # =>
Input Output
Slide 31
Slide 31 text
17
Slide 32
Slide 32 text
18
Slide 33
Slide 33 text
18
Slide 34
Slide 34 text
18
Slide 35
Slide 35 text
18
Slide 36
Slide 36 text
19
Slide 37
Slide 37 text
19
Slide 38
Slide 38 text
19
Slide 39
Slide 39 text
19
Slide 40
Slide 40 text
20
Slide 41
Slide 41 text
20
Slide 42
Slide 42 text
20
Slide 43
Slide 43 text
20
Slide 44
Slide 44 text
21
Slide 45
Slide 45 text
21
Slide 46
Slide 46 text
21
Slide 47
Slide 47 text
21
Slide 48
Slide 48 text
22
Evolution
Slide 49
Slide 49 text
22
Evolution
Input/Output
Control
Slide 50
Slide 50 text
22
Evolution
Input/Output
Control
State
Control
Slide 51
Slide 51 text
22
Evolution
Input/Output
Control
State
Control
Homogeneous
Design
Slide 52
Slide 52 text
23
Problem
solving
Slide 53
Slide 53 text
24
API Client Problem
Slide 54
Slide 54 text
24
API Client Problem
Slide 55
Slide 55 text
24
API Client Problem
Slide 56
Slide 56 text
25
Standalone
Policy
Step 1
Slide 57
Slide 57 text
26
Layout (for DHL)
eBay Orders,
eBaymag Parcels Lots of validations HTTP Client
Confusing
content
Slide 58
Slide 58 text
27
module DHL
module API
def get_tariff(mapped_request)
sampler = TariffSampler.new
sampler.register_request(mapped_request)
response = http_client.get("/GetQuote", mapped_request.to_xml)
sampler.register_response(response)
sampler.tag!
response
end
Slide 59
Slide 59 text
28
class TariffSampler < BaseSampler
def tag!
tag_success!
tag_dhl_address_ambiguity_warning!
tag_dhl_composite_call_warning!
tag_data_invalid_input_error!
tag_destination_invalid_for_dhl!
tag_unexpected_behavior! if tags.empty?
end
# ...
end
Response data “Contract”
Slide 60
Slide 60 text
29
module DHL
class Calculator < ::BaseService
attr_reader :errors
param :parcel
def call
policy = DHL::RegionalPolicy[parcel]
return if policy.invalid?
policy = DHL::Requests::TariffPolicy[parcel]
return if policy.invalid?
response = DHL::API.get_tariff(Requests::TariffMapper[parcel])
mapped_response = Responses::TariffMapper[response]
policy = DHL::Responses::TariffPolicy[response, mapped_response]
return if policy.invalid?
parcel.merge_update!(dhl: mapped_response)
ensure
@errors = policy&.errors
end
end
end
Slide 61
Slide 61 text
30
module DHL
module Responses
class TariffPolicy < Tram::Policy
param :response
param :mapped_response
validate :no_response_errors, stop_on_failure: true
validate :delivery_cost_presence
validate :delivery_currency_presence
validate :delivery_date_presence
def no_response_errors
return if response.errors.empty?
errors.add :dhl_responded_with_error,
errors: response.errors
end
# implementation of other methods
end
end
Slide 62
Slide 62 text
31
module DHL
class Calculator < ::BaseService
attr_reader :errors
param :parcel
def call
policy = DHL::RegionalPolicy[parcel]
return if policy.invalid?
policy = DHL::Requests::TariffSchemaPolicy[parcel]
return if policy.invalid?
response = DHL::API.get_tariff(Requests::TariffMapper[parcel])
mapped_response = Responses::TariffMapper[response]
policy = DHL::Responses::TariffPolicy[response, mapped_response]
return if policy.invalid?
parcel.merge_update!(dhl: mapped_response)
ensure
@errors = policy&.errors
end
end
end
Slide 63
Slide 63 text
32
class DHL::Requests::TariffMapper
extend Dry::Initializer
option :parcel, optional: true
option :origin, default: -> { default_origin }
option :destination, default: -> { default_destination }
option :reference_id, default: -> { default_reference_id }
option :value, default: -> { default_value }
option :weight, default: -> { default_weight }
# other parameters
def call
[
route_attributes,
size_weight_attributes,
package_attributes,
].reduce(:merge)
end
end
Slide 64
Slide 64 text
33
if (service = DHL::Calculator.new(parcel)).call
render json: { tariff: parcel.reload.dhl["tariff"] }
else
render json: { errors: service.errors.to_h }
end
Slide 65
Slide 65 text
34
Dry::Initializer
Tram::Policy
API Sampling
+
+
Slide 66
Slide 66 text
35
Order
Order
Slide 67
Slide 67 text
36
Sagas
Step 2
Slide 68
Slide 68 text
37
Sagas. Orchestration
approach
1. Wrap each mutation in a Command
2.Chain Command
3.Execute chain
4.If one fails run `undo`
Slide 69
Slide 69 text
38
Saga Orchestrator
Parcel
Parcel*
Validate
Validate
Print
Print
…
…
Success
Failure
Slide 70
Slide 70 text
39
Validate PrintLabel
…
Success
Validate#call PrintLabel#call
Slide 71
Slide 71 text
40
Validate PrintLabel
…
Failure
Validate#undo PrintLabel#undo
Slide 72
Slide 72 text
41
Aaaand
that’s it?
Slide 73
Slide 73 text
42
Let’s implement sagas!
Slide 74
Slide 74 text
42
Let’s implement sagas!
Slide 75
Slide 75 text
43
eBaymag
Sagas
Slide 76
Slide 76 text
44
Let’s extract sagas
to a gem!
Slide 77
Slide 77 text
44
Let’s extract sagas
to a gem!
Slide 78
Slide 78 text
44
Let’s extract sagas
to a gem!
Slide 79
Slide 79 text
45
DirtyPipeline
(Sagas)
Slide 80
Slide 80 text
45
DirtyPipeline
(Sagas)
Slide 81
Slide 81 text
46
Slide 82
Slide 82 text
47
Homogeneous
Design
Step 3
Slide 83
Slide 83 text
48
…2 years
of fighting
API issues…
Slide 84
Slide 84 text
49
Functional way
to validation
Slide 85
Slide 85 text
50
Composition
Errors Processing
+
Slide 86
Slide 86 text
51
Algebraic Data Types
module UPS
module Rates
# types product,
# Request passes both RegionalParcel and Schema contracts
RequestPolicy = RegionalPolicy.and_then(SchemaPolicy)
# types sum,
# Error passes either RecoverableInputError or
# InvalidRequest or BasicError contract
ErrorPolicy =
RecoverableInputErrorPolicy
.or(InvalidRequestPolicy)
.or(BasicErrorPolicy)
# combine
ResponsePolicy =
JSONResponsePolicy.and_then(TariffPolicy.or(ErrorPolicy))
end
end
Slide 87
Slide 87 text
52
Refinement Types
— Type (ADT)
— Predicate
Slide 88
Slide 88 text
52
Refinement Types
— Type (ADT)
— Predicate
{ n :: Integer | n > 5 }
Slide 89
Slide 89 text
53
Slide 90
Slide 90 text
54
Slide 91
Slide 91 text
55
Slide 92
Slide 92 text
56
Slide 93
Slide 93 text
57
Context: Why did validation fail?
Contract Failure
Slide 94
Slide 94 text
58
Refinement Type
irb:1> Types::JSONResponse.match(HTTParty.get(“…”))
# => #
# OR #
irb:2> _.unpack
# => {…}
Slide 95
Slide 95 text
59
Refinement Type
module Types
class JSONResponse < BaseType
alias :response :value
def match
context[:body] = response.body
context[:parsed] = ::JSON.parse(response.body)
self
rescue StandardError => error
ContractFailure.new(error, context)
end
def mapped
context[:parsed]
end
end
end
Slide 96
Slide 96 text
60
Refinement Type
module UPS
module Shipping
class InputSchemaType < BaseType
extract :origin
extract :destination
extract :weight
def match
extract!
data = context.slice(*self.class.extractables)
return self if ParcelPolicy[**data].valid?
PolicyFailure.new(policy.errors, context)
end
def mapped
context.slice(*self.class.extractables)
end
end
end
end
Advanced
Slide 97
Slide 97 text
61
class UPS::RatesContract < BaseContract # sampling by types
def match(*input)
input_match = Rates::RequestType.match(*input)
return input_match if input_match.invalid?
result = yield(input_match)
Rates::ResponseType.match(result, input_match.context)
end
end
Slide 98
Slide 98 text
62
module UPS
module Rates
# types product,
# Request passes both RegionalPolicy and SchemaInput
RequestType =
RegionalPolicyType.and_then(InputSchemaType)
# types sum,
# Error passes either RecoverableInputError or
# InvalidRequest or BasicError contract
ErrorType =
RecoverableInputErrorType
.or(InvalidRequestType)
.or(BasicErrorType)
# combine
ResponseType =
Types::JSONResponse.and_then(Types::LogisticsTariff.or(ErrorType))
end
end
Contract Definiton
Slide 99
Slide 99 text
63
class UPS::API
def rates_request(parcel)
RatesContract.match(parcel) do |mapped_parcel|
http_client.get('/1.0/Rates', mapped_parcel.unpack)
end # returns a refined "type"
end
end
Slide 100
Slide 100 text
64
case (match = UPS::API.rate_request(parcel))
when LogisticsTariff
render json: { tariff: match.unpack }
when PolicyFailure
render json: { errors: match.errors }
when InvalidRequest
Honeybadger.notify "Invalid UPS request", context: match.context
render json: { errors: "Unprocessable now, try later" }
when ContractFailure
Honeybadger.notify "Unexpected UPS behavior", context: match.context
render json: { errors: "Unprocessable now, try later" }
else
raise "Invalid contract processing"
end
Slide 101
Slide 101 text
65
Monitoring
module BC
class YabedaInstrument
def call(session)
valid_marker = session.valid? ? "V" : "I"
result = "[#{valid_marker}] #{session.result_type_name}"
Yabeda.api_contract_matches.increment(result: result)
end
end
end
BloodContracts::Instrumentation.configure do |cfg|
# Attach to every BC::Refined ancestor matching UPS.*Contract
cfg.instrument /UPS.*Contract/, BC::YabedaInstrument.new
# Uses Sniffer to record Requests and Responses
cfg.instrument /UPS.*Contract/, BC::SnifferInstrument.new
end
Slide 102
Slide 102 text
65
Monitoring
module BC
class YabedaInstrument
def call(session)
valid_marker = session.valid? ? "V" : "I"
result = "[#{valid_marker}] #{session.result_type_name}"
Yabeda.api_contract_matches.increment(result: result)
end
end
end
BloodContracts::Instrumentation.configure do |cfg|
# Attach to every BC::Refined ancestor matching UPS.*Contract
cfg.instrument /UPS.*Contract/, BC::YabedaInstrument.new
# Uses Sniffer to record Requests and Responses
cfg.instrument /UPS.*Contract/, BC::SnifferInstrument.new
end
Slide 103
Slide 103 text
66
Yabeda & Sniffer
Tram::Policy
BloodContracts
+
+
(on Refinement Types)
Slide 104
Slide 104 text
67
New
Horizons
Outcome
Slide 105
Slide 105 text
68
Evolution
Input/Output
Control
State
Control
Homogeneous
Design
Slide 106
Slide 106 text
69
Future Steps
— Tram::Events
— DirtyPipeline
— BloodContracts docs
Slide 107
Slide 107 text
70
Extra Reading
— Category Theory
for Programmers
— Maybe Haskell
— Chaos Engineering
Slide 108
Slide 108 text
71
Thanks!
@ss_dolganov
@sergey.dolganov
Contracts:
sclinede/blood_contracts
Slide 109
Slide 109 text
71
Thanks!
@ss_dolganov
@sergey.dolganov
Contracts:
sclinede/blood_contracts
Our blog:
evilmartians.com/
chronicles