Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

2 Ruby Devs ! Tests

Slide 3

Slide 3 text

How do you write tests?

Slide 4

Slide 4 text

palkan_tula palkan 4 Martin Fowler, Mocks Aren't Stubs

Slide 5

Slide 5 text

palkan_tula palkan 5 Classical def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end

Slide 6

Slide 6 text

palkan_tula palkan 5 Classical def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end user = create(:user, name: "Vova") expect(search("name=Vova")).to eq([user])

Slide 7

Slide 7 text

palkan_tula palkan Mockist 5 def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end expect(Query) .to receive(:build).with("name=x") .and_return(double(to_sql: :cond)) expect(User).to receive(:find_by_sql).with(:cond) .and_return(:users) expected(search("user=x")).to eq :users

Slide 8

Slide 8 text

palkan_tula palkan To mock or not to mock? Emily Samp @ RubyConf, 2021 6

Slide 9

Slide 9 text

palkan_tula palkan Classical 7

Slide 10

Slide 10 text

palkan_tula palkan 8 Mockist

Slide 11

Slide 11 text

palkan_tula palkan 9 Mocklassicism

Slide 12

Slide 12 text

palkan_tula palkan False positive 10 def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end expect(Query) .to receive(:build).with("name=x") .and_return(double(to_sql: :cond)) expect(User).to receive(:find_by_sql).with(:cond) .and_return(:users) expected(search("user=x")).to eq :users module Query - def self.build(query_str) - query_hash = Hash[query.split("=")] + def self.build(query_hash) # query has to sql end

Slide 13

Slide 13 text

palkan_tula palkan How to avoid false positives? 11

Slide 14

Slide 14 text

palkan_tula palkan 11 How to put seams? ?

Slide 15

Slide 15 text

palkan_tula palkan github.com/palkan 12

Slide 16

Slide 16 text

13

Slide 17

Slide 17 text

14

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Keeping mocks in line with real objects 16

Slide 20

Slide 20 text

palkan_tula palkan 17 Case study: anyway_config class RubyConfConfig < Anyway::Config attr_config :city, :date coerce_types date: :date end github.com/palkan/anyway_config

Slide 21

Slide 21 text

18 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby -r rubyconf_conf.rb -e "pp RubyConfConfig.new" # "Providence" (type=env key=RUBYCONF_CITY), date => # (type=env key=RUBYCONF_DATE)> # "Providence" (type=env key=RUBYCONF_CITY), date => # (type=env key=RUBYCONF_DATE)>

Slide 22

Slide 22 text

palkan_tula palkan 19 ENV parser class Anyway::Env def fetch(prefix) # Scans ENV and parses matching values some_hash end def fetch_with_trace(prefix) [fetch(prefix), traces[prefix]] end end

Slide 23

Slide 23 text

palkan_tula palkan 20 ENV loader class Anyway::Loaders::Env def call(env_prefix:, **_options) env = Anyway::Env.new env.fetch_with_trace(env_prefix) .then do |(data, trace)| Tracing.current_trace&.merge!(trace) data end end end •

Slide 24

Slide 24 text

21 $ bundle exec rspec --tags env Run options: include {:env=>true} Randomized with seed 26894 ........ Finished in 0.00641 seconds (files took 0.27842 seconds to load) 8 examples, 0 failures Coverage is at 100.0%. Coverage report sent to Coveralls.

Slide 25

Slide 25 text

palkan_tula palkan 22 ENV loader describe Anyway::Loaders::Env do subject { described_class.call(env_prefix: "TESTO") } it "loads data from env" do with_env( "TESTO_A" => "x", "TESTO_DATA__KEY" => "value" ) do expect(subject) .to eq({"a" => "x", "data" => { "key" => "value"}}) end end end Aren't we testing ENV parser here, not loader?

Slide 26

Slide 26 text

palkan_tula palkan describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data, nil]) end it "loads data from Anyway::Env" do expect(subject) .to eq({"a" => "x", "data" => {"key" => "value"}}) end end describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data, nil]) end it "loads data from Anyway::Env" do expect(subject) .to eq({"a" => "x", "data" => {"key" => "value"}}) end end 23 Double-factoring

Slide 27

Slide 27 text

palkan_tula palkan describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data end it "loads data from Anyway::Env" do expect(subject) describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data end it "loads data from Anyway::Env" do expect(subject) 23 Double-factoring

Slide 28

Slide 28 text

Case #1. Undefined method 24

Slide 29

Slide 29 text

palkan_tula palkan 25 Refactor class Anyway::Env - def fetch(prefix) + def fetch(prefix, include_trace = false) @@ ... - def fetch_with_trace(prefix) - [fetch(prefix), traces[prefix]] - end end

Slide 30

Slide 30 text

palkan_tula palkan 26 describe Anyway::Loaders::Env do let(:env) { double("env") } before do allow(::Anyway::Env) .to receive(:new).and_return(env) allow(env) .to receive(:fetch_with_trace) .and_return([data, nil]) end # ... end Refactor

Slide 31

Slide 31 text

27 $ bundle exec rspec --tags env Run options: include {:env=>true} Randomized with seed 26894 ........ Finished in 0.00594 seconds (files took 0.21516 seconds to load) 8 examples, 0 failures Coverage is at 100.0%. Coverage report sent to Coveralls.

Slide 32

Slide 32 text

28 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby -r rubyconf_conf.rb -e "pp RubyConfConfig.new" # "Providence" (type=env key=RUBYCONF_CITY), date => # (type=env key=RUBYCONF_DATE)> anyway_config/lib/anyway/loaders/env.rb:11:in `call': undefined method `fetch_with_trace' for # (NoMethodError) env.fetch_with_trace(env_prefix).then do |(conf, trace)| ^^^^^^^^^^^^^^^^^

Slide 33

Slide 33 text

palkan_tula palkan Double trouble — Tests are green ✅ — Coverage 100% # — Code doesn't work ❌ 29

Slide 34

Slide 34 text

30

Slide 35

Slide 35 text

palkan_tula palkan 31 github.com/rspec/rspec-mocks/issues/227

Slide 36

Slide 36 text

palkan_tula palkan 32 github.com/xaviershay/rspec-fire

Slide 37

Slide 37 text

palkan_tula palkan 33 describe Anyway::Loaders::Env do - let(:env) { double("env") } - let(:env) { instance_double("Anyway::Env") } Refactor

Slide 38

Slide 38 text

34 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads data from Anyway::Env Failure/Error: allow(env_double).to receive(:fetch_with_trace).and_return([env, nil]) the Anyway::Env class does not implement the instance method: fetch_with_trace # ./spec/loaders/env_spec.rb:22:in `block (2 levels) in '

Slide 39

Slide 39 text

palkan_tula palkan 35 double No strings attached

Slide 40

Slide 40 text

palkan_tula palkan verified double 35 double No strings attached Method existence

Slide 41

Slide 41 text

Case #2. Incorrect method signature 36

Slide 42

Slide 42 text

palkan_tula palkan 37 Refactor class Anyway::Env - def fetch(prefix, include_trace = false) + def fetch(prefix, include_trace: false) @@ ...

Slide 43

Slide 43 text

palkan_tula palkan 37 Refactor class Anyway::Env - def fetch(prefix, include_trace = false) + def fetch(prefix, include_trace: false) @@ ... describe Anyway::Loaders::Env do let(:env) { instance_double("env") } before do expect(env) .to receive(:fetch) .with("TESTO", true) .and_return([data, nil]) end # ... end

Slide 44

Slide 44 text

38 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads data from Anyway::Env Failure/Error: env.fetch(env_prefix, true) ArgumentError: Wrong number of arguments. Expected 1, got 2. # ./lib/anyway/loaders/env.rb:11:in `call' # ./lib/anyway/loaders/base.rb:10:in `call' # ./spec/loaders/env_spec.rb:18:in

Slide 45

Slide 45 text

palkan_tula palkan 39 class MethodSignatureVerifier def initialize(signature, args=[]) # ... end def valid? missing_kw_args.empty? && invalid_kw_args.empty? && valid_non_kw_args? && arbitrary_kw_args? && unlimited_args? end end rspec/support/method_signature_verifier.rb

Slide 46

Slide 46 text

palkan_tula palkan 39 rspec/support/method_signature_verifier.rb class MethodSignature def initialize(method) # ... classify_parameters end def classify_parameters @method.parameters.each do |(type, name)| # ... end end end

Slide 47

Slide 47 text

palkan_tula palkan class MethodSignature def initialize(method) # ... classify_parameters end def classify_parameters @method.parameters.each do |(type, name)| # ... end end end 40 rspec/support/method_signature_verifier.rb

Slide 48

Slide 48 text

palkan_tula palkan 41 Method#parameters method_obj = Anyway::Env.instance_method(:fetch) method_obj.parameters #=> [ [:req, :prefix], [:key, :include_trace] ]

Slide 49

Slide 49 text

palkan_tula palkan 42 double verified double No strings attached Method existence Method parameters Method signature?

Slide 50

Slide 50 text

palkan_tula palkan Method signature — Parameters shape — Argument types — Return value type 43

Slide 51

Slide 51 text

palkan_tula palkan 44 Refactor class Anyway::Env + Parsed = Struct.new(:data, :trace) def fetch(prefix, include_trace: false) @@ ... - [data, trace] + Parsed.new(data, trace) end

Slide 52

Slide 52 text

palkan_tula palkan 44 Refactor class Anyway::Env + Parsed = Struct.new(:data, :trace) def fetch(prefix, include_trace: false) @@ ... - [data, trace] + Parsed.new(data, trace) end describe Anyway::Loaders::Env do let(:env) { instance_double("env") } before do expect(env) .to receive(:fetch) .with("TESTO", include_trace: true) .and_return([data, nil]) end # ... end

Slide 53

Slide 53 text

45 $ bundle exec rspec --tags env Run options: include {:env=>true} Randomized with seed 43108 ........ Finished in 0.0234 seconds (files took 0.80197 seconds to load) 7 examples, 0 failures

Slide 54

Slide 54 text

46 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby -r rubyconf_conf.rb -e "pp RubyConfConfig.new" # "Providence" (type=env key=RUBYCONF_CITY), date => # (type=env key=RUBYCONF_DATE)> anyway_config/lib/anyway/tracing.rb:59:in `merge!': undefined method `trace?' for nil:NilClass (NoMethodError) from anyway_config/lib/anyway/loaders/env.rb:12:in `block in call' from :124:in `then' from anyway_config/lib/anyway/loaders/env.rb:11:in `call'

Slide 55

Slide 55 text

palkan_tula palkan “Test doubles are sweet for isolating your unit tests, but we lost something in the translation from typed languages. Ruby doesn't have a compiler that can verify the contracts being mocked out are indeed legit.” –rspec-fire's Readme 47

Slide 56

Slide 56 text

Mocks vs. types 48

Slide 57

Slide 57 text

palkan_tula palkan .rbs 49 double verified double No strings attached Method existence Method parameters

Slide 58

Slide 58 text

palkan_tula palkan typed double 49 double verified double No strings attached Method existence Method parameters Method signature

Slide 59

Slide 59 text

palkan_tula palkan typed_double — Intercept mocked calls — Type check them—that's it! 50

Slide 60

Slide 60 text

palkan_tula palkan 51 Interception RSpec::Mocks::VerifyingMethodDouble.prepend( Module.new do def proxy_method_invoked(obj, *args, &block) super.tap { TypedDouble.typecheck!(obj, *args) } end end )

Slide 61

Slide 61 text

palkan_tula palkan 52 RBS::Test

Slide 62

Slide 62 text

palkan_tula palkan 53 anyway_config.rbs module Anyway class Env class Parsed attr_reader data: Hash attr_reader trace: Tracing::Trace? end def fetch: (String prefix, ?include_trace: bool) -> Parsed end end

Slide 63

Slide 63 text

54 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads data from Anyway::Env Failure/Error: raise RBS::Test::Tester::TypeError.new(errors) unless errors.empty? RBS::Test::Tester::TypeError: TypeError: [Anyway::Env#fetch] ReturnTypeError: expected `::Anyway::Env::Parsed` but returns `[{"a"=>"x", "data"=>{"key"=>"value"}}, nil]`

Slide 64

Slide 64 text

palkan_tula palkan What if... we don't have types % 55

Slide 65

Slide 65 text

On-the-fly type signatures generation 56

Slide 66

Slide 66 text

palkan_tula palkan Type generators — rbs prototype / tapioca — TypeProfiler — Tracing → signatures & 57

Slide 67

Slide 67 text

palkan_tula palkan On-the-fly types — Collect method calls made on real objects 58

Slide 68

Slide 68 text

palkan_tula palkan 59 Tracking calls TracePoint.trace(:call, :return) do |tp| next unless trackable?(tp.defined_class, tp.method_id) target, mid = tp.defined_class, tp.method_id if tp.event == :call method = tp.self.method(mid) args = [] kwargs = {} method.parameters.each do |(type, name)| val = tp.binding.local_variable_get(name) # ... end store[target][mid] << CallTrace.new(arguments: args, kwargs:) elsif tp.event == :return store[target][mid].last.return_value = tp.return_value end end

Slide 69

Slide 69 text

palkan_tula palkan 60 Tracking calls if tp.event == :call method = tp.self.method(mid) args = [] kwargs = {} method.parameters.each do |(type, name)| val = tp.binding.local_variable_get(name) # ... end

Slide 70

Slide 70 text

palkan_tula palkan On-the-fly types — Collect method calls made on real objects — Generate types from real call traces 61

Slide 71

Slide 71 text

palkan_tula palkan 62 SignatureGenerator class SignatureGenerator def to_rbs = [header, method_sigs, footer].join("\n") def args_sig(args) args.transpose.map do |arg_values| arg_values.map(&:class).uniq.map do "::#{_1.name}" end end.join(", ") end end

Slide 72

Slide 72 text

palkan_tula palkan 63 env.rbs module Anyway class Env def initialize: (?type_cast: (::Module)) -> (void) def fetch: ( ::String, ?include_trace: (::FalseClass | ::TrueClass) ) -> ::Anyway::Env::Parsed end end

Slide 73

Slide 73 text

palkan_tula palkan On-the-fly types — Collect method calls made on real objects — Generate types from real call traces — Identify tracing targets (mocked classes) & 64

Slide 74

Slide 74 text

Mock fixtures 65

Slide 75

Slide 75 text

palkan_tula palkan Fixturama evilmartians.com/chronicles/a-fixture-based-approach-to-interface-testing-in-rails 66

Slide 76

Slide 76 text

palkan_tula palkan 67 Fixturama github.com/nepalez/fixturama # fixtures/stubs/notifier.yml --- - class: Notifier chain: - create arguments: - :profileDeleted - <%= profile_id %> actions: - return: true - raise: ActiveRecord::RecordNotFound arguments: - "Profile with id: 1 not found" # for error message

Slide 77

Slide 77 text

palkan_tula palkan YAML fixtures — YAML != Ruby, hard to mock with non- primitive types — Existing mocks are not re-usable—a lot of refactoring 68

Slide 78

Slide 78 text

palkan_tula palkan 69 Mock context mock_context "Anyway::Env" do before do env_double = instance_double("Anyway::Env") allow(Anyway::Env).to receive(:new).and_return(env_double) data = {"a" => "x", "data" => {"key" => "value"}} allow(env_double) .to receive(:fetch).with("TESTO", any_args) .and_return([data, nil]) end end

Slide 79

Slide 79 text

palkan_tula palkan Mock context — Just a shared context — Evaluated within a "scratch" example group on initialization to collect information about stubbed classes and methods 70

Slide 80

Slide 80 text

palkan_tula palkan 71 Mock context def evaluate_context!(context_id, tracking) Class.new(RSpec::Core::ExampleGroup) do include_context(context_id) specify("true") { expect(true).to be(true) } after do RSpec::Mocks.space.proxies.values.each do tracking.register_from_proxy(_1) end end end.run end

Slide 81

Slide 81 text

palkan_tula palkan 72 Refactor describe Anyway::Loaders::Env do - let(:env) { instance_double(Anyway::Env) } + include_mock_context "Anyway::Env" @@ ...

Slide 82

Slide 82 text

palkan_tula palkan 73 RSpec post-check config.after(:suite) do TypedDouble.infer_types_from_calls!( CallsTracer.calls ) passed = MocksTracer.calls.all do TypedDouble.typecheck(_1) end unless passed exit(RSpec.configuration.failure_exit_code) end end

Slide 83

Slide 83 text

palkan_tula palkan 74 Mocked objects finder Mocked calls collector Real calls collector Type checker Types generator after(:suite) before(:suite) run time

Slide 84

Slide 84 text

Case #3. Non-matching behaviour 75

Slide 85

Slide 85 text

palkan_tula palkan 76 Refactor class Anyway::Env def fetch(prefix, **) + return if prefix.empty? @@ ...

Slide 86

Slide 86 text

palkan_tula palkan 76 Refactor class Anyway::Env def fetch(prefix, **) + return if prefix.empty? @@ ... mock_context "Anyway::Env" do before do # ... allow(env_double) .to receive(:fetch) .with("", any_args) .and_return( Anyway::Env::Parsed.new({}, nil)) end end

Slide 87

Slide 87 text

palkan_tula palkan 77 env.rbs module Anyway class Env def fetch: ( ::String, ?include_trace: (::FalseClass | ::TrueClass) ) -> ::Anyway::Env::Parsed? end end We cannot specify which string causes the return value to be nil

Slide 88

Slide 88 text

palkan_tula palkan Double contract — Stubs represent contracts — Tests using real objects MUST verify contracts (unit, end-to-end) 78

Slide 89

Slide 89 text

palkan_tula palkan 79 github.com/psyho/bogus github.com/robwold/compact

Slide 90

Slide 90 text

palkan_tula palkan Stubs → Contracts — Collect method stub expected arguments — Check that a real call with the matching arguments was made and its return type matches the mocked one 80

Slide 91

Slide 91 text

palkan_tula palkan 81 Stubs → Contracts allow(env_double).to receive(:fetch) .with("", any_args) .and_return(Anyway::Env::Parsed.new({})) Anyway::Env#fetch: ("", _) -> Anyway::Env::Parsed verification pattern

Slide 92

Slide 92 text

$ bundle exec rspec --tags env Mocks contract verifications are missing: No matching call found for: Anyway::Env#fetch: ("", _) -> Anyway::Env::Parsed Captured calls: ("", _) -> NilClass 82

Slide 93

Slide 93 text

palkan_tula palkan 83 Mocked objects finder Mocked calls collector Real calls collector Type checker Types generator Call patterns verifier after(:suite) before(:suite) run time

Slide 94

Slide 94 text

Are we there yet? 84

Slide 95

Slide 95 text

palkan_tula palkan Limitations — TracePoint could affect performance (~30 50% overhead) 85

Slide 96

Slide 96 text

palkan_tula palkan TP alternatives — Module#prepend — Source rewriting (Ruby Next ⏩) 86

Slide 97

Slide 97 text

palkan_tula palkan Limitations — TracePoint could affect performance — Parallel builds support is tricky — Verification patterns for non-value objects 87

Slide 98

Slide 98 text

To seam or not to seam? 88

Slide 99

Slide 99 text

palkan_tula palkan “Slow and reliable tests are in general much better than fast tests that break without reason (false positives) and don't catch actual breakage (false negatives).” –Jeremy Evans, Polished Ruby programming 89

Slide 100

Slide 100 text

palkan_tula palkan “If you track your test coverage, try for 100% coverage before integrations tests. Then keep writing integration tests until you sleep well at night.” –Active Interactor's Readme 90

Slide 101

Slide 101 text

palkan_tula palkan Keep it real — Integration tests are the best seams 91

Slide 102

Slide 102 text

palkan_tula palkan Keep it close to real — Integration tests are the best seams — Know your doubles 92

Slide 103

Slide 103 text

93 $ stree search ~/dev/double_query.txt spec/**/*.rb spec/broadcast/redis_spec.rb:15 4: allow(Redis).to receive(:new) { redis_conn } spec/broadcast/redis_spec.rb:38 6: allow(redis_conn).to receive(:publish) spec/broadcast/nats_spec.rb:14 4: allow(NATS::Client).to receive(:new) { nats_conn } spec/broadcast/redis_spec.rb:52 6: allow(redis_conn).to receive(:publish) spec/broadcast/nats_spec.rb:15 4: allow(nats_conn).to receive(:connect) spec/broadcast/http_spec.rb:48 6: allow(AnyCable.logger).to receive(:error) spec/anycable_spec.rb:77 16: adapter = double("adapter", broadcast: nil) spec/broadcast/http_spec.rb:86 8: allow(adapter).to receive(:sleep) spec/broadcast/nats_spec.rb:38 6: allow(nats_conn).to receive(:publish) spec/broadcast/http_spec.rb:87 8: allow(AnyCable.logger).to receive(:error)

Slide 104

Slide 104 text

palkan_tula palkan 94 Syntax Tree Search CallNode[ receiver: NilClass, message: Ident[value: "double" | "instance_double"], arguments: ArgParen[ arguments: Args[ parts: [StringLiteral | VarRef[value: Const], BareAssocHash] ] ] ] | Command[ message: Ident[value: "double" | "instance_double"], arguments: Args[ parts: [StringLiteral | VarRef[value: Const], BareAssocHash] ] ] | # ... bit.ly/stree-doubles

Slide 105

Slide 105 text

palkan_tula palkan Keep it close to real 95 — Integration tests are the best seams — Know your doubles — Fixturize your doubles — Embrace types

Slide 106

Slide 106 text

palkan_tula palkan 96 Mock Suey github.com/test-prof/mock-suey gem "mock-suey"

Slide 107

Slide 107 text

Thanks! @palkan @palkan_tula evilmartians.com @evilmartians