Beyond Mocks - Messing with Our Preconceptions of Testing

In this session, “Beyond Mocks: Messing with Our Preconceptions of Testing,” we will experiment with a different approach to testing in Elixir that steps away from traditional reliance on mocks. If you’ve ever been frustrated by broken tests that break every time you refactor, this talk might offer you a fresh perspective.

We’ll begin by discussing the challenges of using test mocks and how they can sometimes lead to unreliable test results. From there, I’ll introduce you to the Nullable pattern, an experimental technique applied to testing with external dependencies in Elixir. Instead of leaning on test mocks, this technique uses “nullable infrastructure wrappers” as an attempt to simplify dependencies and make your tests more robust and easier to maintain.

I’ll walk you through practical examples of how I tried to apply the patterns of Nullables in Elixir. While this approach is in development, it could offer a new way to think about testing that better aligns with Elixir’s strengths. And perhaps even make you use test mocks with clearer intent.

By the end of our time together, my goal is to encourage you to experiment with your own testing strategies, exploring how alternative techniques might lead to cleaner, more effective code. Let’s dive into this experiment and challenge our preconceptions about testing!

Transcript

1. “We create and perceive our world simultaneously.” — Dom Cobb, Inception
2. @nicholasjhenry Messing with Our Preconceptions of Testing Beyond Mocks
5. The Impact of Infrastructure Services
6. The Impact of Infrastructure Services • Unreliable test cases
7. The Impact of Infrastructure Services • Unreliable test cases • Slow execution
8. The Impact of Infrastructure Services • Unreliable test cases • Slow execution • Expensive to run
9. Test Doubles
10. Test Doubles • Test-specific equivalent
11. Test Doubles • Test-specific equivalent • Hard-coded; vs
12. Test Doubles • Test-specific equivalent • Hard-coded; vs • Configurable
13. Test Mocks
14. Test Mocks • Different experience than our usual test flow
15. Test Mocks • Different experience than our usual test flow • From sociable, state- based to solitary, interaction-based
16. Test Mocks • Different experience than our usual test flow • From sociable, state- based to solitary, interaction-based • Design fatigue
17. What is the alternative to testing with mocks?
18. Level 2: Nullables Unreliable Test Results
19. Nullables
20. Chain of Dependencies
21. Weather example
22. MyUmbrella: an example
23. # terminal 1 ❯❯❯ export OPEN_WEATHER_MAP_APP_ID=$(op read op:/#some-vault/app-key) ❯❯❯ mix server 09:43:10.195 [info] Plug now running on localhost:4000
24. # terminal 1 ❯❯❯ export OPEN_WEATHER_MAP_APP_ID=$(op read op:/#some-vault/app-key) ❯❯❯ mix server 09:43:10.195 [info] Plug now running on localhost:4000 # terminal 2 ❯❯❯ http “http:/#localhost:4000/?lat=28.5383&lon=-81.3792” # orlando HTTP/1.1 200 OK cache-control: max-age=0, private, must-revalidate content-type: text/plain; charset=utf-8 date: Thu, 29 Aug 2024 13:43:14 GMT server: Cowboy Yes, definitey! # or No, not today!
26. What to expect today?
27. What to expect today? • An experiment
28. What to expect today? • An experiment • A mental model of working with Infrastructure
29. What to expect today? • An experiment • A mental model of working with Infrastructure • You might learn something new about Elixir
30. What to expect today? • An experiment • A mental model of working with Infrastructure • You might learn something new about Elixir • Better understanding of test mocks
31. What to expect today? • An experiment • A mental model of working with Infrastructure • You might learn something new about Elixir • Better understanding of test mocks • Mess with your preconceptions of testing
32. Level 2: Nullables Unreliable Test Results
40. Test Type Test Case How Impact Duration Solitary Isolate the SUT Change in wrapper invalidates the mock False Pass Persistent Interaction-based Verify Behaviour Change in wrapper AND SUT invalidates the mock False Failure Transient
41. Test Type Test Case How Impact Duration Solitary Isolate the SUT Change in wrapper invalidates the mock False Pass Persistent Interaction-based Verify Behaviour Change in wrapper AND SUT invalidates the mock False Failure Transient
43. - + - defmodule MyUmbrella.WeatherApi.Behaviour do alias MyUmbrella.WeatherApi.Response @type coordinates :( {float(), float()} @callback get_forecast(coordinates, duration :( :today, URI.t()) :( {:ok, Response.t()} | {:error, term} {:ok, jason :( String.t()} | {:error, term} end
44. - + defmodule MyUmbrella.WeatherApi.Http do @behaviour MyUmbrella.WeatherApi.Behaviour @impl true def get_forecast(coordinates, :today, base_url) do url = build_url(coordinates, base_url) case HTTPoison.get(url) do {:ok, %HTTPoison.Response{status_code: 200} = response} -* {:ok, Jason.decode!(response.body)} {:ok, response.body} {:ok, %HTTPoison.Response{status_code: status_code}} -* {:error, {:status, status_code}} end end end
45. - + defmodule MyUmbrella.WeatherApi.HttpTest do describe "getting a forecast" do test "handles a response with a success status code” do london = Coordinates.new(51.5098, -0.118) result = WeatherApi.get_forecast(london, :today, test_server_url) assert {:ok, data} = result assert %{"current" =, _, "hourly" =, _} = data assert %{"current" =, _, "hourly" =, _} = Jason.decode!(data) end end end
47. iex(1)> MyUmbrella.for_today({28.5383, 81.3792}) *. (FunctionClauseError) no function clause matching in Response.to_weather_report/1 The following arguments were given to MyUmbrella.WeatherApi.Response.to_weather_r # 1 "{\"lat\":28.5383,\"lon\":81.3792,\"timezone\":\"Asia/Kathmandu\"",<0 ..2 Attempted function clauses (showing 1 out of 1): def to_weather_report(%{"current" =, current, "hourly" =, hourly} = response) (my_umbrella 0.1.0) lib/my_umbrella/weather_api/response.ex:25: (my_umbrella 0.1.0) lib/my_umbrella.ex:15: MyUmbrella.for_today/2 iex:1: (file)
48. ❯❯❯ mix test Compiling 3 files (.ex) Running ExUnit with seed: 169237, max_cases: 24 Excluding tags: [:exclude] .................... Finished in 0.3 seconds (0.2s async, 0.1s sync) 20 tests, 0 failures ❯❯❯ mix dialyzer Compiling 2 files (.ex) Finding suitable PLTs Checking PLT..2 Total errors: 0, Skipped: 0, Unnecessary Skips: 0 done in 0m6.13s done (passed successfully) ! !
49. defmodule MyUmbrella.WeatherApi do @behaviour MyUmbrella.WeatherApi.Behaviour @weather_api_module Application.compile_env( :my_umbrella, :weather_api_module, MyUmbrella.WeatherApi.Http ) @impl true def get_forecast(coordinates, duration, url \\ nil), do: @weather_api_module.get_forecast(coordinates, duration, url) end " Dialyzer Dynamic invocation Reference: https://elixirforum.com/t/does-dialyzer-check-behaviours-and-callbacks/2101/6
50. test "given it IS raining before end-of-day; then an umbrella IS needed" do london = Coordinates.new(51.5098, -0.118) current_date_time_utc = DateTime.new!(~D[2000-01-01], ~T[21:30:00Z], "Etc/UTC") expect(MyUmbrella.WeatherApi.Mock, :get_forecast, fn coordinates, :today, _test_server_url -* response = %{ "lat" =, lat, "lon" =, lon, "timezone" =, "Etc/UTC", "current" =, %{ "dt" =, 946_762_200, "weather" =, [ %{ "id" =, 802, "main" =, "Clouds", "description" =, "scattered clouds" } ] Keeping in sync with the replaced dependency
52. Nullables
53. Nullable Infrastructure Wrappers AKA Nullables
54. Nullables is an “off-switch” for production code
55. Nullables and Elixir
56. Nullables and Elixir • Object Oriented to Functional?
57. Nullables and Elixir • Object Oriented to Functional? • Dependency Injection?
58. Nullables and Elixir • Object Oriented to Functional? • Dependency Injection? • Align with Elixir?
59. Nullable Patterns
60. Nullable Patterns • Low-level Infrastructure Wrapper
61. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub
62. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub • Configurable Responses
63. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub • Configurable Responses • Output Tracking
64. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub • Configurable Responses • Output Tracking
66. Low-level Infrastructure Wrapper
67. Low-level Infrastructure Wrapper • Simple Interface
68. Low-level Infrastructure Wrapper • Simple Interface • Application Specific
69. Low-level Infrastructure Wrapper • Simple Interface • Application Specific • Own return types
70. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) response = JsonHttp.Response.new( status_code: httpoison_response.status_code, headers: httpoison_response.headers, body: Jason.decode!(httpoison_response.body) ) {:ok, response} end end
71. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) response = JsonHttp.Response.new( status_code: httpoison_response.status_code, headers: httpoison_response.headers, body: Jason.decode!(httpoison_response.body) ) {:ok, response} end end
72. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) response = JsonHttp.Response.new( status_code: httpoison_response.status_code, headers: httpoison_response.headers, body: Jason.decode!(httpoison_response.body) ) {:ok, response} end end
73. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) response = JsonHttp.Response.new( status_code: httpoison_response.status_code, headers: httpoison_response.headers, body: Jason.decode!(httpoison_response.body) ) {:ok, response} end end
75. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub • Configurable Responses • Output Tracking
77. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
78. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
79. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
80. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
81. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
82. # defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
83. Dependency Replacement
84. Dependency Replacement
85. Dependency Replacement • Dependency Parameterization
86. Dependency Replacement • Dependency Parameterization • Dependency Injection
87. Dependency Replacement • Dependency Parameterization • Dependency Injection • Application Configuration; see Mox
88. Dependency Replacement • Dependency Parameterization • Dependency Injection • Application Configuration; see Mox • State Management, e.g. Mix.Shell
89. Dependency Replacement • Dependency Parameterization • Dependency Injection • Application Configuration; see Mox • State Management, e.g. Mix.Shell • AST Manipulation, e.g. Rewire
90. defmodule MyUmbrella.Infrastructure.WeatherApi.Client do alias MyUmbrella.Infrastructure.WeatherApi @type t() :( %WeatherApi.Client{ app_id: String.t(), url: String.t() } defstruct [:app_id, :url] @spec new :( t() def new do config = Application.get_env(:my_umbrella, :weather_api) struct!(WeatherApi.Client, config) end @spec get_forecast(t(), coordinates, duration :( :today) :( {:ok, Response.t()} def get_forecast(weather_api_client, coordinates, :today) do end end
91. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @type t() :( %JsonHttp.Client{ httpoison: HTTPoison | JsonHttp.Client.StubbedHTTPoison } @enforce_keys [:httpoison] defstruct [:httpoison] end
92. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp alias MyUmbrella.Infrastructure.JsonHttp.Client.StubbedHTTPoison @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end end
93. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp alias MyUmbrella.Infrastructure.JsonHttp.Client.StubbedHTTPoison @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end end
94. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp alias MyUmbrella.Infrastructure.JsonHttp.Client.StubbedHTTPoison @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end end
95. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @spec get(t(), String.t()) :( {:ok, JsonHttp.Response.t()} def get(http_client, url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_resp} = http_client.httpoison.get(request.url, request.headers) response = JsonHttp.Response.new( status_code: httpoison_res.status_code, headers: httpoison_resp.headers, body: Jason.decode!(httpoison_resp.body) ) {:ok, response} end HTTPoison or StubbedHTTPoison
96. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO {:ok, %HTTPoison.Response{}} end end @spec get(String.t()) :( {:ok, JsonHttp.Response.t()} def get(url) do request = JsonHttp.Request.new(url: url) {:ok, httpoison_response} = HTTPoison.get(request.url, request.headers) {:ok, response} end
97. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub • Configurable Responses • Output Tracking
99. test "configurable response" do url = "http:/#NOT_CONNECTED/get" responses = [ {url, Response.new(status_code: 200, body: %{"hello" =, "world"})} ] http_client = JsonHttp.Client.create_null(responses) result = JsonHttp.Client.get(http_client, url) assert {:ok, response} = result assert response.status_code =4 200 assert {"Content-Type", "application/json; charset=utf-8"} in response.headers assert %{"hello" =, "world"} =4 response.body end Configured response
100. test "configurable response" do url = "http:/#NOT_CONNECTED/get" responses = [ {url, Response.new(status_code: 200, body: %{"hello" =, "world"})} ] http_client = JsonHttp.Client.create_null(responses) result = JsonHttp.Client.get(http_client, url) assert {:ok, response} = result assert response.status_code =4 200 assert {"Content-Type", "application/json; charset=utf-8"} in response.headers assert %{"hello" =, "world"} =4 response.body end Configured response
101. test "configurable response" do url = "http:/#NOT_CONNECTED/get" responses = [ {url, Response.new(status_code: 200, body: %{"hello" =, "world"})} ] http_client = JsonHttp.Client.create_null(responses) result = JsonHttp.Client.get(http_client, url) assert {:ok, response} = result assert response.status_code =4 200 assert {"Content-Type", "application/json; charset=utf-8"} in response.headers assert %{"hello" =, "world"} =4 response.body end Configured response
102. test "configurable response" do url = "http:/#NOT_CONNECTED/get" responses = [ {url, Response.new(status_code: 200, body: %{"hello" =, "world"})} ] http_client = JsonHttp.Client.create_null(responses) result = JsonHttp.Client.get(http_client, url) assert {:ok, response} = result assert response.status_code =4 200 assert {"Content-Type", "application/json; charset=utf-8"} in response.headers assert %{"hello" =, "world"} =4 response.body end Configured response
103. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do # TODO: Return configurable response {:ok, %HTTPoison.Response{}} end end end end Cannot change the signature to configure responses
104. defmodule Nullables.ConfigurableResponses do use Agent @type responses() :( list({term(), term()}) @spec start_link(module(), responses()) :( Agent.on_start() def start_link(module, responses \\ []) do test_pid = self() agent_name = {test_pid, module} Agent.start_link(fn -* responses end, name: agent_name) end end {#PID<0.111.0>, :httpoison}
105. defmodule Nullables.ConfigurableResponses do use Agent @type responses() :( list({term(), term()}) @spec start_link(module(), responses()) :( Agent.on_start() def start_link(module, responses \\ []) do test_pid = self() agent_name = {test_pid, module} Agent.start_link(fn -* responses end, name: agent_name) end end {#PID<0.111.0>, :httpoison}
106. defmodule Nullables.ConfigurableResponses do use Agent @type responses() :( list({term(), term()}) @spec start_link(module(), responses()) :( Agent.on_start() def start_link(module, responses \\ []) do test_pid = self() agent_name = {test_pid, module} Agent.start_link(fn -* responses end, name: agent_name) end end {#PID<0.111.0>, :httpoison}
107. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null(ConfigurableResponses.responses()) :( t() def create_null(responses \\ []) do {:ok, _pid} = ConfigurableResponses.start_link(:httpoison, responses) %JsonHttp.Client{httpoison: StubbedHTTPoison} end end
108. defmodule StubbedHTTPoison do @spec get(String.t(), HTTPoison.headers()) :( {:ok, HTTPoison.Response.t()} def get(url, _headers) do response = url |6 get_response |6 to_httpoison {:ok, response} end defp get_response(url) do uri = URI.parse(url) endpoint = “#8uri.scheme}:/##8uri.host}#8uri.path}" ConfigurableResponses.get_response(:httpoison, endpoint) end
109. defmodule Nullables.ConfigurableResponses do use Agent @spec get_response(module(), term(), term()) :( term() def get_response(module, key, default \\ nil) do agent_name = {test_pid(), module} Agent.get_and_update(agent_name, &pop_response(&1, key)) |9 default end defp test_pid do case Process.get(:"$callers") do [parent_id | _] -* parent_id nil -* self() end end end {#PID<0.111.0>, :httpoison}
110. defmodule Nullables.ConfigurableResponses do use Agent @spec get_response(module(), term(), term()) :( term() def get_response(module, key, default \\ nil) do agent_name = {test_pid(), module} Agent.get_and_update(agent_name, &pop_response(&1, key)) |9 default end defp test_pid do case Process.get(:"$callers") do [parent_id | _] -* parent_id nil -* self() end end end {#PID<0.111.0>, :httpoison}
111. test "configurable response" do url = "http:/#NOT_CONNECTED/get" responses = [ {url, Response.new(status_code: 200, body: %{"hello" =, "world"})} ] http_client = JsonHttp.Client.create_null(responses) result = JsonHttp.Client.get(http_client, url) assert {:ok, response} = result assert response.status_code =4 200 assert {"Content-Type", "application/json; charset=utf-8"} in response.headers assert %{"hello" =, "world"} =4 response.body end
112. Nullable Patterns • Low-level Infrastructure Wrapper • Embedded Stub • Configurable Responses • Output Tracking
114. defmodule Examples.MixShellTest do use ExUnit.Case, async: true setup do Mix.shell(Mix.Shell.Process) on_exit(fn -* Mix.shell(Mix.Shell.IO) end) end test "output tracking" do Mix.shell().info("hello") assert_receive {:mix_shell, :info, [msg]} assert msg =4 "hello" end end
115. defmodule Examples.MixShellTest do use ExUnit.Case, async: true setup do Mix.shell(Mix.Shell.Process) on_exit(fn -* Mix.shell(Mix.Shell.IO) end) end test "output tracking" do Mix.shell().info("hello") assert_receive {:mix_shell, :info, [msg]} assert msg =4 "hello" end end
116. defmodule Examples.MixShellTest do use ExUnit.Case, async: true setup do Mix.shell(Mix.Shell.Process) on_exit(fn -* Mix.shell(Mix.Shell.IO) end) end test "output tracking" do Mix.shell().info("hello") assert_receive {:mix_shell, :info, [msg]} assert msg =4 "hello" end end
117. defmodule Examples.MixShellTest do use ExUnit.Case, async: true setup do Mix.shell(Mix.Shell.Process) on_exit(fn -* Mix.shell(Mix.Shell.IO) end) end test "output tracking" do Mix.shell().info("hello") assert_receive {:mix_shell, :info, [msg]} assert msg =4 "hello" end end
118. defmodule Examples.MixShellTest do use ExUnit.Case, async: true setup do Mix.shell(Mix.Shell.Process) on_exit(fn -* Mix.shell(Mix.Shell.IO) end) end test "output tracking" do Mix.shell().info("hello") assert_receive {:mix_shell, :info, [msg]} assert msg =4 "hello" end end
119. defmodule Examples.Mix.Shell.Process do def info(message) do send(message_target(), {:mix_shell, :info, [message]}) :ok end defp message_target() do case Process.get(:"$callers") do [parent | _] -* parent _ -* self() end end end
120. defmodule Examples.Mix.Shell.Process do def info(message) do send(message_target(), {:mix_shell, :info, [message]}) :ok end defp message_target() do case Process.get(:"$callers") do [parent | _] -* parent _ -* self() end end end
121. defmodule Examples.Mix.Shell.Process do def info(message) do send(message_target(), {:mix_shell, :info, [message]}) :ok end defp message_target() do case Process.get(:"$callers") do [parent | _] -* parent _ -* self() end end end
122. “Program each dependency with a tested, production-grade trackXxx() method that tracks the otherwise-invisible writes. Have it do so regardless of whether the object is Nulled or not.” “Testing without Mocks” - James Shore
123. “Telemetry is a lightweight library for dynamic dispatching of events, with a focus on metrics and instrumentation. Any Erlang or Elixir library can use telemetry to emit events.” — Telemetry documentation
124. defmodule MyUmbrella.Infrastructure.JsonHttp.ClientTest do use MyUmbrella.TestCase, async: true alias MyUmbrella.Infrastructure.JsonHttp alias Nullables.OutputTracking describe "nullability" do test "output tracking", %{test: test} do http_client = JsonHttp.Client.create_null() ref = OutputTracking.track_output(test, [:json_http, :requested]) _result = JsonHttp.Client.get(http_client, "http:/#NOT_CONNECTED/get") assert_received {[:json_http, :requested], ^ref, %JsonHttp.Request{url: "http:/#NOT_CONNECTED/get"}} end end end
125. defmodule MyUmbrella.Infrastructure.JsonHttp.ClientTest do use MyUmbrella.TestCase, async: true alias MyUmbrella.Infrastructure.JsonHttp alias Nullables.OutputTracking describe "nullability" do test "output tracking", %{test: test} do http_client = JsonHttp.Client.create_null() ref = OutputTracking.track_output(test, [:json_http, :requested]) _result = JsonHttp.Client.get(http_client, "http:/#NOT_CONNECTED/get") assert_received {[:json_http, :requested], ^ref, %JsonHttp.Request{url: "http:/#NOT_CONNECTED/get"}} end end end
126. defmodule MyUmbrella.Infrastructure.JsonHttp.ClientTest do use MyUmbrella.TestCase, async: true alias MyUmbrella.Infrastructure.JsonHttp alias Nullables.OutputTracking describe "nullability" do test "output tracking", %{test: test} do http_client = JsonHttp.Client.create_null() ref = OutputTracking.track_output(test, [:json_http, :requested]) _result = JsonHttp.Client.get(http_client, "http:/#NOT_CONNECTED/get") assert_received {[:json_http, :requested], ^ref, %JsonHttp.Request{url: "http:/#NOT_CONNECTED/get"}} end end end
127. defmodule MyUmbrella.Infrastructure.JsonHttp.ClientTest do use MyUmbrella.TestCase, async: true alias MyUmbrella.Infrastructure.JsonHttp alias Nullables.OutputTracking describe "nullability" do test "output tracking", %{test: test} do http_client = JsonHttp.Client.create_null() ref = OutputTracking.track_output(test, [:json_http, :requested]) _result = JsonHttp.Client.get(http_client, "http:/#NOT_CONNECTED/get") assert_received {[:json_http, :requested], ^ref, %JsonHttp.Request{url: "http:/#NOT_CONNECTED/get"}} end end end
128. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do alias MyUmbrella.Infrastructure.JsonHttp alias Nullables.OutputTracking @spec get(t(), String.t()) :( {:ok, JsonHttp.Response.t()} def get(http_client, url) do req = JsonHttp.Request.new(url: url) {:ok, httpoison_resp} = http_client.httpoison.get(req.url, req.headers) :ok = OutputTracking.emit([:http_client, :requested], req) resp = JsonHttp.Response.new( ) {:ok, resp} end end
129. defmodule Nullables.OutputTracking do def track_output(handler_id, event_name) do ref = make_ref() config = %{caller_pid: self(), ref: ref} :ok = :telemetry.attach( handler_id, event_name, &__MODULE__.handle_event/4, config) ref end def handle_event(event_name, _measurements, data, config) do [module, event_type | _] = event_name send(config.caller_pid, {[module, event_type], config.ref, data}) end end
130. defmodule Nullables.OutputTracking do def track_output(handler_id, event_name) do ref = make_ref() config = %{caller_pid: self(), ref: ref} :ok = :telemetry.attach( handler_id, event_name, &__MODULE__.handle_event/4, config) ref end def handle_event(event_name, _measurements, data, config) do [module, event_type | _] = event_name send(config.caller_pid, {[module, event_type], config.ref, data}) end end
131. defmodule Nullables.OutputTracking do def track_output(handler_id, event_name) do ref = make_ref() config = %{caller_pid: self(), ref: ref} :ok = :telemetry.attach( handler_id, event_name, &__MODULE__.handle_event/4, config) ref end def handle_event(event_name, _measurements, data, config) do [module, event_type | _] = event_name send(config.caller_pid, {[module, event_type], config.ref, data}) end end
132. defmodule Nullables.OutputTracking do @spec emit(:telemetry.event_name(), term()) :( :ok def emit(event_name, data) do :telemetry.execute(event_name, %{}, data) end end
133. defmodule MyUmbrella.Infrastructure.JsonHttp.ClientTest do use MyUmbrella.TestCase, async: true alias MyUmbrella.Infrastructure.JsonHttp alias Nullables.OutputTracking describe "nullability" do test "output tracking", %{test: test} do http_client = JsonHttp.Client.create_null() ref = OutputTracking.track_output(test, [:json_http, :requested]) _result = JsonHttp.Client.get(http_client, "http:/#NOT_CONNECTED/get") assert_received {[:json_http, :requested], ^ref, %JsonHttp.Request{url: "http:/#NOT_CONNECTED/get"}} end end end State-based testing
136. Chain of Dependencies
137. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do @type t() :( %JsonHttp.Client{httpoison: HTTPoison | StubbedHTTPoison} @enforce_keys [:httpoison] defstruct [:httpoison] @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end
138. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do @type t() :( %JsonHttp.Client{httpoison: HTTPoison | StubbedHTTPoison} @enforce_keys [:httpoison] defstruct [:httpoison] @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end
139. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do @type t() :( %JsonHttp.Client{httpoison: HTTPoison | StubbedHTTPoison} @enforce_keys [:httpoison] defstruct [:httpoison] @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end
140. defmodule MyUmbrella.Infrastructure.JsonHttp.Client do @type t() :( %JsonHttp.Client{httpoison: HTTPoison | StubbedHTTPoison} @enforce_keys [:httpoison] defstruct [:httpoison] @spec create() :( t() def create do %JsonHttp.Client{httpoison: HTTPoison} end @spec create_null() :( t() def create_null() do %JsonHttp.Client{httpoison: StubbedHTTPoison} end
142. iex> MyUmbrella.create_null %MyUmbrella{ weather_api_client: %MyUmbrella.Infrastructure.WeatherApi.Client{ http_client: %MyUmbrella.Infrastructure.JsonHttp.Client{ httpoison: MyUmbrella.Infrastructure.JsonHttp.Client.StubbedHTTPoison }, app_id: “[REDACTED]” } } Embedded Stub
144. iex> MyUmbrella.create %MyUmbrella{ weather_api_client: %MyUmbrella.Infrastructure.WeatherApi.Client{ http_client: %MyUmbrella.Infrastructure.JsonHttp.Client{ httpoison: HTTPoison }, app_id: “[REDACTED]” } } Production Dependency
145. test "given it IS raining before end-of-day; then an umbrella IS required” do london = CoordinatesFixture.example(:london) current_date_time = DateTimeFixture.Utc.example(:London) responses = %{weather_api_client: ResponseFixture.Success.example(:London)} weather_result = responses |6 MyUmbrella.create_null() |6 MyUmbrella.for_today(london, current_date_time) assert umbrella_required?(weather_result) end
146. test "given it IS raining before end-of-day; then an umbrella IS required” do london = CoordinatesFixture.example(:london) current_date_time = DateTimeFixture.Utc.example(:London) responses = %{weather_api_client: ResponseFixture.Success.example(:London)} weather_result = responses |6 MyUmbrella.create_null() |6 MyUmbrella.for_today(london, current_date_time) assert umbrella_required?(weather_result) end
147. test "given it IS raining before end-of-day; then an umbrella IS required” do london = CoordinatesFixture.example(:london) current_date_time = DateTimeFixture.Utc.example(:London) responses = %{weather_api_client: ResponseFixture.Success.example(:London)} weather_result = responses |6 MyUmbrella.create_null() |6 MyUmbrella.for_today(london, current_date_time) assert umbrella_required?(weather_result) end
148. test "given it IS raining before end-of-day; then an umbrella IS required” do london = CoordinatesFixture.example(:london) current_date_time = DateTimeFixture.Utc.example(:London) responses = %{weather_api_client: ResponseFixture.Success.example(:London)} weather_result = responses |6 MyUmbrella.create_null() |6 MyUmbrella.for_today(london, current_date_time) assert umbrella_required?(weather_result) end
149. test "given it IS raining before end-of-day; then an umbrella IS required” do london = CoordinatesFixture.example(:london) current_date_time = DateTimeFixture.Utc.example(:London) responses = %{weather_api_client: ResponseFixture.Success.example(:London)} weather_result = responses |6 MyUmbrella.create_null() |6 MyUmbrella.for_today(london, current_date_time) assert umbrella_required?(weather_result) end
150. test "given it IS raining before end-of-day; then an umbrella IS required” do london = CoordinatesFixture.example(:london) current_date_time = DateTimeFixture.Utc.example(:London) responses = %{weather_api_client: ResponseFixture.Success.example(:London)} weather_result = responses |6 MyUmbrella.create_null() |6 MyUmbrella.for_today(london, current_date_time) assert umbrella_required?(weather_result) end
151. defmodule MyUmbrella do alias MyUmbrella @type t() :( %MyUmbrella{weather_api_client: WeatherApi.Client.t()} defstruct [:weather_api_client] @spec create() :( t() def create do %MyUmbrella{weather_api_client: WeatherApi.Client.create()} end @spec create_null(%{weather_api_client: WeatherApi.Response.t()}) :( t() def create_null(responses) do null_weather_client = WeatherApi.Client.create_null(%{http_client: responses.weather_api_client}) %MyUmbrella{weather_api_client: null_weather_client} end end
152. defmodule MyUmbrella.Infrastructure.WeatherApi.Client do @spec create() :( t() def create do %WeatherApi.Client{http_client: JsonHttp.Client.create(), app_id: get_app_id()} end @spec create_null(%{http_client: WeatherApi.Response.t()}) :( t() def create_null(responses) do http_client_responses = [{@default_url <0 @request_path, responses.http_client}] %WeatherApi.Client{ http_client: JsonHttp.Client.create_null(http_client_responses), app_id: get_app_id() } end end
154. Messing with preconceptions of testing
155. Messing with preconceptions of testing • “Test code” with your production code
156. Messing with preconceptions of testing • “Test code” with your production code • BUT Use the off-switch as a “dry-run”
157. Messing with preconceptions of testing • “Test code” with your production code • BUT Use the off-switch as a “dry-run” • Up-front investment
158. Messing with preconceptions of testing • “Test code” with your production code • BUT Use the off-switch as a “dry-run” • Up-front investment • BUT a better understanding of the client library and communication protocol
159. Beyond Mocks and Back
160. Test Mocks
161. Test Mocks • Solitary and interaction- based test cases
162. Test Mocks • Solitary and interaction- based test cases • Unreliable test results
163. Test Mocks • Solitary and interaction- based test cases • Unreliable test results • Fall out of sync with the dependency
164. Nullables
165. Nullables • Low-level infrastructure wrappers
166. Nullables • Low-level infrastructure wrappers • Embedded Stubs
167. Nullables • Low-level infrastructure wrappers • Embedded Stubs • Configurable Responses
168. Nullables • Low-level infrastructure wrappers • Embedded Stubs • Configurable Responses • Output Tracking
169. Testing Elixir
170. Testing Elixir • Example: `Mix.Shell`
171. Testing Elixir • Example: `Mix.Shell` • Embedded Stubs: `Mix.Shell.Process`
172. Testing Elixir • Example: `Mix.Shell` • Embedded Stubs: `Mix.Shell.Process` • Output Tracking: `send/2`
173. Take-aways
174. Take-aways • Consider the trade-offs of test mocks
175. Take-aways • Consider the trade-offs of test mocks • Consider lightweight techniques
176. Take-aways • Consider the trade-offs of test mocks • Consider lightweight techniques • Consider your infrastructure wrappers
177. Explore Nullables
178. Explore Nullables • www.jamesshore.com
179. Explore Nullables • www.jamesshore.com • Pattern language
180. Explore Nullables • www.jamesshore.com • Pattern language • Video series
181. Explore Nullables • www.jamesshore.com • Pattern language • Video series • Complete free training course
182. MyUmbrella • github.com/nicholasjhenry/my-umbrella • branch: nullables • branch: test-mocks
183. Thank you @nicholasjhenry