Events, behaviours and Elixir

Transcript

1. Events, behaviours and Elixir Damian Romanów
2. why events?
3. we all want to create beautiful software
4. so yes, this will be yet another talk about Software Design
5. Module B Module A It always starts simply
6. Module B Module A Module C It always starts simply
7. Module B Module A Module C Module D but it quickly grows
8. Module B Module A Module C Module D things are getting more and more complex
9. Module B Module A Module C Module D and then client wants just a…
10. Module B Module A Module C Module D and then client wants just a… SIMPLE 
 CHANGE
11. Module B Module A Module C Module D which may wreak havoc around your system SIMPLE 
 CHANGE
12. Module B Module A Module C Module D which may wreak havoc around your system SIMPLE 
 CHANGE
13. – Greg Young „Never build big programs. You can literally make a carrier as a consultant, telling people nothing but that.”
14. one crucial word - coupling
15. Module B Module A knowledge about existence of something introduces a logical coupling direct call
16. Sending Email Money Transfer knowledge about existence of something introduces a logical coupling direct call
17. but coupling isn’t a bad thing
18. maybe we just depend on wrong things ?
19. Sending Email Money Transfer depend on abstraction TransferPlaced
20. Sending email Money Transfer depend on abstraction and let your system evolve TransferPlaced
21. Sending email Money Transfer depend on abstraction and let your system evolve TransferPlaced Generate monthly summary
22. Sending email Money Transfer depend on abstraction and let your system evolve TransferPlaced Generate monthly summary Detect fraudulent transaction
23. Problem domain: SportTracker™ It always starts with simple things: • Application for tracking sport activities from various sources • Workout history and summary • Just an MVP
24. Problem domain: SportTracker™ …but it can become complex • Some sort of social aspect - friendship, feed, liking, commenting • Competition aspect - taking part in contests • Maintaining motivation - sending reminders about inactivity • Gamification - user can get badge for running 3 days in a row • Sport performance prediction, based on workout history, ML?
25. Legend depends on module boundaries SP. Tracker SP.Coaching SP.Something SPU.PubSub apps under
 SportsTracker umbrella Important comment a module a module
26. ok, we see that events are decent!
27. we want to have them in our project, we want to build autonomic modules
28. … so it’s the highest time…
29. to install Kafka !!!
30. nope!
31. I know, this one looks amazing,
32. I know, this one looks amazing, …but maybe an ordinary pipe will do the job for now?
33. I know, this one looks amazing, …but maybe an ordinary pipe will do the job for now? and we will spend time and our client money on modeling the problem itself?
34. so let’s follow this path for a while
35. What behaviour we want to have • I want to subscribe on some topic and/or stream • I want to notify subscribers about something
36. What behaviour we want to have • I want to subscribe on some topic and/or stream • I want to notify subscribers about something • So basically a PubSub mechanism
37. Logical diagram Sports Tracker PubSub interface input subscribe
38. Logical diagram Sports Tracker PubSub interface PubSub interface input output subscribe publish
39. Logical diagram Sports Tracker PubSub interface PubSub implementation 1 PubSub implementation 2 PubSub interface PubSub implementation 1 PubSub implementation 2 input output Sports Tracker PubSub interface PubSub interface input output subscribe publish
40. Logical diagram Sports Tracker PubSub interface Registry Kafka PubSub interface Registry Kafka input output Sports Tracker PubSub interface PubSub interface input output subscribe publish
41. Logical diagram Sports Tracker PubSub interface Registry Kafka PubSub interface Registry Kafka my problem domain input output Sports Tracker PubSub interface PubSub interface input output subscribe publish
42. Logical diagram Sports Tracker PubSub interface Registry Kafka PubSub interface Registry Kafka my problem domain implementation detail implementation detail input output Sports Tracker PubSub interface PubSub interface input output subscribe publish
43. it isn’t nothing new in the Phoenix world
44. Logical diagram Sports Tracker PubSub interface PostgreSQL MySQL PubSub interface PostgreSQL MySQL my problem domain implementation detail implementation detail input output Sports Tracker Ecto.Repo Ecto.Repo input output all/one insert
45. let’s define our PubSub abstraction / interface / behaviour
46. but before that, who have ever said that „language is only a tool”?
47. class Event { public String uuid; public int occurredAt; public String topic; }; public interface Subscriber { void handleEvent(Event event); }; public interface PubSub { void publish(Event event); void subscribe(String topic, Subscriber subscriber); }
48. class Event { public String uuid; public int occurredAt; public String topic; }; public interface Subscriber { void handleEvent(Event event); }; public interface PubSub { void publish(Event event); void subscribe(String topic, Subscriber subscriber); } OK I am just playing with you :D
49. defmodule SPU.PubSub.Event do @enforce_keys [:uuid, :topic, :occurred_at, :data] defstruct [:uuid, :topic,:occurred_at, :data] def new(%topic{} = data) do %__MODULE__{ uuid: uuid(), occurred_at: current_timestamp(), topic: topic, data: data } end defp uuid(), do: UUID.uuid4() defp current_timestamp(), do: :os.system_time(:millisecond) end
50. defmodule SPU.PubSub do alias SPU.PubSub.Event @adapter Application.fetch_env!(:spu_pub_sub, :adapter) def publish(%{__struct__: _} = event_data) do event_data |> Event.new() |> @adapter.publish() end def subscribe(subsriber \\ self(), topic) do @adapter.subscribe( subsriber, topic ) end end
51. defmodule SPU.PubSub.Adapter do alias SPU.PubSub.Event @type event :: %Event{} @type subscriber :: pid() | atom() @type topic :: atom() | String.t() @callback start_link() :: :ok @callback publish(event) :: :ok @callback subscribe(subscriber, topic | :all) :: :ok end
52. now we need to implement simple adapter
53. and we can use Elixir Registry for that https://hexdocs.pm/elixir/Registry.html
55. defmodule SPU.PubSub.Adapter.Registry do @behaviour SPU.PubSub.Adapter @registry_name __MODULE__.Reg def start_link(_ \\ []) do import Supervisor.Spec, warn: false children = [ supervisor(Registry, [:duplicate, @registry_name]) ] opts = [strategy: :one_for_one, name: SPU.PubSub.Adapter.Registry] Supervisor.start_link(children, opts) end … end
56. defmodule SPU.PubSub.Adapter.Registry do … alias SPU.PubSub.Event def subscribe(_subscriber, topic) do {:ok, _pid} = Registry.register(@registry_name, topic, []) :ok end def publish(%Event{topic: topic} = event) do @registry_name |> Registry.dispatch(topic, fn subscribers -> for {pid, _} <- subscribers, do: send(pid, event) end) end end
57. then we can use this adapter in config use Mix.Config config :spu_pub_sub, adapter: SPU.PubSub.Adapter.Registry
58. and start it somewhere @adapter Application.fetch_env!(:spu_pub_sub, :adapter) 
 def start(_type, _args) do import Supervisor.Spec children = [ supervisor(@adapter, []) ] opts = [strategy: :one_for_one, name: SPU.PubSub.Supervisor] Supervisor.start_link(children, opts) end
59. that’s all, let’s see how can we use it
60. An example - tracking activities • Athlete (not user :D ) finishes workout and upload gpx file to our backend • This is an example of deep action, looks simple from UI under the hood is complex • After activity is uploaded, various submodules update they state
61. defmodule SP.Tracker.TrackActivity do alias SP.Repo def call(params) do Repo.transaction(fn -> with {:ok, activity} <- insert_activity(params), with {:ok, _notification} <- send_notification(activity), with {:ok, _feed} <- add_to_feed(activity), with {:ok, _summary} <- update_monthly_summary(activity) with {:ok, _athlete} <- recalculate_athlete_performance(activity) with {:ok, _athlete_reminder} <- reset_inactive_reminders(activity) end) end end
62. writing tests for this kind of stuff is horrible
63. we need to prepare very exhausting setup or mock extensively
64. and those tests will change with changing requirements
65. how can we do it better?
66. defmodule SP.Tracker.TrackActivity do alias SP.Repo def call(params) do Repo.transaction(fn -> with {:ok, activity} <- insert_activity(params), with {:ok, _notification} <- send_notification(activity), with {:ok, _feed} <- add_to_feed(activity), with {:ok, _summary} <- update_monthly_summary(activity) with {:ok, _athlete} <- recalculate_athlete_performance(activity) with {:ok, _athlete_reminder} <- reset_inactive_reminders(activity) end) end end
67. defmodule SP.Tracker.TrackActivity do alias SP.Repo def call(params) do Repo.transaction(fn -> with {:ok, activity} <- insert_activity(params), with {:ok, _notification} <- send_notification(activity), with {:ok, _feed} <- add_to_feed(activity), with {:ok, _summary} <- update_monthly_summary(activity) with {:ok, _athlete} <- recalculate_athlete_performance(activity) with {:ok, _athlete_reminder} <- reset_inactive_reminders(activity) end) end end
68. defmodule SP.Tracker.TrackActivity do alias SP.Repo def call(params) do Repo.transaction(fn -> {:ok, activity} = insert_activity(params) :ok <- SP.PubSub.publish(%ActivityTracked{}) end) end end
69. defmodule SP.Feed.Handler do alias SP.Repo def handle_event(%ActivityTracked{activity_id: activity_id, type: type}) do %FeedEntry{activity_id: activity_id, type: type} |> Repo.insert( on_conflict: :replace_all, conflict_target: :id ) end end
70. Integration with events - common pitfalls
71. #1 publishing for one particular consumer
72. defmodule SP.Tracker do def track_activity(athlete_id, gpx_xml) do Multi.new() |> Multi.insert(:activity, %Activity{athlete_id: athlete_id, gpx: gpx_xml}) |> Multi.run(:events, fn _, %{activity: activity} -> %SendEmail{ activity_id: activity.id, athlete_id: athlete_id, type: „AfterActivityMailer”, distance: activity.distance, duration: activity.duration } |> PubSub.publish() end) |> Repo.transaction() end end
73. defmodule SP.Tracker do def track_activity(athlete_id, gpx_xml) do Multi.new() |> Multi.insert(:activity, %Activity{athlete_id: athlete_id, gpx: gpx_xml}) |> Multi.run(:events, fn _, %{activity: activity} -> %SendEmail{ activity_id: activity.id, athlete_id: athlete_id, type: „AfterActivityMailer”, distance: activity.distance, duration: activity.duration } |> PubSub.publish() end) |> Repo.transaction() end end This is not even an event!
74. defmodule SP.Web.Handler do def handle_event(%SendEmail{athlete_id: recipient_id, activity_type: :running, distance: distance}) do email = SP.Web.Mailing.Recipients.email_for(recipient_id) content = SP.Web.Mailing.build_email(:running, distance) SP.Web.Mailing.send_email(email, content) end end
75. it often reveals more serious problem
76. Module B Module A this one is ok
77. Module B Module A how about this?
78. SP. Web SP.Tracker it won’t even compile (as umbrellas)
79. SP. Web SP.Tracker so we cheat a little bit… direct call - track_activity/2
80. SP. Web SP.Tracker so we cheat a little bit… direct call - track_activity/2 an „event”
81. how can we avoid that?
82. describe changes in state, not the consequences
83. don’t tell others what to do, let them decide
84. use past tense
85. #2 failing event handlers
86. event is a fact, you can’t deny it happen
87. defmodule SP.ActivityLog.Handler do alias SP.Repo def handle_event(%ActivityTracked{activity_id: activity_id, type: :running}) do IO.outs("Great work dude!!!!oneone") :ok end end
88. defmodule SP.ActivityLog.Handler do alias SP.Repo def handle_event(%ActivityTracked{activity_id: activity_id, type: :running}) do IO.outs("Great work dude!!!!oneone") :ok end def handle_event(%ActivityTracked{activity_id: activity_id, type: :swimming}) do raise ArgumentError, "Lol, u can't even swim man!!!!" end end
89. defmodule SP.ActivityLog.Handler do alias SP.Repo def handle_event(%ActivityTracked{activity_id: activity_id, type: :running}) do IO.outs("Great work dude!!!!oneone") :ok end def handle_event(%ActivityTracked{activity_id: activity_id, type: :swimming}) do raise ArgumentError, "Lol, u can't even swim man!!!!" end end Lol Handler, u know nothing about me!
90. #3 unawareness of message delivery guarantees
91. at most once delivery vs at least once delivery
92. at most once delivery vs at least once delivery
93. key word - idempotency
94. defmodule SP.ActivityLog.Handler do alias SP.Repo def handle_event(%ActivityTracked{activity_id: activity_id, type: type}) do %Activity{activity_id: activity_id, type: type} |> Repo.insert( on_conflict: :replace_all, conflict_target: :id ) end end
95. #4 using events for modeling synchronous, strong consistent process
96. publishing events inverts the control flow
97. … so we know less about whole process
98. and sometimes we want to govern the whole thing
99. defmodule SP.Coaching.Coach do def go_for_vacation(coach_id, replacement_couch_id, period) do coach_assignments = Assignment |> where([a], a.coach_id == ^couch_id) Ecto.Multi.new() |> Ecto.insert(:vacation, %Vacation{coach_id: coach_id, period: period}) |> Ecto.update_all(:assignments, coach_assignments, set: [coach_id: replacement_couch_id, period: period]) |> Repo.transaction() end end
100. What this approach can give you: • Good reflection of business domain in code • Form of system documentation • Promote open-close principle • Force you to think about consistency boundaries
101. thank you!