Modular Design in Elixir (ElixirConf EU 2019)

Modular Design in Elixir (ElixirConf EU 2019)

A talk given at ElixirConf EU 2019 in Prague. It talks about splitting big systems into smaller, simpler, more manageable modules that are easier to understand, test and work with.

373dd7c51433dc3c38436dcfdec79cdc?s=128

Maciej Kaszubowski

April 08, 2019
Tweet

Transcript

  1. mkaszubowski94 Maciej Kaszubowski

  2. mkaszubowski94 Maciej Kaszubowski THE BIG BALL OF NOUNS

  3. mkaszubowski94 time complexity & size

  4. mkaszubowski94 915

  5. mkaszubowski94 1556

  6. mkaszubowski94 527

  7. mkaszubowski94 270

  8. mkaszubowski94

  9. mkaszubowski94 4 -7

  10. mkaszubowski94 Maciej Kaszubowski Dealing with complexity

  11. mkaszubowski94 https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Event Sourcing / CQRS

  12. mkaszubowski94 Domain Driven Design

  13. mkaszubowski94 Microservices

  14. mkaszubowski94 They are all about system decomposition

  15. mkaszubowski94 ... and we can do the same in a

    monolith with a single PostgreSQL instance
  16. mkaszubowski94 Maciej Kaszubowski Modular Software Design

  17. mkaszubowski94 1972

  18. mkaszubowski94 one big problem

  19. mkaszubowski94

  20. mkaszubowski94 Modular design smaller, simpler modules strong boundaries low coupling

  21. mkaszubowski94 Each module can be: understood in isolation modified in

    isolation tested in isolation replaced or removed
  22. mkaszubowski94 ok, but how?

  23. mkaszubowski94 Maciej Kaszubowski Example Project

  24. mkaszubowski94 create a new job name image url price publish

    John Alchemist create new job manage jobs find a job logout description date
  25. mkaszubowski94 your jobs John Alchemist create new job manage jobs

    find a job logout my awesome job accept my other job cancel accept cancel yet another one and one more accepted this didn’t go well canceled accept cancel
  26. mkaszubowski94 browse jobs John Alchemist create new job manage jobs

    find a job logout my awesome job my other job yet another one and one more filters 100$ 50$ 3500$ 99$ xxxx yyy zzz search
  27. mkaszubowski94 start this job John Alchemist create new job manage

    jobs find a job logout Job Title I need something done start a job https://example.com/jobs/1234 100$
  28. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  29. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish()
  30. mkaszubowski94 new feature: slugs

  31. mkaszubowski94 start this job John Alchemist create new job manage

    jobs find a job logout my awesome job Don’t ask, you’ll get paid start a job https://example.com/jobs/my-awesome-job
  32. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :price, :integer field :slug, :string field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  33. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug()
  34. mkaszubowski94 new feature: pay before publishing

  35. mkaszubowski94 create a new job name image url price continue

    John Alchemist create new job manage jobs find a job logout description date
  36. mkaszubowski94 pay for job pay 110$ John Alchemist create new

    job manage jobs find a job logout my awesome job job price………….………..100$ fee (10%)…………………….10$ total……………………………..110$ Don’t ask, you’ll get paid Your job will be published after the payment
  37. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :slug, :string field :paid, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  38. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    — accept() - cancel() - publish() - create()
  39. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    — accept() - cancel() - publish() - create()
  40. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create()
  41. mkaszubowski94 open-closed principle

  42. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create()
  43. mkaszubowski94 new feature: job reminders

  44. mkaszubowski94 browse jobs John Alchemist create new job manage jobs

    find a job logout my awesome job my other job yet another one and one more filters 100$ 50$ 3500$ 99$ xxxx yyy zzz search Upcoming job Job starting in 54 minutes ! x
  45. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  46. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  47. mkaszubowski94 JobContext jobs

  48. mkaszubowski94 jobs JobContext Payments preload jobs

  49. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload

    jobs
  50. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload

    jobs Chat Context
  51. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload

    jobs Chat Context JobSubtask Context
  52. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments JobSubtask

    Context preload jobs Chat Context JOB USER CONVERSATION MESSAGE PAYMENT SUBTASK
  53. mkaszubowski94 In most projects: • few nouns / entities •

    a lot of interactions between them • logic is organised around the nouns • most actions represented by updates • a lot of coupling as a result
  54. mkaszubowski94 We’re used to nouns • Object Oriented Programming •

    Databases • REST • CRUD
  55. mkaszubowski94 nouns are not universal

  56. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  57. mkaszubowski94 a “job” means many different things

  58. mkaszubowski94 Maciej Kaszubowski Extracting behaviour

  59. mkaszubowski94 what is the desired outcome?

  60. mkaszubowski94 1. what is the desired outcome?

  61. mkaszubowski94 1. what is the desired outcome? 2. what data

    do I need to do this?
  62. mkaszubowski94 1. what is the desired outcome? 2. what data

    do I need to do this? 3. how can I get this data?
  63. mkaszubowski94 1. what is the desired outcome? 2. what data

    do I need to do this? 3. how can I get this data? 4. what should be exposed in the interface?
  64. mkaszubowski94 Maciej Kaszubowski Extracting Slugs

  65. mkaszubowski94 start this job John Alchemist create new job manage

    jobs find a job logout my awesome job Don’t ask, you’ll get paid start a job https://example.com/jobs/my-awesome-job
  66. mkaszubowski94 desired outcome: jobs should be accessible via human-friendly slugs

  67. mkaszubowski94 required data: id-slug mapping for jobs

  68. mkaszubowski94 SEO Context

  69. mkaszubowski94 SEO Context slugs

  70. mkaszubowski94 SEO Context slugs - register_slug() - get_id_by_slug() - get_slugs_by_ids()

    Interface:
  71. mkaszubowski94 def show(conn, %{"job_id"  slug}) do job = slug

     SeoContext.fetch_id_by_slug("job")  JobContext.fetch_by_id() #  end
  72. mkaszubowski94 defp put_slugs(jobs) do ids = Enum.map(jobs, & &1.id) slugs

    = SeoContext.fetch_slugs_by_ids(ids, "job") Enum.map(jobs, fn job  Map.put(job, :slug, slugs[job.id]) end) end
  73. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  74. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  75. mkaszubowski94 SEO Context • slugs for other resources - easy

    to do
  76. mkaszubowski94 SEO Context • slugs for other resources - easy

    to do • multiple slugs - easy to do
  77. mkaszubowski94 SEO Context • slugs for other resources - easy

    to do • multiple slugs - easy to do • slugs are resolved at the controller level • no slugs in domain code
  78. mkaszubowski94 no slugs in domain logic /api/jobs/my-awesome-job/subtasks from( s in

    Subtask, join: j in Job, on: s.job_id  j.id where: j.id  ^job_id or j.slug  ^job_id, # … ) /api/jobs/1/subtasks
  79. mkaszubowski94 Maciej Kaszubowski Extracting Notifications

  80. mkaszubowski94 browse jobs John Alchemist create new job manage jobs

    find a job logout my awesome job my other job yet another one and one more filters 100$ 50$ 3500$ 99$ xxxx yyy zzz search Upcoming job Job starting in 54 minutes ! x
  81. mkaszubowski94 desired outcome: notification is sent before the start of

    a job
  82. mkaszubowski94 required data: job information

  83. mkaszubowski94 Job Reminders upcoming_jobs

  84. mkaszubowski94 Job Reminders upcoming_jobs - put_job() - remove_job() Interface:

  85. mkaszubowski94 Job Reminders upcoming_jobs - put_job() - remove_job() Interface: It's

    hard to use
  86. mkaszubowski94 Interface: Job Context search Job Reminders sent_reminders

  87. mkaszubowski94 defmodule Notifications.Worker do use GenServer def handle_info(:work, state) do

    Notifications.send_notifications(now()) Process.send_after(self(), :work, @interval) {:noreply, state) end end
  88. mkaszubowski94 defmodule Notifications do def send_notifications(datetime) do jobs = JobContext.search(…)

    already_sent = fetch_already_sent() sender = FirebaseAdapter :ok = Logic.send( jobs, datetime, already_sent, sender ) end end
  89. mkaszubowski94 defmodule Notifications.Logic do def send(jobs, datetime, already_sent, sender) do

    Enum.each(jobs, fn job -> if starting_soon?(job) && not_sent_yet?(job, already_sent) do sender.send_notification(job) end end) end end
  90. mkaszubowski94 logic is isolated from the data source

  91. mkaszubowski94 Logic desired outcome Notifications Job Context required data data

    source
  92. mkaszubowski94 Logic desired outcome Notifications required data events stream

  93. mkaszubowski94 Logic desired outcome Notifications required data

  94. mkaszubowski94 Logic desired outcome Notifications required data TESTS

  95. mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field

    :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  96. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  97. mkaszubowski94 Maciej Kaszubowski Extracting Publishing

  98. mkaszubowski94 desired outcome: job should be published only after the

    payment is made
  99. mkaszubowski94 problem: the same model is used for multiple behaviours

  100. mkaszubowski94 JobContext • adding / publishing jobs • searching for

    available jobs • managing ongoing jobs • archive / history • …
  101. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  102. mkaszubowski94 Job Publishing drafts Job Context jobs publish

  103. mkaszubowski94 Job Publishing drafts Job Context jobs publish

  104. mkaszubowski94 Job Publishing drafts Job Board jobs publish

  105. mkaszubowski94 Job Publishing drafts Job Board jobs publish enforces job

    structure in params
  106. mkaszubowski94 defmodule JobBoard do defmodule Job do @keys [:name, :description,

    ] @enforce_keys @keys defstruct @keys end def publish(%Job{} = job) do #  end end
  107. mkaszubowski94 Job Publishing drafts Job Board jobs publish enforces job

    structure in params controls the rules for publishing
  108. mkaszubowski94 Job Publishing - save_new_draft() - get_drafts() - pay()

  109. mkaszubowski94 defmodule MyApp.JobBoard.Job do use Ecto.Schema @schema_prefix “job_board” schema "jobs"

    do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  110. mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  111. mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  112. mkaszubowski94 JobBoard • can be tested in isolation (without mocking

    payments) • logic without conditionals • no risk of showing unpublished jobs
  113. mkaszubowski94 JobPublishing • rules for publishing can be changed in

    isolation • can be used alongside other modules
  114. mkaszubowski94 Job Board JobBoard params are enforced by the interface

    Job Publishing
  115. mkaszubowski94 Job Board JobBoard params are enforced by the interface

    Admin
  116. mkaszubowski94 Job Board JobBoard params are enforced by the interface

    TESTS
  117. mkaszubowski94 JobBoard jobs Ongoing Jobs jobs

  118. mkaszubowski94 defmodule MyApp.JobBoard.Job do use Ecto.Schema @schema_prefix “job_board” schema "jobs"

    do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
  119. mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start()

    - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
  120. mkaszubowski94 Maciej Kaszubowski Isolate important decisions

  121. mkaszubowski94 start with a list of important decisions

  122. mkaszubowski94 Important decisions • what are the rules for publishing

    jobs? • when/what notifications are sent? • which jobs are visible on the job board? • how is a contractor chosen for each job? • what happens after a job is started? when the job ends? • how payments work? • …
  123. mkaszubowski94 then, create a module for each decision

  124. mkaszubowski94 Maciej Kaszubowski Summary

  125. mkaszubowski94 JobContext jobs

  126. mkaszubowski94 JobContext • no clear responsibility • hard to change

    and understand • hard to test in isolation • grows with each new feature • hard to delete old features
  127. mkaszubowski94 Job Publishing drafts Job Board jobs Reminders devices SEO

    slugs Ongoing Jobs jobs Job Archive jobs
  128. mkaszubowski94 focused on behaviour, not nouns

  129. mkaszubowski94 Smaller modules • clear responsibilities for each module •

    easy to change, refactor, understand • easy to test in isolation • tend to stay small • trivial to delete if no longer necessary • written once and then forgotten
  130. mkaszubowski94 this is not a silver bullet

  131. mkaszubowski94 Downsides • statistics / reports / admin panels •

    UI has to change sometimes • More initial work • Performance (?) • ...?
  132. mkaszubowski94 it’s just a mindset

  133. mkaszubowski94 focus on behaviour, not data

  134. mkaszubowski94 organise code around behaviour

  135. mkaszubowski94 encapsulate important decisions

  136. mkaszubowski94 Maciej Kaszubowski THANKS!