Modular Design in Elixir (ElixirConf EU 2019)

Slide 1

Slide 1 text

mkaszubowski94 Maciej Kaszubowski

Slide 2

Slide 2 text

mkaszubowski94 Maciej Kaszubowski THE BIG BALL OF NOUNS

Slide 3

Slide 3 text

mkaszubowski94 time complexity & size

Slide 4

Slide 4 text

mkaszubowski94 915

Slide 5

Slide 5 text

mkaszubowski94 1556

Slide 6

Slide 6 text

mkaszubowski94 527

Slide 7

Slide 7 text

mkaszubowski94 270

Slide 8

Slide 8 text

mkaszubowski94

Slide 9

Slide 9 text

mkaszubowski94 4 -7

Slide 10

Slide 10 text

mkaszubowski94 Maciej Kaszubowski Dealing with complexity

Slide 11

Slide 11 text

mkaszubowski94 https://www.conﬂuent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Event Sourcing / CQRS

Slide 12

Slide 12 text

mkaszubowski94 Domain Driven Design

Slide 13

Slide 13 text

mkaszubowski94 Microservices

Slide 14

Slide 14 text

mkaszubowski94 They are all about system decomposition

Slide 15

Slide 15 text

mkaszubowski94 ... and we can do the same in a monolith with a single PostgreSQL instance

Slide 16

Slide 16 text

mkaszubowski94 Maciej Kaszubowski Modular Software Design

Slide 17

Slide 17 text

mkaszubowski94 1972

Slide 18

Slide 18 text

mkaszubowski94 one big problem

Slide 19

Slide 19 text

mkaszubowski94

Slide 20

Slide 20 text

mkaszubowski94 Modular design smaller, simpler modules strong boundaries low coupling

Slide 21

Slide 21 text

mkaszubowski94 Each module can be: understood in isolation modiﬁed in isolation tested in isolation replaced or removed

Slide 22

Slide 22 text

mkaszubowski94 ok, but how?

Slide 23

Slide 23 text

mkaszubowski94 Maciej Kaszubowski Example Project

Slide 24

Slide 24 text

mkaszubowski94 create a new job name image url price publish John Alchemist create new job manage jobs ﬁnd a job logout description date

Slide 25

Slide 25 text

mkaszubowski94 your jobs John Alchemist create new job manage jobs ﬁnd a job logout my awesome job accept my other job cancel accept cancel yet another one and one more accepted this didn’t go well canceled accept cancel

Slide 26

Slide 26 text

mkaszubowski94 browse jobs John Alchemist create new job manage jobs ﬁnd a job logout my awesome job my other job yet another one and one more ﬁlters 100$ 50$ 3500$ 99$ xxxx yyy zzz search

Slide 27

Slide 27 text

mkaszubowski94 start this job John Alchemist create new job manage jobs ﬁnd a job logout Job Title I need something done start a job https://example.com/jobs/1234 100$

Slide 28

Slide 28 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 29

Slide 29 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish()

Slide 30

Slide 30 text

mkaszubowski94 new feature: slugs

Slide 31

Slide 31 text

mkaszubowski94 start this job John Alchemist create new job manage jobs ﬁnd a job logout my awesome job Don’t ask, you’ll get paid start a job https://example.com/jobs/my-awesome-job

Slide 32

Slide 32 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :price, :integer field :slug, :string field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 33

Slide 33 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug()

Slide 34

Slide 34 text

mkaszubowski94 new feature: pay before publishing

Slide 35

Slide 35 text

mkaszubowski94 create a new job name image url price continue John Alchemist create new job manage jobs ﬁnd a job logout description date

Slide 36

Slide 36 text

mkaszubowski94 pay for job pay 110$ John Alchemist create new job manage jobs ﬁnd a job logout my awesome job job price………….………..100$ fee (10%)…………………….10$ total……………………………..110$ Don’t ask, you’ll get paid Your job will be published after the payment

Slide 37

Slide 37 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :slug, :string field :paid, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 38

Slide 38 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() — accept() - cancel() - publish() - create()

Slide 39

Slide 39 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() — accept() - cancel() - publish() - create()

Slide 40

Slide 40 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create()

Slide 41

Slide 41 text

mkaszubowski94 open-closed principle

Slide 42

Slide 42 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create()

Slide 43

Slide 43 text

mkaszubowski94 new feature: job reminders

Slide 44

Slide 44 text

mkaszubowski94 browse jobs John Alchemist create new job manage jobs ﬁnd a job logout my awesome job my other job yet another one and one more ﬁlters 100$ 50$ 3500$ 99$ xxxx yyy zzz search Upcoming job Job starting in 54 minutes ! x

Slide 45

Slide 45 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 46

Slide 46 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 47

Slide 47 text

mkaszubowski94 JobContext jobs

Slide 48

Slide 48 text

mkaszubowski94 jobs JobContext Payments preload jobs

Slide 49

Slide 49 text

mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload jobs

Slide 50

Slide 50 text

mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload jobs Chat Context

Slide 51

Slide 51 text

mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload jobs Chat Context JobSubtask Context

Slide 52

Slide 52 text

mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments JobSubtask Context preload jobs Chat Context JOB USER CONVERSATION MESSAGE PAYMENT SUBTASK

Slide 53

Slide 53 text

mkaszubowski94 In most projects: • few nouns / entities • a lot of interactions between them • logic is organised around the nouns • most actions represented by updates • a lot of coupling as a result

Slide 54

Slide 54 text

mkaszubowski94 We’re used to nouns • Object Oriented Programming • Databases • REST • CRUD

Slide 55

Slide 55 text

mkaszubowski94 nouns are not universal

Slide 56

Slide 56 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 57

Slide 57 text

mkaszubowski94 a “job” means many different things

Slide 58

Slide 58 text

mkaszubowski94 Maciej Kaszubowski Extracting behaviour

Slide 59

Slide 59 text

mkaszubowski94 what is the desired outcome?

Slide 60

Slide 60 text

mkaszubowski94 1. what is the desired outcome?

Slide 61

Slide 61 text

mkaszubowski94 1. what is the desired outcome? 2. what data do I need to do this?

Slide 62

Slide 62 text

mkaszubowski94 1. what is the desired outcome? 2. what data do I need to do this? 3. how can I get this data?

Slide 63

Slide 63 text

mkaszubowski94 1. what is the desired outcome? 2. what data do I need to do this? 3. how can I get this data? 4. what should be exposed in the interface?

Slide 64

Slide 64 text

mkaszubowski94 Maciej Kaszubowski Extracting Slugs

Slide 65

Slide 65 text

mkaszubowski94 start this job John Alchemist create new job manage jobs ﬁnd a job logout my awesome job Don’t ask, you’ll get paid start a job https://example.com/jobs/my-awesome-job

Slide 66

Slide 66 text

mkaszubowski94 desired outcome: jobs should be accessible via human-friendly slugs

Slide 67

Slide 67 text

mkaszubowski94 required data: id-slug mapping for jobs

Slide 68

Slide 68 text

mkaszubowski94 SEO Context

Slide 69

Slide 69 text

mkaszubowski94 SEO Context slugs

Slide 70

Slide 70 text

mkaszubowski94 SEO Context slugs - register_slug() - get_id_by_slug() - get_slugs_by_ids() Interface:

Slide 71

Slide 71 text

mkaszubowski94 def show(conn, %{"job_id"  slug}) do job = slug  SeoContext.fetch_id_by_slug("job")  JobContext.fetch_by_id() #  end

Slide 72

Slide 72 text

mkaszubowski94 defp put_slugs(jobs) do ids = Enum.map(jobs, & &1.id) slugs = SeoContext.fetch_slugs_by_ids(ids, "job") Enum.map(jobs, fn job  Map.put(job, :slug, slugs[job.id]) end) end

Slide 73

Slide 73 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 74

Slide 74 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 75

Slide 75 text

mkaszubowski94 SEO Context • slugs for other resources - easy to do

Slide 76

Slide 76 text

mkaszubowski94 SEO Context • slugs for other resources - easy to do • multiple slugs - easy to do

Slide 77

Slide 77 text

mkaszubowski94 SEO Context • slugs for other resources - easy to do • multiple slugs - easy to do • slugs are resolved at the controller level • no slugs in domain code

Slide 78

Slide 78 text

mkaszubowski94 no slugs in domain logic /api/jobs/my-awesome-job/subtasks from( s in Subtask, join: j in Job, on: s.job_id  j.id where: j.id  ^job_id or j.slug  ^job_id, # … ) /api/jobs/1/subtasks

Slide 79

Slide 79 text

mkaszubowski94 Maciej Kaszubowski Extracting Notiﬁcations

Slide 80

Slide 80 text

Slide 81

Slide 81 text

mkaszubowski94 desired outcome: notiﬁcation is sent before the start of a job

Slide 82

Slide 82 text

mkaszubowski94 required data: job information

Slide 83

Slide 83 text

mkaszubowski94 Job Reminders upcoming_jobs

Slide 84

Slide 84 text

mkaszubowski94 Job Reminders upcoming_jobs - put_job() - remove_job() Interface:

Slide 85

Slide 85 text

mkaszubowski94 Job Reminders upcoming_jobs - put_job() - remove_job() Interface: It's hard to use

Slide 86

Slide 86 text

mkaszubowski94 Interface: Job Context search Job Reminders sent_reminders

Slide 87

Slide 87 text

mkaszubowski94 defmodule Notifications.Worker do use GenServer def handle_info(:work, state) do Notifications.send_notifications(now()) Process.send_after(self(), :work, @interval) {:noreply, state) end end

Slide 88

Slide 88 text

mkaszubowski94 defmodule Notifications do def send_notifications(datetime) do jobs = JobContext.search(…) already_sent = fetch_already_sent() sender = FirebaseAdapter :ok = Logic.send( jobs, datetime, already_sent, sender ) end end

Slide 89

Slide 89 text

mkaszubowski94 defmodule Notifications.Logic do def send(jobs, datetime, already_sent, sender) do Enum.each(jobs, fn job -> if starting_soon?(job) && not_sent_yet?(job, already_sent) do sender.send_notification(job) end end) end end

Slide 90

Slide 90 text

mkaszubowski94 logic is isolated from the data source

Slide 91

Slide 91 text

mkaszubowski94 Logic desired outcome Notiﬁcations Job Context required data data source

Slide 92

Slide 92 text

mkaszubowski94 Logic desired outcome Notiﬁcations required data events stream

Slide 93

Slide 93 text

mkaszubowski94 Logic desired outcome Notiﬁcations required data

Slide 94

Slide 94 text

mkaszubowski94 Logic desired outcome Notiﬁcations required data TESTS

Slide 95

Slide 95 text

mkaszubowski94 defmodule MyApp.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 96

Slide 96 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 97

Slide 97 text

mkaszubowski94 Maciej Kaszubowski Extracting Publishing

Slide 98

Slide 98 text

mkaszubowski94 desired outcome: job should be published only after the payment is made

Slide 99

Slide 99 text

mkaszubowski94 problem: the same model is used for multiple behaviours

Slide 100

Slide 100 text

mkaszubowski94 JobContext • adding / publishing jobs • searching for available jobs • managing ongoing jobs • archive / history • …

Slide 101

Slide 101 text

mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 102

Slide 102 text

mkaszubowski94 Job Publishing drafts Job Context jobs publish

Slide 103

Slide 103 text

mkaszubowski94 Job Publishing drafts Job Context jobs publish

Slide 104

Slide 104 text

mkaszubowski94 Job Publishing drafts Job Board jobs publish

Slide 105

Slide 105 text

mkaszubowski94 Job Publishing drafts Job Board jobs publish enforces job structure in params

Slide 106

Slide 106 text

mkaszubowski94 defmodule JobBoard do defmodule Job do @keys [:name, :description, ] @enforce_keys @keys defstruct @keys end def publish(%Job{} = job) do #  end end

Slide 107

Slide 107 text

mkaszubowski94 Job Publishing drafts Job Board jobs publish enforces job structure in params controls the rules for publishing

Slide 108

Slide 108 text

mkaszubowski94 Job Publishing - save_new_draft() - get_drafts() - pay()

Slide 109

Slide 109 text

mkaszubowski94 defmodule MyApp.JobBoard.Job do use Ecto.Schema @schema_prefix “job_board” schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end

Slide 110

Slide 110 text

mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 111

Slide 111 text

mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 112

Slide 112 text

mkaszubowski94 JobBoard • can be tested in isolation (without mocking payments) • logic without conditionals • no risk of showing unpublished jobs

Slide 113

Slide 113 text

mkaszubowski94 JobPublishing • rules for publishing can be changed in isolation • can be used alongside other modules

Slide 114

Slide 114 text

mkaszubowski94 Job Board JobBoard params are enforced by the interface Job Publishing

Slide 115

Slide 115 text

mkaszubowski94 Job Board JobBoard params are enforced by the interface Admin

Slide 116

Slide 116 text

mkaszubowski94 Job Board JobBoard params are enforced by the interface TESTS

Slide 117

Slide 117 text

mkaszubowski94 JobBoard jobs Ongoing Jobs jobs

Slide 118

Slide 118 text

Slide 119

Slide 119 text

mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()

Slide 120

Slide 120 text

mkaszubowski94 Maciej Kaszubowski Isolate important decisions

Slide 121

Slide 121 text

mkaszubowski94 start with a list of important decisions

Slide 122

Slide 122 text

mkaszubowski94 Important decisions • what are the rules for publishing jobs? • when/what notiﬁcations are sent? • which jobs are visible on the job board? • how is a contractor chosen for each job? • what happens after a job is started? when the job ends? • how payments work? • …

Slide 123

Slide 123 text

mkaszubowski94 then, create a module for each decision

Slide 124

Slide 124 text

mkaszubowski94 Maciej Kaszubowski Summary

Slide 125

Slide 125 text

mkaszubowski94 JobContext jobs

Slide 126

Slide 126 text

mkaszubowski94 JobContext • no clear responsibility • hard to change and understand • hard to test in isolation • grows with each new feature • hard to delete old features

Slide 127

Slide 127 text

mkaszubowski94 Job Publishing drafts Job Board jobs Reminders devices SEO slugs Ongoing Jobs jobs Job Archive jobs

Slide 128

Slide 128 text

mkaszubowski94 focused on behaviour, not nouns

Slide 129

Slide 129 text

mkaszubowski94 Smaller modules • clear responsibilities for each module • easy to change, refactor, understand • easy to test in isolation • tend to stay small • trivial to delete if no longer necessary • written once and then forgotten

Slide 130

Slide 130 text

mkaszubowski94 this is not a silver bullet

Slide 131

Slide 131 text

mkaszubowski94 Downsides • statistics / reports / admin panels • UI has to change sometimes • More initial work • Performance (?) • ...?