$30 off During Our Annual Pro Sale. View Details »

Modular Design in Elixir (ElixirConf EU 2019)

Modular Design in Elixir (ElixirConf EU 2019)

A talk given at ElixirConf EU 2019 in Prague. It talks about splitting big systems into smaller, simpler, more manageable modules that are easier to understand, test and work with.

Maciej Kaszubowski

April 08, 2019
Tweet

More Decks by Maciej Kaszubowski

Other Decks in Programming

Transcript

  1. mkaszubowski94
    Maciej Kaszubowski

    View Slide

  2. mkaszubowski94
    Maciej Kaszubowski
    THE BIG BALL
    OF NOUNS

    View Slide

  3. mkaszubowski94
    time
    complexity & size

    View Slide

  4. mkaszubowski94
    915

    View Slide

  5. mkaszubowski94
    1556

    View Slide

  6. mkaszubowski94
    527

    View Slide

  7. mkaszubowski94
    270

    View Slide

  8. mkaszubowski94

    View Slide

  9. mkaszubowski94
    4 -7

    View Slide

  10. mkaszubowski94
    Maciej Kaszubowski
    Dealing with
    complexity

    View Slide

  11. mkaszubowski94
    https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
    Event Sourcing / CQRS

    View Slide

  12. mkaszubowski94
    Domain Driven Design

    View Slide

  13. mkaszubowski94
    Microservices

    View Slide

  14. mkaszubowski94
    They are all about
    system decomposition

    View Slide

  15. mkaszubowski94
    ... and we can do the same
    in a monolith with a single
    PostgreSQL instance

    View Slide

  16. mkaszubowski94
    Maciej Kaszubowski
    Modular Software
    Design

    View Slide

  17. mkaszubowski94
    1972

    View Slide

  18. mkaszubowski94
    one big
    problem

    View Slide

  19. mkaszubowski94

    View Slide

  20. mkaszubowski94
    Modular design
    smaller, simpler modules
    strong boundaries
    low coupling

    View Slide

  21. mkaszubowski94
    Each module can be:
    understood in isolation
    modified in isolation
    tested in isolation
    replaced or removed

    View Slide

  22. mkaszubowski94
    ok, but how?

    View Slide

  23. mkaszubowski94
    Maciej Kaszubowski
    Example Project

    View Slide

  24. mkaszubowski94
    create a new job
    name
    image url
    price
    publish
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    description
    date

    View Slide

  25. mkaszubowski94
    your jobs
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job accept
    my other job
    cancel
    accept cancel
    yet another one
    and one more accepted
    this didn’t go well canceled
    accept cancel

    View Slide

  26. mkaszubowski94
    browse jobs
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job
    my other job
    yet another one
    and one more
    filters
    100$
    50$
    3500$
    99$
    xxxx
    yyy
    zzz
    search

    View Slide

  27. mkaszubowski94
    start this job
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    Job Title
    I need something done
    start a job
    https://example.com/jobs/1234
    100$

    View Slide

  28. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  29. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()

    View Slide

  30. mkaszubowski94
    new feature:
    slugs

    View Slide

  31. mkaszubowski94
    start this job
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job
    Don’t ask, you’ll get paid
    start a job
    https://example.com/jobs/my-awesome-job

    View Slide

  32. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :price, :integer
    field :slug, :string
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  33. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()

    View Slide

  34. mkaszubowski94
    new feature:
    pay before publishing

    View Slide

  35. mkaszubowski94
    create a new job
    name
    image url
    price
    continue
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    description
    date

    View Slide

  36. mkaszubowski94
    pay for job
    pay 110$
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job
    job price………….………..100$
    fee (10%)…………………….10$
    total……………………………..110$
    Don’t ask, you’ll get paid
    Your job will be published after the payment

    View Slide

  37. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :slug, :string
    field :paid, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  38. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    — accept()
    - cancel()
    - publish()
    - create()

    View Slide

  39. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    — accept()
    - cancel()
    - publish()
    - create()

    View Slide

  40. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()

    View Slide

  41. mkaszubowski94
    open-closed principle

    View Slide

  42. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()

    View Slide

  43. mkaszubowski94
    new feature:
    job reminders

    View Slide

  44. mkaszubowski94
    browse jobs
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job
    my other job
    yet another one
    and one more
    filters
    100$
    50$
    3500$
    99$
    xxxx
    yyy
    zzz
    search
    Upcoming job
    Job starting in 54 minutes
    ! x

    View Slide

  45. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :slug, :string
    field :started, :boolean
    field :notification_sent, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  46. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  47. mkaszubowski94
    JobContext
    jobs

    View Slide

  48. mkaszubowski94
    jobs
    JobContext
    Payments
    preload jobs

    View Slide

  49. mkaszubowski94
    jobs
    JobContext
    User
    Context
    users
    join (jobs_count)
    Payments
    preload jobs

    View Slide

  50. mkaszubowski94
    jobs
    JobContext
    User
    Context
    users
    join (jobs_count)
    Payments
    preload jobs
    Chat
    Context

    View Slide

  51. mkaszubowski94
    jobs
    JobContext
    User
    Context
    users
    join (jobs_count)
    Payments
    preload jobs
    Chat
    Context
    JobSubtask
    Context

    View Slide

  52. mkaszubowski94
    jobs
    JobContext
    User
    Context
    users
    join (jobs_count)
    Payments
    JobSubtask
    Context
    preload jobs
    Chat
    Context
    JOB
    USER
    CONVERSATION
    MESSAGE
    PAYMENT
    SUBTASK

    View Slide

  53. mkaszubowski94
    In most projects:
    • few nouns / entities
    • a lot of interactions between them
    • logic is organised around the nouns
    • most actions represented by updates
    • a lot of coupling as a result

    View Slide

  54. mkaszubowski94
    We’re used to nouns
    • Object Oriented Programming
    • Databases
    • REST
    • CRUD

    View Slide

  55. mkaszubowski94
    nouns are not
    universal

    View Slide

  56. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :slug, :string
    field :started, :boolean
    field :notification_sent, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  57. mkaszubowski94
    a “job” means many
    different things

    View Slide

  58. mkaszubowski94
    Maciej Kaszubowski
    Extracting behaviour

    View Slide

  59. mkaszubowski94
    what is the desired outcome?

    View Slide

  60. mkaszubowski94
    1. what is the desired outcome?

    View Slide

  61. mkaszubowski94
    1. what is the desired outcome?
    2. what data do I need to do this?

    View Slide

  62. mkaszubowski94
    1. what is the desired outcome?
    2. what data do I need to do this?
    3. how can I get this data?

    View Slide

  63. mkaszubowski94
    1. what is the desired outcome?
    2. what data do I need to do this?
    3. how can I get this data?
    4. what should be exposed in the
    interface?

    View Slide

  64. mkaszubowski94
    Maciej Kaszubowski
    Extracting Slugs

    View Slide

  65. mkaszubowski94
    start this job
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job
    Don’t ask, you’ll get paid
    start a job
    https://example.com/jobs/my-awesome-job

    View Slide

  66. mkaszubowski94
    desired outcome:
    jobs should be accessible via
    human-friendly slugs

    View Slide

  67. mkaszubowski94
    required data:
    id-slug mapping for jobs

    View Slide

  68. mkaszubowski94
    SEO
    Context

    View Slide

  69. mkaszubowski94
    SEO
    Context
    slugs

    View Slide

  70. mkaszubowski94
    SEO
    Context
    slugs
    - register_slug()
    - get_id_by_slug()
    - get_slugs_by_ids()
    Interface:

    View Slide

  71. mkaszubowski94
    def show(conn, %{"job_id"  slug}) do
    job =
    slug
     SeoContext.fetch_id_by_slug("job")
     JobContext.fetch_by_id()
    # 
    end

    View Slide

  72. mkaszubowski94
    defp put_slugs(jobs) do
    ids = Enum.map(jobs, & &1.id)
    slugs = SeoContext.fetch_slugs_by_ids(ids, "job")
    Enum.map(jobs, fn job 
    Map.put(job, :slug, slugs[job.id])
    end)
    end

    View Slide

  73. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :slug, :string
    field :started, :boolean
    field :notification_sent, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  74. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  75. mkaszubowski94
    SEO Context
    • slugs for other resources - easy to do

    View Slide

  76. mkaszubowski94
    SEO Context
    • slugs for other resources - easy to do
    • multiple slugs - easy to do

    View Slide

  77. mkaszubowski94
    SEO Context
    • slugs for other resources - easy to do
    • multiple slugs - easy to do
    • slugs are resolved at the controller level
    • no slugs in domain code

    View Slide

  78. mkaszubowski94
    no slugs in domain logic
    /api/jobs/my-awesome-job/subtasks
    from(
    s in Subtask,
    join: j in Job, on: s.job_id  j.id
    where: j.id  ^job_id or j.slug  ^job_id,
    # …
    )
    /api/jobs/1/subtasks

    View Slide

  79. mkaszubowski94
    Maciej Kaszubowski
    Extracting Notifications

    View Slide

  80. mkaszubowski94
    browse jobs
    John Alchemist
    create new job
    manage jobs
    find a job
    logout
    my awesome job
    my other job
    yet another one
    and one more
    filters
    100$
    50$
    3500$
    99$
    xxxx
    yyy
    zzz
    search
    Upcoming job
    Job starting in 54 minutes
    ! x

    View Slide

  81. mkaszubowski94
    desired outcome:
    notification is sent before the
    start of a job

    View Slide

  82. mkaszubowski94
    required data:
    job information

    View Slide

  83. mkaszubowski94
    Job
    Reminders
    upcoming_jobs

    View Slide

  84. mkaszubowski94
    Job
    Reminders
    upcoming_jobs
    - put_job()
    - remove_job()
    Interface:

    View Slide

  85. mkaszubowski94
    Job
    Reminders
    upcoming_jobs
    - put_job()
    - remove_job()
    Interface:
    It's hard to use

    View Slide

  86. mkaszubowski94
    Interface:
    Job
    Context
    search
    Job
    Reminders
    sent_reminders

    View Slide

  87. mkaszubowski94
    defmodule Notifications.Worker do
    use GenServer
    def handle_info(:work, state) do
    Notifications.send_notifications(now())
    Process.send_after(self(), :work, @interval)
    {:noreply, state)
    end
    end

    View Slide

  88. mkaszubowski94
    defmodule Notifications do
    def send_notifications(datetime) do
    jobs = JobContext.search(…)
    already_sent = fetch_already_sent()
    sender = FirebaseAdapter
    :ok = Logic.send(
    jobs, datetime, already_sent, sender
    )
    end
    end

    View Slide

  89. mkaszubowski94
    defmodule Notifications.Logic do
    def send(jobs, datetime, already_sent, sender) do
    Enum.each(jobs, fn job ->
    if starting_soon?(job) &&
    not_sent_yet?(job, already_sent) do
    sender.send_notification(job)
    end
    end)
    end
    end

    View Slide

  90. mkaszubowski94
    logic is isolated from the data
    source

    View Slide

  91. mkaszubowski94
    Logic
    desired
    outcome
    Notifications
    Job
    Context
    required
    data
    data
    source

    View Slide

  92. mkaszubowski94
    Logic
    desired
    outcome
    Notifications
    required
    data
    events stream

    View Slide

  93. mkaszubowski94
    Logic
    desired
    outcome
    Notifications
    required
    data

    View Slide

  94. mkaszubowski94
    Logic
    desired
    outcome
    Notifications
    required
    data
    TESTS

    View Slide

  95. mkaszubowski94
    defmodule MyApp.Job do
    use Ecto.Schema
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :slug, :string
    field :started, :boolean
    field :notification_sent, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  96. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  97. mkaszubowski94
    Maciej Kaszubowski
    Extracting Publishing

    View Slide

  98. mkaszubowski94
    desired outcome:
    job should be published only
    after the payment is made

    View Slide

  99. mkaszubowski94
    problem:
    the same model is used for
    multiple behaviours

    View Slide

  100. mkaszubowski94
    JobContext
    • adding / publishing jobs
    • searching for available jobs
    • managing ongoing jobs
    • archive / history
    • …

    View Slide

  101. mkaszubowski94
    JobContext
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  102. mkaszubowski94
    Job
    Publishing
    drafts
    Job
    Context
    jobs
    publish

    View Slide

  103. mkaszubowski94
    Job
    Publishing
    drafts
    Job
    Context
    jobs
    publish

    View Slide

  104. mkaszubowski94
    Job
    Publishing
    drafts
    Job
    Board
    jobs
    publish

    View Slide

  105. mkaszubowski94
    Job
    Publishing
    drafts
    Job
    Board
    jobs
    publish
    enforces job
    structure in params

    View Slide

  106. mkaszubowski94
    defmodule JobBoard do
    defmodule Job do
    @keys [:name, :description, ]
    @enforce_keys @keys
    defstruct @keys
    end
    def publish(%Job{} = job) do
    # 
    end
    end

    View Slide

  107. mkaszubowski94
    Job
    Publishing
    drafts
    Job
    Board
    jobs
    publish
    enforces job
    structure in params
    controls the rules
    for publishing

    View Slide

  108. mkaszubowski94
    Job
    Publishing
    - save_new_draft()
    - get_drafts()
    - pay()

    View Slide

  109. mkaszubowski94
    defmodule MyApp.JobBoard.Job do
    use Ecto.Schema
    @schema_prefix “job_board”
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :slug, :string
    field :started, :boolean
    field :notification_sent, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  110. mkaszubowski94
    JobBoard
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  111. mkaszubowski94
    JobBoard
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  112. mkaszubowski94
    JobBoard
    • can be tested in isolation
    (without mocking payments)
    • logic without conditionals
    • no risk of showing unpublished jobs

    View Slide

  113. mkaszubowski94
    JobPublishing
    • rules for publishing can be changed
    in isolation
    • can be used alongside other
    modules

    View Slide

  114. mkaszubowski94
    Job
    Board
    JobBoard
    params are enforced
    by the interface
    Job
    Publishing

    View Slide

  115. mkaszubowski94
    Job
    Board
    JobBoard
    params are enforced
    by the interface
    Admin

    View Slide

  116. mkaszubowski94
    Job
    Board
    JobBoard
    params are enforced
    by the interface
    TESTS

    View Slide

  117. mkaszubowski94
    JobBoard
    jobs
    Ongoing
    Jobs
    jobs

    View Slide

  118. mkaszubowski94
    defmodule MyApp.JobBoard.Job do
    use Ecto.Schema
    @schema_prefix “job_board”
    schema "jobs" do
    field :name, :string
    field :description, :string
    field :image_url, :string
    field :paid, :boolean
    field :slug, :string
    field :started, :boolean
    field :notification_sent, :boolean
    field :price, :integer
    field :datetime, :utc_datetime
    belongs_to :creator, ModularElixir.User
    belongs_to :contractor, ModularElixir.User
    end
    end

    View Slide

  119. mkaszubowski94
    JobBoard
    - fetch_by_id()
    - search()
    - find_by_user_id()
    - start()
    - accept()
    - cancel()
    - publish()
    - create()
    - find_by_slug()
    - send_reminder()

    View Slide

  120. mkaszubowski94
    Maciej Kaszubowski
    Isolate important
    decisions

    View Slide

  121. mkaszubowski94
    start with a list of
    important decisions

    View Slide

  122. mkaszubowski94
    Important decisions
    • what are the rules for publishing jobs?
    • when/what notifications are sent?
    • which jobs are visible on the job board?
    • how is a contractor chosen for each job?
    • what happens after a job is started? when the job
    ends?
    • how payments work?
    • …

    View Slide

  123. mkaszubowski94
    then, create a module
    for each decision

    View Slide

  124. mkaszubowski94
    Maciej Kaszubowski
    Summary

    View Slide

  125. mkaszubowski94
    JobContext
    jobs

    View Slide

  126. mkaszubowski94
    JobContext
    • no clear responsibility
    • hard to change and understand
    • hard to test in isolation
    • grows with each new feature
    • hard to delete old features

    View Slide

  127. mkaszubowski94
    Job
    Publishing
    drafts
    Job
    Board
    jobs
    Reminders
    devices
    SEO
    slugs
    Ongoing
    Jobs
    jobs
    Job
    Archive
    jobs

    View Slide

  128. mkaszubowski94
    focused on
    behaviour, not nouns

    View Slide

  129. mkaszubowski94
    Smaller modules
    • clear responsibilities for each module
    • easy to change, refactor, understand
    • easy to test in isolation
    • tend to stay small
    • trivial to delete if no longer necessary
    • written once and then forgotten

    View Slide

  130. mkaszubowski94
    this is not a silver
    bullet

    View Slide

  131. mkaszubowski94
    Downsides
    • statistics / reports / admin panels
    • UI has to change sometimes
    • More initial work
    • Performance (?)
    • ...?

    View Slide

  132. mkaszubowski94
    it’s just a mindset

    View Slide

  133. mkaszubowski94
    focus on behaviour,
    not data

    View Slide

  134. mkaszubowski94
    organise code
    around behaviour

    View Slide

  135. mkaszubowski94
    encapsulate
    important decisions

    View Slide

  136. mkaszubowski94
    Maciej Kaszubowski
    THANKS!

    View Slide