Building an Umbrella Project

One of the many goals that both Elixir & Phoenix share is developer’s productivity. There can be two aspects of productivity: short-term and long-term.
In this talk we’ll explore how using Elixir’s umbrella project feature can help with long-term productivity when working with potentially large codebases. As a example we’ll walk through building an imaginary system: Acme Bank. We will see how we can break up this system into a collection of smaller applications, each of which has a well defined boundary, domain model, set of responsibilities, and can be worked on independently. We’ll compare this approach to microservices-based architecture which is being adopted by many organizations and we’ll see if we can achieve some of the benefits without incurring some of the drawbacks.

Transcript

1. Building an Umbrella Project Wojtek Mach @wojtekmach @wojtekmach
2. Me • Ruby • GitHub API adapter for Ecto • OOP library for Elixir • Acme Bank
3. Me • Ruby • GitHub API adapter for Ecto • OOP library for Elixir • Acme Bank
4. Me • Ruby • GitHub API adapter for Ecto • OOP library for Elixir • Acme Bank
5. Me • Ruby • GitHub API adapter for Ecto • OOP library for Elixir • Acme Bank
6. What’s an Umbrella Project?
7. Inspiration
8. Inspiration Architecture: The Lost Years
 Robert C. Martin, Midwest RubyConf 2011
 • Typical Rails app • app/{models,controllers,view} • Top-level architecture should scream it’s intent
9. Inspiration Architecture: The Lost Years
 Robert C. Martin, Midwest RubyConf 2011
 • Typical Rails app • app/{models,controllers,view} • Top-level architecture should scream it’s intent
10. Inspiration Architecture: The Lost Years
 Robert C. Martin, Midwest RubyConf 2011
 • Typical Rails app • app/{models,controllers,views,…} • Top-level architecture should scream it’s intent
11. Inspiration Architecture: The Lost Years
 Robert C. Martin, Midwest RubyConf 2011
 • Typical Rails app • app/{models,controllers,views,…} • Top-level architecture should scream it’s intent
12. Inspiration Wrangling Large Rails Codebases
 Stephan Hagemann, Rocky Mountain Ruby 2012
 • Advice about writing large apps • Components
13. Inspiration Wrangling Large Rails Codebases
 Stephan Hagemann, Rocky Mountain Ruby 2012
 • Advice about writing a large app • Components
14. Inspiration Wrangling Large Rails Codebases
 Stephan Hagemann, Rocky Mountain Ruby 2012
 • Don’t write a large app • Components
15. Inspiration Wrangling Large Rails Codebases
 Stephan Hagemann, Rocky Mountain Ruby 2012
 • Don’t write a large app • Components: gems & engines
16. Inspiration The Art of Destroying Software
 Greg Young, Leetspeek 2014 
 • organize your project in a way that makes it easy to delete code.
17. Inspiration The Art of Destroying Software
 Greg Young, Leetspeek 2014 
 • organize your project in a way that makes it easy to delete code.
18. What’s an Umbrella Project?
19. What’s an Umbrella Project? • Mixfile • apps/ directory • OTP applications
20. OTP application „In Elixir (actually, in Erlang/OTP), an application is a component implementing some specific functionality, that can be started and stopped as a unit, and which can be re-used in other systems.” — Elixir API docs
21. OTP application • application dependencies • configuration • start/stop behaviour • regular applications & library applications
22. OTP application • application dependencies • configuration: config.exs, Application.put_env, mix.exs • start/stop behaviour • regular applications & library applications
23. OTP application • application dependencies • configuration • start/stop behaviour • regular applications & library applications
24. OTP application • application dependencies • configuration • start/stop behaviour • regular applications & library applications
25. OTP application • application dependencies • configuration • start/stop behaviour • regular applications & library applications • standard
26. OTP application „Application”
27. OTP application Server
28. OTP application Service
29. OTP application Microservice
30. Building Acme Bank
31. $ mix new bank_platform --umbrella
32. # mix.exs defmodule BankPlatform.Mixfile do use Mix.Project def project do [apps_path: "apps", deps: deps(), aliases: aliases()] end end Bank Platform
33. First OTP App
34. Bank
35. $ cd apps/
 $ mix new bank --sup Bank
36. # apps/bank/mix.exs defmodule Bank.Mixfile do use Mix.Project def project do [app: :bank, version: "0.1.0", build_path: "../../_build", config_path: "../../config/config.exs", deps_path: "../../deps", lockfile: "../../mix.lock", deps: deps()] end end Bank
37. First Feature
38. First Feature: Account Balance
39. defmodule Bank.Account do defstruct name: "", balance: 0 end Naive
40. defmodule Bank.Account do defstruct name: "", balance: 0 def deposit(account, amount) do %Account{account | balance: account.balance + amount} end end Naive
41. Accounting 101
42. Accounting 101 • Any movement of money happens between at least 2 accounts: one account is debited and the other account is credited • Example: Alice transfers $100 to Bob
43. Accounting 101 • Any movement of money happens between at least 2 accounts: one account is debited and the other account is credited • Example: Alice transfers $100 to Bob
44. Alice transfers $100 to Bob | Bob | :credit | 100.00 | Alice | :debit | 100.00 Accounting 101
45. Bob transfers $20 to Alice | Bob | :credit | 100.00 | Alice | :debit | 100.00 | Alice | :credit | 20.00 | Bob | :debit | 20.00 Accounting 101
46. Bob transfers $10 to Alice | Bob | :credit | 100.00 | Alice | :debit | 100.00 | Alice | :credit | 20.00 | Bob | :debit | 20.00 | Bob | :debit | 10.00 | Alice | :credit | 10.00 Accounting 101
47. Bob’s balance = $100 - $20 - $10 = $70 | Bob | :credit | 100.00 | Alice | :debit | 100.00 | Alice | :credit | 20.00 | Bob | :debit | 20.00 | Bob | :debit | 10.00 | Alice | :credit | 10.00 Accounting 101
48. Bob’s balance = $100 - $20 - $10 = $70 | Bob | :credit | 100.00 | Alice | :debit | 100.00 | Alice | :credit | 20.00 | Bob | :debit | 20.00 | Bob | :debit | 10.00 | Alice | :credit | 10.00 Accounting 101
49. Alice’s balance = -$100 + $20 + $10 = -$70 | Bob | :credit | 100.00 | Alice | :debit | 100.00 | Alice | :credit | 20.00 | Bob | :debit | 20.00 | Bob | :debit | 10.00 | Alice | :credit | 10.00 Accounting 101
50. Alice’s balance = -$100 + $20 + $10 = -$70 | Bob | :credit | 100.00 | Alice | :debit | 100.00 | Alice | :credit | 20.00 | Bob | :debit | 20.00 | Bob | :debit | 10.00 | Alice | :credit | 10.00 Double-entry accounting system Accounting 101
51. Accounting 101 Why mention this? • it’s a Bank • it’s functional • mathematical properties • complexity
52. Accounting 101 Why mention this? • it’s functional • mathematical properties • complexity
53. Accounting 101 Why mention this? • it’s functional • mathematical properties • complexity
54. Accounting 101 Why mention this? • it’s functional • associativity • complexity
55. Accounting 101 Why mention this? • it’s functional • associativity, commutativity • complexity
56. Accounting 101 Why mention this? • it’s functional • associativity, commutativity, idempotence • complexity
57. Accounting 101 Why mention this? • it’s functional • CRDT • complexity
58. Accounting 101 Why mention this? • it’s functional • eventual consistency [1] • complexity [1] http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html
59. Accounting 101 Why mention this? • it’s functional • eventual consistency [1] • complexity [1] http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html
60. Accounting 101 Why mention this? • it’s functional • eventual consistency [1] • complexity • complexity [1] http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html
61. Bank: Entities
62. Bank: Entities • Account • Entry • Customer
63. Bank: Entities • Account: web/models/account.ex • Entry: web/models/entry.ex • Customer: web/models/customer.ex
64. Bank: Entities • Account: lib/bank/account.ex • Entry: lib/bank/entry.ex • Customer: lib/bank/customer.ex
65. Bank: Entities • Account: lib/bank/account.ex • Entry: lib/bank/entry.ex • Customer: lib/bank/customer.ex Different rates of change
66. Bank: Entities • Account: lib/bank/ledger/account.ex • Entry: lib/bank/ledger/entry.ex • Customer: lib/bank/customer.ex Different rates of change
67. Bank: Persistence
68. Bank: Persistence • Ecto • Where to put Repo access?
69. Bank: Persistence • Ecto • Where to put Repo access?
70. Bank.Ledger # lib/bank/ledger.ex defmodule Bank.Ledger do def balance(account) do Repo.one("SELECT SUM(...)") end def write(entries) do # ... end end
71. Bank.Ledger # lib/bank/ledger.ex defmodule Bank.Ledger do def balance(account) do Repo.one("SELECT SUM(...)") end def write(entries) do # ... end end
72. Bank.Ledger # lib/bank/ledger.ex defmodule Bank.Ledger do def balance(account) do Repo.one("SELECT SUM(...)") end def write(entries) do # ... end end
73. Bank # lib/bank.ex defmodule Bank do def create_customer(params) do Bank.Customer.changeset(params) |> Repo.insert end end
74. Bank # lib/bank.ex defmodule Bank do def create_customer(params) do Bank.Customer.changeset(params) |> Repo.insert end end
75. Bank # lib/bank.ex defmodule Bank do def create_customer(params) do Bank.Customer.changeset(params) |> Repo.insert end end
76. Bank Bank.create_customer/1
 Bank.Ledger.balance/1
 Bank.Ledger.write/1 • Use cases • Use in controllers, tests, iex, seeds
77. Bank Bank.create_customer/1
 Bank.Ledger.balance/1
 Bank.Ledger.write/1 • Use cases • Use in controllers, tests, iex, seeds
78. Bank Bank.create_customer/1
 Bank.Ledger.balance/1
 Bank.Ledger.write/1 • Use cases • Use in controllers, tests, iex, seeds
79. Money
80. Money • Amount + currency • Operations • Persistence • Isolation • OSS • Productivity
81. Money • Amount + currency • Operations • Persistence • Isolation • OSS • Productivity
82. Money • Amount + currency • Operations • Persistence • Isolation • OSS • Productivity
83. Money • Amount + currency • Operations • Persistence • Isolation • OSS • Productivity
84. Money • Amount + currency • Operations • Persistence • Isolation • OSS • Productivity
85. Money • Amount + currency • Operations • Persistence • Isolation • OSS • Productivity
86. Money # apps/bank/mix.exs defp deps do [ {:money, in_umbrella: true} ] end
87. Money # apps/bank/mix.exs def application do [applications: [..., :money]] end
88. BankWeb
89. BankWeb $ cd apps/ $ mix phoenix.new bank_web # add bank as dependency
90. BankWeb $ cd apps/ $ mix phoenix.new bank_web # add bank as dependency
91. BankWeb $ cd apps/ $ mix phoenix.new bank_web # add bank as dependency
92. BankWeb • boring! • connect controllers to business logic
93. BankWeb • boring! • separation of concerns
94. Backoffice
95. Backoffice • ExAdmin. Works with umbrellas out of the box • ports setup • deployment • Phoenix is Not Your Application
96. Backoffice • ExAdmin. Works with umbrellas out of the box • setup different ports in development • deployment • Phoenix is Not Your Application
97. Backoffice • ExAdmin. Works with umbrellas out of the box • setup different ports in development • deployment • Phoenix is Not Your Application
98. Backoffice • ExAdmin. Works with umbrellas out of the box • setup different ports in development • deployment • Phoenix is Not Your Application
99. Backoffice • ExAdmin. Works with umbrellas out of the box • setup different ports in development • deployment • Phoenix is Not Your Application
100. Deployment
101. Deployment • Heroku • MasterProxy.
102. Deployment • Heroku • MasterProxy.
103. Deployment • Heroku • MasterProxy. Thanks @TheGazler! • distillery, exrm replacement, has 1st class support for umbrellas
104. Deployment • Heroku • MasterProxy. Thanks @TheGazler! • distillery, exrm replacement, has 1st class support for umbrellas
105. Auth
106. Auth • Upfront design? • Features: audit logs
107. Auth • Upfront design? • Features: strategies
108. Auth • Upfront design? • Features: strategies, 2FA
109. Auth • Upfront design? • Features: strategies, 2FA, tracking
110. Auth • Upfront design? • Features: strategies, 2FA, tracking
111. Auth • Upfront design? • Features: strategies, 2FA, tracking • Separate Repo, separate DB
112. Auth • Upfront design? • Features: strategies, 2FA, tracking • Separate Repo, separate DB • Thanks @chris_mccord!
113. Messenger
114. Bank Platform
115. BankPlatform • auth • bank • bank_web • backoffice • master_proxy • messenger • money
116. BankPlatform
117. BankPlatform
118. BankPlatform
119. Microservices
120. Microservices Benefits: • Strong Module Boundaries • Independent Deployments • Technology Diversity http://martinfowler.com/articles/microservice-trade-offs.html
121. Microservices Benefits: • Strong Module Boundaries • Independent Deployments • Technology Diversity http://martinfowler.com/articles/microservice-trade-offs.html Drawbacks: • Distribution • Eventual Consistency • Operational Complexity
124. Conclusion
125. Conclusion • keep an eye on the big picture
126. Conclusion • keep an eye on the big picture • don’t write a large app
127. Conclusion • keep an eye on the big picture • don’t write a large app • isolate by functionality and rate of change
128. Conclusion • keep an eye on the big picture • don’t write a large app • isolate by functionality and rate of change • delete code liberally
129. Thanks! wojtekmach/acme_bank