Selling Food With Elixir – ElixirConf 2016

Slides from my talk at Elixir Conf looking at the architecture and design of the Cava Grill Elixir API built using Elixir / Phoenix and OTP.

Transcript

1. ELIXIR CONF 2016 Selling food with Elixir Chris Bell • @cjbell_
2. Chris Bell @cjbell_
4. What our system needed to do • Process and send orders to a Point of Sale system in a store. • Throttle the volume of orders to a store. • Have resilience against stores being down & having inconsistent internet access.
5. Application Design 1 Stories from Production 2 Doing it Again 3
6. Application Design 1 Stories from Production 2 Doing it Again 3
7. How we might have approached this in Rails
8. 1. rails new cava_grill 2. bundle install sidekiq 3. rails generate Order 4. Start coding!
9. All state lives in the database, including any background processes.
10. And we could do this again in Phoenix…
11. 1. mix phoenix.new cava_grill 2. {install verk} 3. mix phoenix.gen.model Order 4. Start coding!
12. “a classic McLuhanesque mistake of appropriating the shape of the previous technology as the content of the new technology.” Scott McCloud
13. Approaching it in a different way
14. 1. Phoenix is not your application. 2. Embrace state outside of the database. 3. If it’s concurrent, extract it into an OTP application. 4. (Don’t just) Let it crash. Elixir Design principles
15. Order Scheduler Store Availability Web Order Tracker Store Store Store Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
16. Order Scheduler Just in time delivery of orders to a store Store Store Store Supervisor
17. Queues orders to send to store, delivering orders 15 minutes before the timeslot. Must handle stores being down / delayed without impacting other stores. PURPOSE
18. Order Scheduler Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker
19. Store #1 Store #2 Order Scheduler Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker
20. Scheduling orders to send
21. def handle_info(:process, state) do
 # Do work
 state = do_process_orders(state) # Call ourselves again 
 Process.send_after(self, :process, @send_time_ms) {:noreply, state} end
22. Database Store Supervisor Store Manager Delivery Supervisor
23. Store Supervisor Store Manager Delivery Supervisor Database
24. Store Supervisor Store Manager Delivery Supervisor In-store POS Database
25. Expecting and handling failed deliveries 
 (making sure you get your lunch)
26. github.com/appcues/gen_retry
27. defp attempt_delivery(store_id, order) do do_delivery = fn -> # Potential to raise an error DeliveryService.deliver(store_id, order) end GenRetry.retry(do_delivery, [ retries: 3, delay: 3_000, jitter: 0.2 ]) end
28. Order Scheduler Store Availability Web Order Tracker Store Store Store Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
29. Order Tracker Keeping track of orders that have been delivered to a store. Store Store Store Supervisor
30. Store Supervisor Store Manager Order Tracker Task Supervisor Store Supervisor Store Manager Task Supervisor Delivery Worker Delivery Worker Order Update Delivery Worker Delivery Worker Order Update
31. Each StoreManager fetches a feed of order changes and then we process those changes concurrently
32. def process_feed_items(store_id, events) do
 events |> Enum.map(&start_tracking_worker(store_id, &1)) end def start_tracking_worker(store_id, event) do Task.Supervisor.start_child(worker_sup_ref(store_id), fn -> # Process the XML, update db state etc OrderEventProcessor.process(store_id, event) end) end
33. We keep track of the last successful time we ran, so we know where to start off next time.
34. But how do we persist this between restarts of the process?
35. def terminate(_reason, {store_id, last_fetch_at}) do # Persist our last known good state persist_state_to_db(store_id, last_fetch_at) :ok end
36. Order Scheduler Store Availability Web Order Tracker Store Store Store Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
37. Store Availability Track the capacity for a given store Stores Supervisor Store Sup Store Sup
38. Store Availability Store Supervisor ETS Table Store Manager ETS Table Manager Store Supervisor ETS Table Store Manager ETS Table Manager Store Supervisor ETS Table Store Manager ETS Table Manager
39. High demand for timeslots during the lunch rush.
40. We could use the database to get the stores availability but the data is accessed frequently and the call is expensive
41. We use ETS to store the capacity and number of confirmed and pending orders for that given day.
42. {{{2016,8,14},{11,45,0}},15,0,10}
43. {{{2016,8,14},{11,45,0}},15,0,10} The Timeslot
44. {{{2016,8,14},{11,45,0}},15,0,10} Capacity
45. {{{2016,8,14},{11,45,0}},15,0,10} Pending & Confirmed 
 Orders Count
46. Retrieving a stores availability
47. def handle_call(:availability, table) do
 # Read the availability matrix result = :ets.tab2list(table)
 {:reply, result, table) end
48. Updating the availability of a timeslot
49. def handle_call({:confirm, datetime, id}, table) do # Update the counter for the timeslot in ETS :ets.update_counter(
 table, 
 datetime, 
 [{2,0},{3,0},{4,1}] ) {:reply, :ok, table}
 end
50. High demand for timeslots during the lunch rush. How do we not let that affect customers?
52. StoreAvailability.hold_timeslot(store_id, timeslot, order_id)
53. Store Supervisor Store Manager Held timeslot ETS Table Manager
54. Store Supervisor Store Manager Held timeslot ETS Table Manager Monitor the 
 Held Timeslot process Ref
55. Store Supervisor Store Manager Held timeslot Ref Start a timer using Process.send_after ETS Table Manager
56. Store Supervisor Store Manager Held timeslot Terminate process ETS Table Manager Ref
57. def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do state = do_handle_expired_timeslot(ref, state) {:noreply, state} end
58. Booting from a cold start
59. defp start_availability_manager(%{id: store_id} = store) do # Setup the manager with the initial state from database availability = StoreAvailabilityGenerator.get_matrix(store) # Start the store manager with the state # This is really just a call to `Supervisor.start_child` StoreAvailability.start_store_manager(store_id, availability) end
60. Look after your ETS tables PROTIP
61. github.com/danielberkompas/immortal ETSTableManager
62. Application Design 1 Stories from Production 2 Doing it Again 3
63. Application Design 1 Stories from Production 2 Doing it Again 3
64. Turn down the concurrency 1
65. “We had to suspend your API access because you were making requests too frequently” Our POS provider
66. Sending orders twice 2
67. Name your processes if you only want one per node. LESSON LEARNED
68. def start_link(%{id: store_id} = store) do name = :"#{__MODULE__}.#{store_id}" GenServer.start_link(__MODULE__, store, name: name) end
69. There’s no really good way to debug what processes are running on Heroku. LESSON LEARNED
70. Turn off the retries 3
71. Your failure model is only as good as they API you’re calling. LESSON LEARNED
72. {:error, :req_timeout} 4
73. HTTPotion (ibrowse) pools connections per host and the default pool size is 10 with a queue of 10 per connection. LESSON LEARNED
74. # Number of concurrent sessions in pool
 :ibrowse.set_max_sessions(host, port, @max_sessions) # Queue size per session :ibrowse.set_max_pipeline_size(host, port, @max_pipeline)
75. Understand the process design (and the bottlenecks) of the libraries you’re using. LESSON LEARNED
76. Application Design 1 Stories from Production 2 Doing it Again 3
77. Application Design 1 Stories from Production 2 Doing it Again 3
78. Tips for next time 1. Feed work, don’t read work. 2. Start with an umbrella app. 3. Don’t use Heroku?
79. Would use Elixir again IN CONCLUSION A+++
80. Thank you. Questions? ~ Cava Grill are hiring Elixir engineers in DC ~
 cavagrill.com chris@madebymany.com