Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Selling Food With Elixir – ElixirConf 2016

9ffad8bbc282b748763697965f27b3c8?s=47 Chris Bell
September 02, 2016

Selling Food With Elixir – ElixirConf 2016

Slides from my talk at Elixir Conf looking at the architecture and design of the Cava Grill Elixir API built using Elixir / Phoenix and OTP.

9ffad8bbc282b748763697965f27b3c8?s=128

Chris Bell

September 02, 2016
Tweet

Transcript

  1. ELIXIR CONF 2016 Selling food with Elixir Chris Bell •

    @cjbell_
  2. Chris Bell @cjbell_

  3. None
  4. What our system needed to do • Process and send

    orders to a Point of Sale system in a store. • Throttle the volume of orders to a store. • Have resilience against stores being down & having inconsistent internet access.
  5. Application Design 1 Stories from Production 2 Doing it Again

    3
  6. Application Design 1 Stories from Production 2 Doing it Again

    3
  7. How we might have approached this in Rails

  8. 1. rails new cava_grill 2. bundle install sidekiq 3. rails

    generate Order 4. Start coding!
  9. All state lives in the database, including any background processes.

  10. And we could do this again in Phoenix…

  11. 1. mix phoenix.new cava_grill 2. {install verk} 3. mix phoenix.gen.model

    Order 4. Start coding!
  12. “a classic McLuhanesque mistake of appropriating the shape of the

    previous technology as the content of the new technology.” Scott McCloud
  13. Approaching it in a different way

  14. 1. Phoenix is not your application. 2. Embrace state outside

    of the database. 3. If it’s concurrent, extract it into an OTP application. 4. (Don’t just) Let it crash. Elixir Design principles
  15. Order Scheduler Store Availability Web Order Tracker Store Store Store

    Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
  16. Order Scheduler Just in time delivery of orders to a

    store Store Store Store Supervisor
  17. Queues orders to send to store, delivering orders 15 minutes

    before the timeslot. Must handle stores being down / delayed without impacting other stores. PURPOSE
  18. Order Scheduler Store Supervisor Store Manager Delivery Supervisor Delivery Worker

    Delivery Worker Delivery Worker Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker
  19. Store #1 Store #2 Order Scheduler Store Supervisor Store Manager

    Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker
  20. Scheduling orders to send

  21. def handle_info(:process, state) do
 # Do work
 state = do_process_orders(state)

    # Call ourselves again 
 Process.send_after(self, :process, @send_time_ms) {:noreply, state} end
  22. Database Store Supervisor Store Manager Delivery Supervisor

  23. Store Supervisor Store Manager Delivery Supervisor Database

  24. Store Supervisor Store Manager Delivery Supervisor In-store POS Database

  25. Expecting and handling failed deliveries 
 (making sure you get

    your lunch)
  26. github.com/appcues/gen_retry

  27. defp attempt_delivery(store_id, order) do do_delivery = fn -> # Potential

    to raise an error DeliveryService.deliver(store_id, order) end GenRetry.retry(do_delivery, [ retries: 3, delay: 3_000, jitter: 0.2 ]) end
  28. Order Scheduler Store Availability Web Order Tracker Store Store Store

    Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
  29. Order Tracker Keeping track of orders that have been delivered

    to a store. Store Store Store Supervisor
  30. Store Supervisor Store Manager Order Tracker Task Supervisor Store Supervisor

    Store Manager Task Supervisor Delivery Worker Delivery Worker Order Update Delivery Worker Delivery Worker Order Update
  31. Each StoreManager fetches a feed of order changes and then

    we process those changes concurrently
  32. def process_feed_items(store_id, events) do
 events |> Enum.map(&start_tracking_worker(store_id, &1)) end def

    start_tracking_worker(store_id, event) do Task.Supervisor.start_child(worker_sup_ref(store_id), fn -> # Process the XML, update db state etc OrderEventProcessor.process(store_id, event) end) end
  33. We keep track of the last successful time we ran,

    so we know where to start off next time.
  34. But how do we persist this between restarts of the

    process?
  35. def terminate(_reason, {store_id, last_fetch_at}) do # Persist our last known

    good state persist_state_to_db(store_id, last_fetch_at) :ok end
  36. Order Scheduler Store Availability Web Order Tracker Store Store Store

    Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
  37. Store Availability Track the capacity for a given store Stores

    Supervisor Store Sup Store Sup
  38. Store Availability Store Supervisor ETS Table Store Manager ETS Table

    Manager Store Supervisor ETS Table Store Manager ETS Table Manager Store Supervisor ETS Table Store Manager ETS Table Manager
  39. High demand for timeslots during the lunch rush.

  40. We could use the database to get the stores availability

    but the data is accessed frequently and the call is expensive
  41. We use ETS to store the capacity and number of

    confirmed and pending orders for that given day.
  42. {{{2016,8,14},{11,45,0}},15,0,10}

  43. {{{2016,8,14},{11,45,0}},15,0,10} The Timeslot

  44. {{{2016,8,14},{11,45,0}},15,0,10} Capacity

  45. {{{2016,8,14},{11,45,0}},15,0,10} Pending & Confirmed 
 Orders Count

  46. Retrieving a stores availability

  47. def handle_call(:availability, table) do
 # Read the availability matrix result

    = :ets.tab2list(table)
 {:reply, result, table) end
  48. Updating the availability of a timeslot

  49. def handle_call({:confirm, datetime, id}, table) do # Update the counter

    for the timeslot in ETS :ets.update_counter(
 table, 
 datetime, 
 [{2,0},{3,0},{4,1}] ) {:reply, :ok, table}
 end
  50. High demand for timeslots during the lunch rush. How do

    we not let that affect customers?
  51. None
  52. StoreAvailability.hold_timeslot(store_id, timeslot, order_id)

  53. Store Supervisor Store Manager Held timeslot ETS Table Manager

  54. Store Supervisor Store Manager Held timeslot ETS Table Manager Monitor

    the 
 Held Timeslot process Ref
  55. Store Supervisor Store Manager Held timeslot Ref Start a timer

    using Process.send_after ETS Table Manager
  56. Store Supervisor Store Manager Held timeslot Terminate process ETS Table

    Manager Ref
  57. def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do state =

    do_handle_expired_timeslot(ref, state) {:noreply, state} end
  58. Booting from a cold start

  59. defp start_availability_manager(%{id: store_id} = store) do # Setup the manager

    with the initial state from database availability = StoreAvailabilityGenerator.get_matrix(store) # Start the store manager with the state # This is really just a call to `Supervisor.start_child` StoreAvailability.start_store_manager(store_id, availability) end
  60. Look after your ETS tables PROTIP

  61. github.com/danielberkompas/immortal ETSTableManager

  62. Application Design 1 Stories from Production 2 Doing it Again

    3
  63. Application Design 1 Stories from Production 2 Doing it Again

    3
  64. Turn down the concurrency 1

  65. “We had to suspend your API access because you were

    making requests too frequently” Our POS provider
  66. Sending orders twice 2

  67. Name your processes if you only want one per node.

    LESSON LEARNED
  68. def start_link(%{id: store_id} = store) do name = :"#{__MODULE__}.#{store_id}" GenServer.start_link(__MODULE__,

    store, name: name) end
  69. There’s no really good way to debug what processes are

    running on Heroku. LESSON LEARNED
  70. Turn off the retries 3

  71. Your failure model is only as good as they API

    you’re calling. LESSON LEARNED
  72. {:error, :req_timeout} 4

  73. HTTPotion (ibrowse) pools connections per host and the default pool

    size is 10 with a queue of 10 per connection. LESSON LEARNED
  74. # Number of concurrent sessions in pool
 :ibrowse.set_max_sessions(host, port, @max_sessions)

    # Queue size per session :ibrowse.set_max_pipeline_size(host, port, @max_pipeline)
  75. Understand the process design (and the bottlenecks) of the libraries

    you’re using. LESSON LEARNED
  76. Application Design 1 Stories from Production 2 Doing it Again

    3
  77. Application Design 1 Stories from Production 2 Doing it Again

    3
  78. Tips for next time 1. Feed work, don’t read work.

    2. Start with an umbrella app. 3. Don’t use Heroku?
  79. Would use Elixir again IN CONCLUSION A+++

  80. Thank you. Questions? ~ Cava Grill are hiring Elixir engineers

    in DC ~
 cavagrill.com chris@madebymany.com