Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Selling Food With Elixir – ElixirConf 2016

Chris Bell
September 02, 2016

Selling Food With Elixir – ElixirConf 2016

Slides from my talk at Elixir Conf looking at the architecture and design of the Cava Grill Elixir API built using Elixir / Phoenix and OTP.

Chris Bell

September 02, 2016
Tweet

More Decks by Chris Bell

Other Decks in Programming

Transcript

  1. What our system needed to do • Process and send

    orders to a Point of Sale system in a store. • Throttle the volume of orders to a store. • Have resilience against stores being down & having inconsistent internet access.
  2. “a classic McLuhanesque mistake of appropriating the shape of the

    previous technology as the content of the new technology.” Scott McCloud
  3. 1. Phoenix is not your application. 2. Embrace state outside

    of the database. 3. If it’s concurrent, extract it into an OTP application. 4. (Don’t just) Let it crash. Elixir Design principles
  4. Order Scheduler Store Availability Web Order Tracker Store Store Store

    Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
  5. Order Scheduler Just in time delivery of orders to a

    store Store Store Store Supervisor
  6. Queues orders to send to store, delivering orders 15 minutes

    before the timeslot. Must handle stores being down / delayed without impacting other stores. PURPOSE
  7. Order Scheduler Store Supervisor Store Manager Delivery Supervisor Delivery Worker

    Delivery Worker Delivery Worker Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker
  8. Store #1 Store #2 Order Scheduler Store Supervisor Store Manager

    Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker Store Supervisor Store Manager Delivery Supervisor Delivery Worker Delivery Worker Delivery Worker
  9. def handle_info(:process, state) do
 # Do work
 state = do_process_orders(state)

    # Call ourselves again 
 Process.send_after(self, :process, @send_time_ms) {:noreply, state} end
  10. defp attempt_delivery(store_id, order) do do_delivery = fn -> # Potential

    to raise an error DeliveryService.deliver(store_id, order) end GenRetry.retry(do_delivery, [ retries: 3, delay: 3_000, jitter: 0.2 ]) end
  11. Order Scheduler Store Availability Web Order Tracker Store Store Store

    Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
  12. Order Tracker Keeping track of orders that have been delivered

    to a store. Store Store Store Supervisor
  13. Store Supervisor Store Manager Order Tracker Task Supervisor Store Supervisor

    Store Manager Task Supervisor Delivery Worker Delivery Worker Order Update Delivery Worker Delivery Worker Order Update
  14. Each StoreManager fetches a feed of order changes and then

    we process those changes concurrently
  15. def process_feed_items(store_id, events) do
 events |> Enum.map(&start_tracking_worker(store_id, &1)) end def

    start_tracking_worker(store_id, event) do Task.Supervisor.start_child(worker_sup_ref(store_id), fn -> # Process the XML, update db state etc OrderEventProcessor.process(store_id, event) end) end
  16. We keep track of the last successful time we ran,

    so we know where to start off next time.
  17. def terminate(_reason, {store_id, last_fetch_at}) do # Persist our last known

    good state persist_state_to_db(store_id, last_fetch_at) :ok end
  18. Order Scheduler Store Availability Web Order Tracker Store Store Store

    Supervisor Store Store Store Supervisor Stores Supervisor Store Sup Store Sup API Web worker Web worker Web worker
  19. Store Availability Store Supervisor ETS Table Store Manager ETS Table

    Manager Store Supervisor ETS Table Store Manager ETS Table Manager Store Supervisor ETS Table Store Manager ETS Table Manager
  20. We could use the database to get the stores availability

    but the data is accessed frequently and the call is expensive
  21. We use ETS to store the capacity and number of

    confirmed and pending orders for that given day.
  22. def handle_call(:availability, table) do
 # Read the availability matrix result

    = :ets.tab2list(table)
 {:reply, result, table) end
  23. def handle_call({:confirm, datetime, id}, table) do # Update the counter

    for the timeslot in ETS :ets.update_counter(
 table, 
 datetime, 
 [{2,0},{3,0},{4,1}] ) {:reply, :ok, table}
 end
  24. Store Supervisor Store Manager Held timeslot Ref Start a timer

    using Process.send_after ETS Table Manager
  25. def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do state =

    do_handle_expired_timeslot(ref, state) {:noreply, state} end
  26. defp start_availability_manager(%{id: store_id} = store) do # Setup the manager

    with the initial state from database availability = StoreAvailabilityGenerator.get_matrix(store) # Start the store manager with the state # This is really just a call to `Supervisor.start_child` StoreAvailability.start_store_manager(store_id, availability) end
  27. “We had to suspend your API access because you were

    making requests too frequently” Our POS provider
  28. Your failure model is only as good as they API

    you’re calling. LESSON LEARNED
  29. HTTPotion (ibrowse) pools connections per host and the default pool

    size is 10 with a queue of 10 per connection. LESSON LEARNED
  30. # Number of concurrent sessions in pool
 :ibrowse.set_max_sessions(host, port, @max_sessions)

    # Queue size per session :ibrowse.set_max_pipeline_size(host, port, @max_pipeline)
  31. Tips for next time 1. Feed work, don’t read work.

    2. Start with an umbrella app. 3. Don’t use Heroku?