$30 off During Our Annual Pro Sale. View Details »

Selling Food With Elixir – ElixirConf 2016

Chris Bell
September 02, 2016

Selling Food With Elixir – ElixirConf 2016

Slides from my talk at Elixir Conf looking at the architecture and design of the Cava Grill Elixir API built using Elixir / Phoenix and OTP.

Chris Bell

September 02, 2016
Tweet

More Decks by Chris Bell

Other Decks in Programming

Transcript

  1. ELIXIR CONF 2016
    Selling food with Elixir
    Chris Bell • @cjbell_

    View Slide

  2. Chris Bell
    @cjbell_

    View Slide

  3. View Slide

  4. What our system needed to do
    • Process and send orders to a Point of Sale
    system in a store.
    • Throttle the volume of orders to a store.
    • Have resilience against stores being down &
    having inconsistent internet access.

    View Slide

  5. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3

    View Slide

  6. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3

    View Slide

  7. How we might have
    approached this in Rails

    View Slide

  8. 1. rails new cava_grill
    2. bundle install sidekiq
    3. rails generate Order
    4. Start coding!

    View Slide

  9. All state lives in the
    database, including any
    background processes.

    View Slide

  10. And we could do this
    again in Phoenix…

    View Slide

  11. 1. mix phoenix.new cava_grill
    2. {install verk}
    3. mix phoenix.gen.model Order
    4. Start coding!

    View Slide

  12. “a classic McLuhanesque mistake of
    appropriating the shape of the
    previous technology as the content
    of the new technology.”
    Scott McCloud

    View Slide

  13. Approaching it in
    a different way

    View Slide

  14. 1. Phoenix is not your application.
    2. Embrace state outside of the database.
    3. If it’s concurrent, extract it into an OTP application.
    4. (Don’t just) Let it crash.
    Elixir Design principles

    View Slide

  15. Order
    Scheduler
    Store
    Availability
    Web
    Order
    Tracker
    Store Store
    Store
    Supervisor
    Store Store
    Store
    Supervisor
    Stores
    Supervisor
    Store Sup
    Store Sup
    API
    Web
    worker
    Web
    worker
    Web
    worker

    View Slide

  16. Order Scheduler
    Just in time delivery of orders to a store
    Store Store
    Store
    Supervisor

    View Slide

  17. Queues orders to send to store, delivering orders
    15 minutes before the timeslot.
    Must handle stores being down / delayed
    without impacting other stores.
    PURPOSE

    View Slide

  18. Order Scheduler
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker

    View Slide

  19. Store #1 Store #2
    Order Scheduler
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker

    View Slide

  20. Scheduling
    orders to send

    View Slide

  21. def handle_info(:process, state) do

    # Do work

    state = do_process_orders(state)
    # Call ourselves again 

    Process.send_after(self, :process, @send_time_ms)
    {:noreply, state}
    end

    View Slide

  22. Database
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor

    View Slide

  23. Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Database

    View Slide

  24. Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    In-store POS
    Database

    View Slide

  25. Expecting and handling failed deliveries 

    (making sure you get your lunch)

    View Slide

  26. github.com/appcues/gen_retry

    View Slide

  27. defp attempt_delivery(store_id, order) do
    do_delivery = fn ->
    # Potential to raise an error
    DeliveryService.deliver(store_id, order)
    end
    GenRetry.retry(do_delivery, [
    retries: 3, delay: 3_000, jitter: 0.2
    ])
    end

    View Slide

  28. Order
    Scheduler
    Store
    Availability
    Web
    Order
    Tracker
    Store Store
    Store
    Supervisor
    Store Store
    Store
    Supervisor
    Stores
    Supervisor
    Store Sup
    Store Sup
    API
    Web
    worker
    Web
    worker
    Web
    worker

    View Slide

  29. Order Tracker
    Keeping track of orders that have
    been delivered to a store.
    Store Store
    Store
    Supervisor

    View Slide

  30. Store Supervisor
    Store
    Manager
    Order Tracker
    Task Supervisor
    Store Supervisor
    Store
    Manager
    Task Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Order
    Update
    Delivery
    Worker
    Delivery
    Worker
    Order
    Update

    View Slide

  31. Each StoreManager fetches a feed
    of order changes and then we
    process those changes
    concurrently

    View Slide

  32. def process_feed_items(store_id, events) do

    events
    |> Enum.map(&start_tracking_worker(store_id, &1))
    end
    def start_tracking_worker(store_id, event) do
    Task.Supervisor.start_child(worker_sup_ref(store_id), fn ->
    # Process the XML, update db state etc
    OrderEventProcessor.process(store_id, event)
    end)
    end

    View Slide

  33. We keep track of the last
    successful time we ran, so we
    know where to start off next time.

    View Slide

  34. But how do we persist this
    between restarts of the process?

    View Slide

  35. def terminate(_reason, {store_id, last_fetch_at}) do
    # Persist our last known good state
    persist_state_to_db(store_id, last_fetch_at)
    :ok
    end

    View Slide

  36. Order
    Scheduler
    Store
    Availability
    Web
    Order
    Tracker
    Store Store
    Store
    Supervisor
    Store Store
    Store
    Supervisor
    Stores
    Supervisor
    Store Sup
    Store Sup
    API
    Web
    worker
    Web
    worker
    Web
    worker

    View Slide

  37. Store Availability
    Track the capacity for a given store
    Stores
    Supervisor
    Store Sup
    Store Sup

    View Slide

  38. Store
    Availability
    Store Supervisor
    ETS Table
    Store
    Manager
    ETS Table
    Manager
    Store Supervisor
    ETS Table
    Store
    Manager
    ETS Table
    Manager
    Store Supervisor
    ETS Table
    Store
    Manager
    ETS Table
    Manager

    View Slide

  39. High demand for timeslots
    during the lunch rush.

    View Slide

  40. We could use the database to
    get the stores availability but
    the data is accessed frequently
    and the call is expensive

    View Slide

  41. We use ETS to store the
    capacity and number of
    confirmed and pending orders
    for that given day.

    View Slide

  42. {{{2016,8,14},{11,45,0}},15,0,10}

    View Slide

  43. {{{2016,8,14},{11,45,0}},15,0,10}
    The Timeslot

    View Slide

  44. {{{2016,8,14},{11,45,0}},15,0,10}
    Capacity

    View Slide

  45. {{{2016,8,14},{11,45,0}},15,0,10}
    Pending & Confirmed 

    Orders Count

    View Slide

  46. Retrieving a stores
    availability

    View Slide

  47. def handle_call(:availability, table) do

    # Read the availability matrix
    result = :ets.tab2list(table)

    {:reply, result, table)
    end

    View Slide

  48. Updating the availability
    of a timeslot

    View Slide

  49. def handle_call({:confirm, datetime, id}, table) do
    # Update the counter for the timeslot in ETS
    :ets.update_counter(

    table, 

    datetime, 

    [{2,0},{3,0},{4,1}]
    )
    {:reply, :ok, table}

    end

    View Slide

  50. High demand for timeslots
    during the lunch rush. How do
    we not let that affect
    customers?

    View Slide

  51. View Slide

  52. StoreAvailability.hold_timeslot(store_id, timeslot, order_id)

    View Slide

  53. Store Supervisor
    Store
    Manager
    Held
    timeslot
    ETS Table
    Manager

    View Slide

  54. Store Supervisor
    Store
    Manager
    Held
    timeslot
    ETS Table
    Manager
    Monitor the 

    Held Timeslot process
    Ref

    View Slide

  55. Store Supervisor
    Store
    Manager
    Held
    timeslot
    Ref
    Start a timer using
    Process.send_after
    ETS Table
    Manager

    View Slide

  56. Store Supervisor
    Store
    Manager
    Held
    timeslot
    Terminate process
    ETS Table
    Manager
    Ref

    View Slide

  57. def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do
    state = do_handle_expired_timeslot(ref, state)
    {:noreply, state}
    end

    View Slide

  58. Booting from a cold start

    View Slide

  59. defp start_availability_manager(%{id: store_id} = store) do
    # Setup the manager with the initial state from database
    availability = StoreAvailabilityGenerator.get_matrix(store)
    # Start the store manager with the state
    # This is really just a call to `Supervisor.start_child`
    StoreAvailability.start_store_manager(store_id, availability)
    end

    View Slide

  60. Look after your
    ETS tables
    PROTIP

    View Slide

  61. github.com/danielberkompas/immortal
    ETSTableManager

    View Slide

  62. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3

    View Slide

  63. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3

    View Slide

  64. Turn down the concurrency
    1

    View Slide

  65. “We had to suspend your API access
    because you were making requests
    too frequently”
    Our POS provider

    View Slide

  66. Sending orders twice
    2

    View Slide

  67. Name your processes if you only
    want one per node.
    LESSON LEARNED

    View Slide

  68. def start_link(%{id: store_id} = store) do
    name = :"#{__MODULE__}.#{store_id}"
    GenServer.start_link(__MODULE__, store, name: name)
    end

    View Slide

  69. There’s no really good way to
    debug what processes are
    running on Heroku.
    LESSON LEARNED

    View Slide

  70. Turn off the retries
    3

    View Slide

  71. Your failure model is only as good
    as they API you’re calling.
    LESSON LEARNED

    View Slide

  72. {:error, :req_timeout}
    4

    View Slide

  73. HTTPotion (ibrowse) pools
    connections per host and the
    default pool size is 10 with a queue
    of 10 per connection.
    LESSON LEARNED

    View Slide

  74. # Number of concurrent sessions in pool

    :ibrowse.set_max_sessions(host, port, @max_sessions)
    # Queue size per session
    :ibrowse.set_max_pipeline_size(host, port, @max_pipeline)

    View Slide

  75. Understand the process design
    (and the bottlenecks) of the
    libraries you’re using.
    LESSON LEARNED

    View Slide

  76. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3

    View Slide

  77. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3

    View Slide

  78. Tips for next time
    1. Feed work, don’t read work.
    2. Start with an umbrella app.
    3. Don’t use Heroku?

    View Slide

  79. Would use Elixir again
    IN CONCLUSION
    A+++

    View Slide

  80. Thank you. Questions?
    ~ Cava Grill are hiring Elixir engineers in DC ~

    cavagrill.com
    [email protected]

    View Slide