Selling Food With Elixir – ElixirConf 2016

Slides from my talk at Elixir Conf looking at the architecture and design of the Cava Grill Elixir API built using Elixir / Phoenix and OTP.

Transcript

1. ELIXIR CONF 2016
   Selling food with Elixir
   Chris Bell • @cjbell_
   

   View Slide

2. Chris Bell
   @cjbell_
   

   View Slide

3. View Slide

4. What our system needed to do
   • Process and send orders to a Point of Sale
   system in a store.
   • Throttle the volume of orders to a store.
   • Have resilience against stores being down &
   having inconsistent internet access.
   

   View Slide

5. Application Design
   1
   Stories from Production
   2
   Doing it Again
   3
   

   View Slide

6. Application Design
   1
   Stories from Production
   2
   Doing it Again
   3
   

   View Slide

7. How we might have
   approached this in Rails
   

   View Slide

8. 1. rails new cava_grill
   2. bundle install sidekiq
   3. rails generate Order
   4. Start coding!
   

   View Slide

9. All state lives in the
   database, including any
   background processes.
   

   View Slide

10. And we could do this
    again in Phoenix…
    

    View Slide

11. 1. mix phoenix.new cava_grill
    2. {install verk}
    3. mix phoenix.gen.model Order
    4. Start coding!
    

    View Slide

12. “a classic McLuhanesque mistake of
    appropriating the shape of the
    previous technology as the content
    of the new technology.”
    Scott McCloud
    

    View Slide

13. Approaching it in
    a different way
    

    View Slide

14. 1. Phoenix is not your application.
    2. Embrace state outside of the database.
    3. If it’s concurrent, extract it into an OTP application.
    4. (Don’t just) Let it crash.
    Elixir Design principles
    

    View Slide

15. Order
    Scheduler
    Store
    Availability
    Web
    Order
    Tracker
    Store Store
    Store
    Supervisor
    Store Store
    Store
    Supervisor
    Stores
    Supervisor
    Store Sup
    Store Sup
    API
    Web
    worker
    Web
    worker
    Web
    worker
    

    View Slide

16. Order Scheduler
    Just in time delivery of orders to a store
    Store Store
    Store
    Supervisor
    

    View Slide

17. Queues orders to send to store, delivering orders
    15 minutes before the timeslot.
    Must handle stores being down / delayed
    without impacting other stores.
    PURPOSE
    

    View Slide

18. Order Scheduler
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker
    

    View Slide

19. Store #1 Store #2
    Order Scheduler
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Delivery
    Worker
    

    View Slide

20. Scheduling
    orders to send
    

    View Slide

21. def handle_info(:process, state) do

    # Do work

    state = do_process_orders(state)
    # Call ourselves again 

    Process.send_after(self, :process, @send_time_ms)
    {:noreply, state}
    end
    

    View Slide

22. Database
    Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    

    View Slide

23. Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    Database
    

    View Slide

24. Store Supervisor
    Store
    Manager
    Delivery
    Supervisor
    In-store POS
    Database
    

    View Slide

25. Expecting and handling failed deliveries 

    (making sure you get your lunch)
    

    View Slide

26. github.com/appcues/gen_retry
    

    View Slide

27. defp attempt_delivery(store_id, order) do
    do_delivery = fn ->
    # Potential to raise an error
    DeliveryService.deliver(store_id, order)
    end
    GenRetry.retry(do_delivery, [
    retries: 3, delay: 3_000, jitter: 0.2
    ])
    end
    

    View Slide

28. Order
    Scheduler
    Store
    Availability
    Web
    Order
    Tracker
    Store Store
    Store
    Supervisor
    Store Store
    Store
    Supervisor
    Stores
    Supervisor
    Store Sup
    Store Sup
    API
    Web
    worker
    Web
    worker
    Web
    worker
    

    View Slide

29. Order Tracker
    Keeping track of orders that have
    been delivered to a store.
    Store Store
    Store
    Supervisor
    

    View Slide

30. Store Supervisor
    Store
    Manager
    Order Tracker
    Task Supervisor
    Store Supervisor
    Store
    Manager
    Task Supervisor
    Delivery
    Worker
    Delivery
    Worker
    Order
    Update
    Delivery
    Worker
    Delivery
    Worker
    Order
    Update
    

    View Slide

31. Each StoreManager fetches a feed
    of order changes and then we
    process those changes
    concurrently
    

    View Slide

32. def process_feed_items(store_id, events) do

    events
    |> Enum.map(&start_tracking_worker(store_id, &1))
    end
    def start_tracking_worker(store_id, event) do
    Task.Supervisor.start_child(worker_sup_ref(store_id), fn ->
    # Process the XML, update db state etc
    OrderEventProcessor.process(store_id, event)
    end)
    end
    

    View Slide

33. We keep track of the last
    successful time we ran, so we
    know where to start off next time.
    

    View Slide

34. But how do we persist this
    between restarts of the process?
    

    View Slide

35. def terminate(_reason, {store_id, last_fetch_at}) do
    # Persist our last known good state
    persist_state_to_db(store_id, last_fetch_at)
    :ok
    end
    

    View Slide

36. Order
    Scheduler
    Store
    Availability
    Web
    Order
    Tracker
    Store Store
    Store
    Supervisor
    Store Store
    Store
    Supervisor
    Stores
    Supervisor
    Store Sup
    Store Sup
    API
    Web
    worker
    Web
    worker
    Web
    worker
    

    View Slide

37. Store Availability
    Track the capacity for a given store
    Stores
    Supervisor
    Store Sup
    Store Sup
    

    View Slide

38. Store
    Availability
    Store Supervisor
    ETS Table
    Store
    Manager
    ETS Table
    Manager
    Store Supervisor
    ETS Table
    Store
    Manager
    ETS Table
    Manager
    Store Supervisor
    ETS Table
    Store
    Manager
    ETS Table
    Manager
    

    View Slide

39. High demand for timeslots
    during the lunch rush.
    

    View Slide

40. We could use the database to
    get the stores availability but
    the data is accessed frequently
    and the call is expensive
    

    View Slide

41. We use ETS to store the
    capacity and number of
    confirmed and pending orders
    for that given day.
    

    View Slide

42. {{{2016,8,14},{11,45,0}},15,0,10}
    

    View Slide

43. {{{2016,8,14},{11,45,0}},15,0,10}
    The Timeslot
    

    View Slide

44. {{{2016,8,14},{11,45,0}},15,0,10}
    Capacity
    

    View Slide

45. {{{2016,8,14},{11,45,0}},15,0,10}
    Pending & Confirmed 

    Orders Count
    

    View Slide

46. Retrieving a stores
    availability
    

    View Slide

47. def handle_call(:availability, table) do

    # Read the availability matrix
    result = :ets.tab2list(table)

    {:reply, result, table)
    end
    

    View Slide

48. Updating the availability
    of a timeslot
    

    View Slide

49. def handle_call({:confirm, datetime, id}, table) do
    # Update the counter for the timeslot in ETS
    :ets.update_counter(

    table, 

    datetime, 

    [{2,0},{3,0},{4,1}]
    )
    {:reply, :ok, table}

    end
    

    View Slide

50. High demand for timeslots
    during the lunch rush. How do
    we not let that affect
    customers?
    

    View Slide

51. View Slide

52. StoreAvailability.hold_timeslot(store_id, timeslot, order_id)
    

    View Slide

53. Store Supervisor
    Store
    Manager
    Held
    timeslot
    ETS Table
    Manager
    

    View Slide

54. Store Supervisor
    Store
    Manager
    Held
    timeslot
    ETS Table
    Manager
    Monitor the 

    Held Timeslot process
    Ref
    

    View Slide

55. Store Supervisor
    Store
    Manager
    Held
    timeslot
    Ref
    Start a timer using
    Process.send_after
    ETS Table
    Manager
    

    View Slide

56. Store Supervisor
    Store
    Manager
    Held
    timeslot
    Terminate process
    ETS Table
    Manager
    Ref
    

    View Slide

57. def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do
    state = do_handle_expired_timeslot(ref, state)
    {:noreply, state}
    end
    

    View Slide

58. Booting from a cold start
    

    View Slide

59. defp start_availability_manager(%{id: store_id} = store) do
    # Setup the manager with the initial state from database
    availability = StoreAvailabilityGenerator.get_matrix(store)
    # Start the store manager with the state
    # This is really just a call to `Supervisor.start_child`
    StoreAvailability.start_store_manager(store_id, availability)
    end
    

    View Slide

60. Look after your
    ETS tables
    PROTIP
    

    View Slide

61. github.com/danielberkompas/immortal
    ETSTableManager
    

    View Slide

62. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3
    

    View Slide

63. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3
    

    View Slide

64. Turn down the concurrency
    1
    

    View Slide

65. “We had to suspend your API access
    because you were making requests
    too frequently”
    Our POS provider
    

    View Slide

66. Sending orders twice
    2
    

    View Slide

67. Name your processes if you only
    want one per node.
    LESSON LEARNED
    

    View Slide

68. def start_link(%{id: store_id} = store) do
    name = :"#{__MODULE__}.#{store_id}"
    GenServer.start_link(__MODULE__, store, name: name)
    end
    

    View Slide

69. There’s no really good way to
    debug what processes are
    running on Heroku.
    LESSON LEARNED
    

    View Slide

70. Turn off the retries
    3
    

    View Slide

71. Your failure model is only as good
    as they API you’re calling.
    LESSON LEARNED
    

    View Slide

72. {:error, :req_timeout}
    4
    

    View Slide

73. HTTPotion (ibrowse) pools
    connections per host and the
    default pool size is 10 with a queue
    of 10 per connection.
    LESSON LEARNED
    

    View Slide

74. # Number of concurrent sessions in pool

    :ibrowse.set_max_sessions(host, port, @max_sessions)
    # Queue size per session
    :ibrowse.set_max_pipeline_size(host, port, @max_pipeline)
    

    View Slide

75. Understand the process design
    (and the bottlenecks) of the
    libraries you’re using.
    LESSON LEARNED
    

    View Slide

76. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3
    

    View Slide

77. Application Design
    1
    Stories from Production
    2
    Doing it Again
    3
    

    View Slide

78. Tips for next time
    1. Feed work, don’t read work.
    2. Start with an umbrella app.
    3. Don’t use Heroku?
    

    View Slide

79. Would use Elixir again
    IN CONCLUSION
    A+++
    

    View Slide

80. Thank you. Questions?
    ~ Cava Grill are hiring Elixir engineers in DC ~

    cavagrill.com
    chris@madebymany.com
    

    View Slide