ELIXIR CONF 2016
Selling food with Elixir
Chris Bell • @cjbell_
Slide 2
Slide 2 text
Chris Bell
@cjbell_
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
What our system needed to do
• Process and send orders to a Point of Sale
system in a store.
• Throttle the volume of orders to a store.
• Have resilience against stores being down &
having inconsistent internet access.
Slide 5
Slide 5 text
Application Design
1
Stories from Production
2
Doing it Again
3
Slide 6
Slide 6 text
Application Design
1
Stories from Production
2
Doing it Again
3
Slide 7
Slide 7 text
How we might have
approached this in Rails
Slide 8
Slide 8 text
1. rails new cava_grill
2. bundle install sidekiq
3. rails generate Order
4. Start coding!
Slide 9
Slide 9 text
All state lives in the
database, including any
background processes.
“a classic McLuhanesque mistake of
appropriating the shape of the
previous technology as the content
of the new technology.”
Scott McCloud
Slide 13
Slide 13 text
Approaching it in
a different way
Slide 14
Slide 14 text
1. Phoenix is not your application.
2. Embrace state outside of the database.
3. If it’s concurrent, extract it into an OTP application.
4. (Don’t just) Let it crash.
Elixir Design principles
Slide 15
Slide 15 text
Order
Scheduler
Store
Availability
Web
Order
Tracker
Store Store
Store
Supervisor
Store Store
Store
Supervisor
Stores
Supervisor
Store Sup
Store Sup
API
Web
worker
Web
worker
Web
worker
Slide 16
Slide 16 text
Order Scheduler
Just in time delivery of orders to a store
Store Store
Store
Supervisor
Slide 17
Slide 17 text
Queues orders to send to store, delivering orders
15 minutes before the timeslot.
Must handle stores being down / delayed
without impacting other stores.
PURPOSE
Slide 18
Slide 18 text
Order Scheduler
Store Supervisor
Store
Manager
Delivery
Supervisor
Delivery
Worker
Delivery
Worker
Delivery
Worker
Store Supervisor
Store
Manager
Delivery
Supervisor
Delivery
Worker
Delivery
Worker
Delivery
Worker
Slide 19
Slide 19 text
Store #1 Store #2
Order Scheduler
Store Supervisor
Store
Manager
Delivery
Supervisor
Delivery
Worker
Delivery
Worker
Delivery
Worker
Store Supervisor
Store
Manager
Delivery
Supervisor
Delivery
Worker
Delivery
Worker
Delivery
Worker
Slide 20
Slide 20 text
Scheduling
orders to send
Slide 21
Slide 21 text
def handle_info(:process, state) do
# Do work
state = do_process_orders(state)
# Call ourselves again
Process.send_after(self, :process, @send_time_ms)
{:noreply, state}
end
Slide 22
Slide 22 text
Database
Store Supervisor
Store
Manager
Delivery
Supervisor
Slide 23
Slide 23 text
Store Supervisor
Store
Manager
Delivery
Supervisor
Database
Slide 24
Slide 24 text
Store Supervisor
Store
Manager
Delivery
Supervisor
In-store POS
Database
Slide 25
Slide 25 text
Expecting and handling failed deliveries
(making sure you get your lunch)
Slide 26
Slide 26 text
github.com/appcues/gen_retry
Slide 27
Slide 27 text
defp attempt_delivery(store_id, order) do
do_delivery = fn ->
# Potential to raise an error
DeliveryService.deliver(store_id, order)
end
GenRetry.retry(do_delivery, [
retries: 3, delay: 3_000, jitter: 0.2
])
end
Slide 28
Slide 28 text
Order
Scheduler
Store
Availability
Web
Order
Tracker
Store Store
Store
Supervisor
Store Store
Store
Supervisor
Stores
Supervisor
Store Sup
Store Sup
API
Web
worker
Web
worker
Web
worker
Slide 29
Slide 29 text
Order Tracker
Keeping track of orders that have
been delivered to a store.
Store Store
Store
Supervisor
Slide 30
Slide 30 text
Store Supervisor
Store
Manager
Order Tracker
Task Supervisor
Store Supervisor
Store
Manager
Task Supervisor
Delivery
Worker
Delivery
Worker
Order
Update
Delivery
Worker
Delivery
Worker
Order
Update
Slide 31
Slide 31 text
Each StoreManager fetches a feed
of order changes and then we
process those changes
concurrently
Slide 32
Slide 32 text
def process_feed_items(store_id, events) do
events
|> Enum.map(&start_tracking_worker(store_id, &1))
end
def start_tracking_worker(store_id, event) do
Task.Supervisor.start_child(worker_sup_ref(store_id), fn ->
# Process the XML, update db state etc
OrderEventProcessor.process(store_id, event)
end)
end
Slide 33
Slide 33 text
We keep track of the last
successful time we ran, so we
know where to start off next time.
Slide 34
Slide 34 text
But how do we persist this
between restarts of the process?
Slide 35
Slide 35 text
def terminate(_reason, {store_id, last_fetch_at}) do
# Persist our last known good state
persist_state_to_db(store_id, last_fetch_at)
:ok
end
Slide 36
Slide 36 text
Order
Scheduler
Store
Availability
Web
Order
Tracker
Store Store
Store
Supervisor
Store Store
Store
Supervisor
Stores
Supervisor
Store Sup
Store Sup
API
Web
worker
Web
worker
Web
worker
Slide 37
Slide 37 text
Store Availability
Track the capacity for a given store
Stores
Supervisor
Store Sup
Store Sup
Slide 38
Slide 38 text
Store
Availability
Store Supervisor
ETS Table
Store
Manager
ETS Table
Manager
Store Supervisor
ETS Table
Store
Manager
ETS Table
Manager
Store Supervisor
ETS Table
Store
Manager
ETS Table
Manager
Slide 39
Slide 39 text
High demand for timeslots
during the lunch rush.
Slide 40
Slide 40 text
We could use the database to
get the stores availability but
the data is accessed frequently
and the call is expensive
Slide 41
Slide 41 text
We use ETS to store the
capacity and number of
confirmed and pending orders
for that given day.
def handle_call(:availability, table) do
# Read the availability matrix
result = :ets.tab2list(table)
{:reply, result, table)
end
Slide 48
Slide 48 text
Updating the availability
of a timeslot
Slide 49
Slide 49 text
def handle_call({:confirm, datetime, id}, table) do
# Update the counter for the timeslot in ETS
:ets.update_counter(
table,
datetime,
[{2,0},{3,0},{4,1}]
)
{:reply, :ok, table}
end
Slide 50
Slide 50 text
High demand for timeslots
during the lunch rush. How do
we not let that affect
customers?
Store Supervisor
Store
Manager
Held
timeslot
ETS Table
Manager
Slide 54
Slide 54 text
Store Supervisor
Store
Manager
Held
timeslot
ETS Table
Manager
Monitor the
Held Timeslot process
Ref
Slide 55
Slide 55 text
Store Supervisor
Store
Manager
Held
timeslot
Ref
Start a timer using
Process.send_after
ETS Table
Manager
Slide 56
Slide 56 text
Store Supervisor
Store
Manager
Held
timeslot
Terminate process
ETS Table
Manager
Ref
Slide 57
Slide 57 text
def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do
state = do_handle_expired_timeslot(ref, state)
{:noreply, state}
end
Slide 58
Slide 58 text
Booting from a cold start
Slide 59
Slide 59 text
defp start_availability_manager(%{id: store_id} = store) do
# Setup the manager with the initial state from database
availability = StoreAvailabilityGenerator.get_matrix(store)
# Start the store manager with the state
# This is really just a call to `Supervisor.start_child`
StoreAvailability.start_store_manager(store_id, availability)
end
Application Design
1
Stories from Production
2
Doing it Again
3
Slide 63
Slide 63 text
Application Design
1
Stories from Production
2
Doing it Again
3
Slide 64
Slide 64 text
Turn down the concurrency
1
Slide 65
Slide 65 text
“We had to suspend your API access
because you were making requests
too frequently”
Our POS provider
Slide 66
Slide 66 text
Sending orders twice
2
Slide 67
Slide 67 text
Name your processes if you only
want one per node.
LESSON LEARNED
Slide 68
Slide 68 text
def start_link(%{id: store_id} = store) do
name = :"#{__MODULE__}.#{store_id}"
GenServer.start_link(__MODULE__, store, name: name)
end
Slide 69
Slide 69 text
There’s no really good way to
debug what processes are
running on Heroku.
LESSON LEARNED
Slide 70
Slide 70 text
Turn off the retries
3
Slide 71
Slide 71 text
Your failure model is only as good
as they API you’re calling.
LESSON LEARNED
Slide 72
Slide 72 text
{:error, :req_timeout}
4
Slide 73
Slide 73 text
HTTPotion (ibrowse) pools
connections per host and the
default pool size is 10 with a queue
of 10 per connection.
LESSON LEARNED
Slide 74
Slide 74 text
# Number of concurrent sessions in pool
:ibrowse.set_max_sessions(host, port, @max_sessions)
# Queue size per session
:ibrowse.set_max_pipeline_size(host, port, @max_pipeline)
Slide 75
Slide 75 text
Understand the process design
(and the bottlenecks) of the
libraries you’re using.
LESSON LEARNED
Slide 76
Slide 76 text
Application Design
1
Stories from Production
2
Doing it Again
3
Slide 77
Slide 77 text
Application Design
1
Stories from Production
2
Doing it Again
3
Slide 78
Slide 78 text
Tips for next time
1. Feed work, don’t read work.
2. Start with an umbrella app.
3. Don’t use Heroku?
Slide 79
Slide 79 text
Would use Elixir again
IN CONCLUSION
A+++
Slide 80
Slide 80 text
Thank you. Questions?
~ Cava Grill are hiring Elixir engineers in DC ~
cavagrill.com
chris@madebymany.com