How did we scalably connect 1000s of stores more reliably & securely? Preserving Purity What benefits did our pure serverless architecture give us? Promoting Practical Progress What can we take away from this serverless problem-solving process?
Had previously only enabled other integrations via XML-based API Quickly build an online store Multi-tenant, scalable Opting to build serverless system on AWS (full story another time) API-based to support enterprise integrations
new online system to price online orders always exactly matching in-store orders Pricing parity a critical core business requirement for online offering Too low is bad! Too high is bad! Short timeline But should still build to align with further dev & product plans
and order context Regional, store-level, customer-level Prices even change based on time or in-store situation Happy hour, delivery fee rates change, deals expire, toppings out of stock, etc. Existing pricing logic to match is 15+ years of complicated option-driven business logic locked in hardware and software running in each restaurant with store-level configuration, pricing, and status information. None of these are (yet) available to our serverless system, online
Challenging to configure, expensive to run (1+ VM per store), no status Rewrite all existing pricing logic to run in the cloud Extremely challenging to code and configure; would require years to do right Rewrite some subset of existing pricing logic in the cloud Too limiting for the business Connect the cloud to existing pricing logic already running in stores Higher latency, but business’s preferred approach
a pool of SQS queues When asked for a queue, it looks for an existing queue with no current lease If there is no available queue, it will make a new one and add it to the pool It will assign a timed queue lease to the calling Lambda instance It stores its data in a simple DynamoDB table DynamoDB writes to assign/claim a lease will fail if the record was changed since lookup Then it just retries everything Only returns once the lease has been confirmed AWS Lambda Amazon DynamoDB Amazon Simple Queue Service (Amazon SQS)
use Long-Polling We call SQS asking for a message, but tell it to wait if there is none Wait time can be up to 20 seconds When SQS gets a new message, it immediately sends it to a waiting worker Compared to checking every 2s in a loop: This saves 90% on cost This saves ~1s of latency (0-2s random distribution) Amazon Simple Queue Service (Amazon SQS)
been failure-prone A well-meaning employee might turn off the designated Primary Inbound connections can only go to one server Unless proxied/load-balanced Price requests are single-threaded and block the next request Several idle In-Store Terminals could easily process requests Take advantage: Make multiple In-Store Terminals watch for pricing requests
Generate a Request ID (UUID or Hash) Create response-queue-restricted IAM credentials via STS Include these in the pricing request In-Store server must include Request ID with the response Pricing Lambda will ignore any non-matching responses (presumed stale) AWS Simple Token Service (STS)
200-1000 requests per day Very busy store may process 10,000 requests per day (Still <1RPS peak) Yearly store request rate from 50K to 4M, averaging ~200K Total Average store costs $3.15 to connect and $0.62 to process messages Very busy store costs $3.15 to connect and $12.40 to process messages Business finds this profitable
was cost-effective and profitable Versus old integration approach: Easier setup (minutes instead of hours) Better security (no open ports) More robust (eliminated in-store SPoF) Quicker responses (parallel calls) Created helpful business analytics stream (real usage patterns) Cleared a path for future dev (proven connection; source of truth)
only rises when business value rises Zero cost environments Great fit with Immutable Infrastructure via Infrastructure as Code (IaC) (Externalize data for long-term envs) Dev environments to suit New environment per push, for QA & forensics But…
Cloud Summit talk, I wanted some short links I could later update as needed Quickly whipped up a link redirector Control Tower & Organizations to make a new AWS account Route 53 for domain registration (by far the slowest step!) Route 53 for DNS (automatic) AWS Certificate Manager for TLS cert CloudFront for HTTPS & caching API Gateway for endpoint handling Later: DynamoDB for reading links Even Later: Cognito for auth, writing links to DynamoDB
than most realize! Moves requests in & responses out Remembers configuration Transforms requests & responses Apache Velocity Template Language (VTL) Amazon API Gateway
Cloud Summit talk, I wanted some short links I could later update as needed Quickly whipped up a link redirector Control Tower & Organizations to make a new AWS account Route 53 for domain registration (by far the slowest step!) Route 53 for DNS (automatic) AWS Certificate Manager for TLS cert CloudFront for HTTPS & caching API Gateway for endpoint handling Later: DynamoDB for reading links Even Later: Cognito for auth, writing links to DynamoDB
Cloud Summit talk, I wanted some short links I could later update as needed Quickly whipped up a link redirector Control Tower & Organizations to make a new AWS account Route 53 for domain registration (by far the slowest step!) Route 53 for DNS (automatic) AWS Certificate Manager for TLS cert CloudFront for HTTPS & caching API Gateway for endpoint handling Later: DynamoDB for reading links Even Later: Cognito for auth, writing links to DynamoDB
1000s of stores reliably & securely via SQS & friends Preserving Purity Pure serverless architecture offered Immutable Infra & forensic envs Promoting Practical Progress Innovatively compose systems from service features that Remember, Transform, & Move data
mattias
zozotech
megascus
masahirokawahara
ykiyota
civitaspo
tetsuyaooooo
budougumi0617
jinwatanabe
tkc66buzz
news_it_enj
hponka
masaruogura
lara
tammyeverts
lfama
katarinadahlin
afnizarnur
mkilby
brettharned
ivargrimstad
speakerdeck
tonyrice
mfonobong
uxyall
