Share
Embed
The serverless journey of shop.LEGO.com
re:Invent, Las Vegas, December 2nd, 2019
Connecting the LEGO play experience with millions of people requires an innovative platform. This has fueled the cloud migration of the legacy e-commerce application. In this session, we walk you through the principles, the approach, the learnings, and of course the serverless technologies that made the vision a reality. We cover multiple real-world use cases such as the integration of the e-commerce platform with the tax system, and the implementation of an event-streaming platform.
Danilo Poccia
December 02, 2019
Transcript
68 slides
-
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. The serverless journey of shop.LEGO.com S V S 3 2 0 Sheen Brisals Senior Application Engineer The LEGO Group Danilo Poccia Principal Evangelist Amazon Web Services -
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. -
The “monolith” at the start of this journey Corporate data
center Server contents Server contents Tax Products CRM Fulfillment Rewards Server contents -
The “monolith” at the start of this journey VPC Node.js
+ React Elastic Beanstalk container Corporate data center Server contents Server contents Tax Products CRM Fulfillment Rewards Server contents -
The “monolith” at the start of this journey VPC Node.js
+ React Elastic Beanstalk container Server contents Server contents Tax Products CRM Fulfillment Rewards VPC Gateway Corporate data center Server contents -
Then, during Black Friday 2017 … VPC Node.js + React
Elastic Beanstalk container Server contents Server contents Tax Products CRM Fulfillment Rewards VPC Gateway 503 Service Unavailable Corporate data center Server contents -
September 2018 – Decoupling the backend VPC Node.js + React
Elastic Beanstalk container Server contents Server contents Tax Products CRM Fulfillment Rewards AWS Cloud SaaS VPC Gateway Server contents Corporate data center Amazon API Gateway AWS Lambda function -
AWS services in use Amazon S3 Amazon SNS AWS Fargate
AWS Lambda Amazon Kinesis Data Firehose Amazon DynamoDB Amazon API Gateway AWS Step Functions Amazon SQS Amazon EC2 Amazon ECS Amazon ECR Amazon VPC AWS WAF AWS KMS AWS Identity and Access Management (IAM) AWS X-Ray Amazon Route 53 AWS CloudTrail Amazon ElastiCache Amazon CloudWatch Amazon EventBridge AWS CloudFormation Parameter Store -
Growing usage stats from production 165+ Lambda functions 35+ microservices
30+ API Gateway endpoints 10+ DynamoDB tables 5+ S3 buckets 20+ SNS topics 60+ SQS queues + DLQs 70+ Systems Manager params -
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. -
Use case – Add item to shopping basket Customer Product
Validation Basket -
Pattern – Atomic request-response API Shopping app Add item API
Business logic Basket data store Commerce platform Request Response Invoke Check/add • Performance is priority • Logic evaluation • No partial failure • Success or error response • Validation steps • Limit data relay -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API -
Use case – Status polling for long-running processes “What’s the
status of my order?” -
Pattern – CQRS* with status cache Shop POST GET Submit
order API Submit order function Submit order queue Process order function Update status Order status API Status provider function Order cache (TTL) Order number Query status Payment providers New commerce platform Frontend visibility Processing Failed Complete Backend visibility Validating Validation failed Payment authorized Payment processing Card declined Sent to fulfillment * Command Query Responsibility Segregation -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache -
Use case – Voucher codes generation and notification Admin Voucher
codes Voucher store Admin File download Email -
Pattern – Email notification with signed URL Admin user Vouchers
API Request handler Generate vouchers Voucher backup Voucher table Voucher topic Email notification Email with Amazon S3 signed URL Acknowledge POST 1. Create vouchers 2. Update voucher table 3. Store batch on Amazon S3 4. Generate Amazon S3 signed URL 5. Send out notification • Vouchers from 1 to 1M • Unique voucher codes • Secure download option • User notification GET Async Sync -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL -
Use case – User identity lookup in different systems Customer
Customer identity Authenticate Authorize Rewards -
What happens if identities don’t match? Customer Profile API Request
handler LEGO identity Client app Rewards platform Commerce platform LEGO ID Rewards ID -
Pattern – API authorizer with identity lookup Customer Profile API
Request handler • Session must be valid • Session must belong to the user • Profile must exist • Profile ID must match LEGO identity Authorizer Commerce platform Rewards platform LEGO ID Rewards ID Client app -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup -
Use case – On-demand customer data migration Customer New site
New platform Notify Process New commerce platform Fetch Update -
Pattern – Publish-subscribe sync • On-demand data sync • Real-time
migration • Data in JSON format • No impact to user experience Customer topic New commerce platform New customer handler Customer created Fetch data Update data -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync -
Use case – Product catalog import and update Commerce platform
Feeds transform Products feeds -
Pattern – Event-driven data pipeline with buffering Transform feed Transformed
feeds queue Feeds store Commerce platform API Update data Data feeds Trigger Push Trigger Update • Feeds in JSON format • Independent feed flow • Immediate handling of feeds • Decoupled processing • Failure retry process • Failure notification DLQ Thousands per day processed as soon as possible Fine-grained control on throughput and errors Observability -
Same pattern works for multiple object types Commerce platform API
Data feeds Trigger Push Trigger Update Push Trigger Push Trigger Product Pricing SKU Observability DLQs -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering -
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. “Code is a liability. Write less functions.” On Twitter Lots of People -
Use case – API-driven data ingestion Processing Data store Data
ingestion pipeline Event producer Ingest API -
A first solution… Can we do better? Lambda function receive
event & send to Kinesis Kinesis Data Firehose buffering API Gateway receive events Processing application S3 bucket store events Event producer -
Pattern – Codeless data ingestion Kinesis Data Firehose API Gateway
S3 bucket Event producer • API to stream directly • Less point of failures • Native integration • Less compute costs • Less code to maintain • Fully managed and scalable Transformation and validation Backup Fan-out function Processing functions Trigger Async Errors -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion -
Use case – Unique order ID generation Checkout flow Loyalty
customers Order number Customer ID -
Pattern – Codeless sequence generator Sequences table Sequence number API
Sequence item • No Lambda function • Repeatable pattern • Fully managed • Stateful resource • Atomic counter • Single table partition Counter attribute Any microservice -
DynamoDB Atomic Counter { "TableName": ”sequences", "Key": { "id": {
”S": ”order" } }, "ExpressionAttributeValues": { ":one": { "N": 1 } }, "UpdateExpression": "SET num = num + :one", "ReturnValues": "UPDATED_NEW" } Atomic update -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion ✓ Unique order number generation Codeless sequence generator -
Use case – Website migration with URL changes Old site
URL New site Identify destination Content provider -
Pattern – URL redirects cached by CDN URL redirect function
Content store URL rules Cache on /tmp Application Load Balancer CDN • 512 MB /tmp storage • 128MB – 3GB memory -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion ✓ Unique order number generation Codeless sequence generator ✓ Website migration with URL changes URL redirects cached by CDN -
Use case – Keeping website sitemaps updated Timer Process Store
sitemaps Publish Search engines -
Pattern – Scheduled workflow Step Functions Trigger rule • Amazon
S3 for storage • Amazon S3 as origin server • Runs daily – off peak • Sitemaps in XML Amazon CloudWatch Events -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion ✓ Unique order number generation Codeless sequence generator ✓ Website migration with URL changes URL redirects cached by CDN ✓ Keeping website sitemaps updated Scheduled workflow -
Use case – Checkout event processing Login Checkout Payment Order
Data sync Shipping -
Use case – Hub-and-spoke event bus Login Checkout Payment Order
Data sync Shipping -
Pattern Customer login Login Shipping Send order to SAP Data
sync Customer, VIP, wishlist sync Checkout Submit order Payment Authorize payment Commerce platform Order Process order Order and customer updates Event relay Customer login Invoke every minute Events Order complete Customer login Payment authorized Order submit Order complete EventBridge FIFO queue -
Pattern – Hub-and-spoke event bus { "version": "0", "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
"detail-type": "State change Notification", "source": "service-order-submit-dev", "account": "111122223333", "time": "2019-08-29T12:10:21Z", "region": "eu-central-1", "resources": [ "arn:aws:events:event-bus/checkout-bus" ], "detail": { } } Customer-specific data goes in the “detail” -
Pattern – Hub-and-spoke event bus { "version": "0", "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
"detail-type": "State change Notification", "source": "service-order-submit-dev", "account": "123456789012", "time": "2019-08-29T12:10:21Z", "region": "eu-central-1", "resources": ["arn:aws:events:event-bus/checkout-bus"], "detail": { "event": { "meta_data": { "site_id": "LEGO Shop", "type": "CHECKOUT", "subtype": "ORDER", "status": "COMPLETE" }, "data": { "order_number": "T123456789", "customer_id": "bf3703467718-29T12-6a7e8feb" } } } } Standard syntax across multiple services Custom for each service -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion ✓ Unique order number generation Codeless sequence generator ✓ Website migration with URL changes URL redirects cached by CDN ✓ Keeping website sitemaps updated Scheduled workflow ✓ Checkout event processing Hub-and-spoke event bus -
A journey through patterns Use case Pattern ✓ Add item
to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion ✓ Unique order number generation Codeless sequence generator ✓ Website migration with URL changes URL redirects cached by CDN ✓ Keeping website sitemaps updated Scheduled workflow ✓ Checkout event processing Hub-and-spoke event bus -
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. -
Takeaways Look for something simple to begin with Implement automated
integration tests Separate dev, test, prod accounts Architect in “set pieces” No throwaway PoCs Leverage patterns -
Thank you! © 2019, Amazon Web Services, Inc. or its
affiliates. All rights reserved. Sheen Brisals @sheenbrisals Danilo Poccia @danilop