How LEGO.com accelerates innovation with serverless

AWS re:Invent 2020

Mostly presented by Sheen Brisals, AWS Serverless Hero, Senior Engineering Manager, The LEGO Group.

After experiencing scaling issues on Black Friday, the LEGO team fully refactored its monolith to serverless microservices on AWS. In this session, LEGO Senior Engineering Manager and AWS Serverless Hero Sheen Brisals shows you how LEGO.com rebuilt a monolith into an event-driven architecture with Amazon EventBridge and AWS Step Functions at its core. He also discusses how designing with a serverless-first mindset can provide surprising benefits. Come learn about successful architectural patterns and see how LEGO.com manages and grows its serverless team to deliver more in less time at the scale of millions of customers.

Transcript

1. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How LEGO.com accelerates innovation with serverless Sheen Brisals AWS Serverless Hero Senior Engineering Manager The LEGO Group Danilo Poccia Chief Evangelist (EMEA) AWS S V S 3 0 2
2. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Last year at re:Invent Accelerating with AWS Step Functions Accelerating with Amazon EventBridge Serverless team growth Business value acceleration Agenda
3. Last year at re:Invent © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. The serverless journey of shop.LEGO.com S V S 3 2 0 Sheen Brisals Senior Application Engineer The LEGO Group Danilo Poccia Principal Evangelist Amazon Web Services A journey through patterns Use case Pattern ✓ Add item to shopping basket Atomic request-response API ✓ Status polling for long-running processes CQRS with status cache ✓ Voucher codes generation and notification Email notification with signed URL ✓ User identity lookup in different systems API authorizer with identity lookup ✓ On-demand customer data migration Publish-subscribe sync ✓ Product catalog import and update Event-driven data pipeline with buffering ✓ API-driven data ingestion Codeless data ingestion ✓ Unique order number generation Codeless sequence generator ✓ Website migration with URL changes URL redirects cached by CDN ✓ Keeping website sitemaps updated Scheduled workflow ✓ Checkout event processing Hub-and-spoke event bus Pattern – Codeless data ingestion Kinesis Data Firehose API Gateway S3 bucket Event producer • API to stream directly • Less point of failures • Native integration • Less compute costs • Less code to maintain • Fully managed and scalable Transformation and validation Backup Fan-out function Processing functions Trigger Async Errors Pattern – Hub-and-spoke event bus { "version": "0", "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", "detail-type": "State change Notification", "source": "service-order-submit-dev", "account": "123456789012", "time": "2019-08-29T12:10:21Z", "region": "eu-central-1", "resources": ["arn:aws:events:event-bus/checkout-bus"], "detail": { "event": { "meta_data": { "site_id": "LEGO Shop", "type": "CHECKOUT", "subtype": "ORDER", "status": "COMPLETE" }, "data": { "order_number": "T123456789", "customer_id": "bf3703467718-29T12-6a7e8feb" } } } } Standard syntax across multiple services Custom for each service
4. Accelerating with serverless
5. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Accelerating with AWS Step Functions
6. Catalog feed generation in parallel U S I N G D Y N A M I C P A R A L L E L I S M Timer Process Product catalogs Publish Social media
7. Dynamic parallelism AWS Step Functions Trigger rule Amazon CloudWatch Events
8. Dynamic parallelism { "CreateFeedsForCountries": { "Type": "Map", "ItemsPath": "$.countryList", "MaxConcurrency": 5, "Iterator": { "StartAt": "CreateCountryFeed", "States": { "CreateCountryFeed": { "Type": "Task", "Resource": “feed-create", "End": true } } } } }
9. Failure message replay with state machine Recommendations engine Product feeds Failed product feeds Check status Process failed feeds
10. Failure event Failed feeds DLQ Failure message replay with state machine Product recommendation feed generator Product feed store Product feed dispatcher Recommendations API Alarm channel notifier Resumption event Failure event Failure messages Resumption event Reprocess failed feeds (AWS Step Functions) Service state Event bus (Amazon EventBridge)
11. In a way, this is implementing the circuit breaker pattern using AWS Step Functions Failure message replay with state machine
12. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Accelerating with Amazon EventBridge
13. Back-in-stock notification Email store LEGO.com Notify customer Stock checker Customer
14. Back-in-stock notification Email registration Product status check Customer notification Email dispatch Email feedback Business insights Registration request Registration response Registration request Registration response Product status Product available Amazon SES Email feedback events Email feedback events Status events Amazon EventBridge
15. Choreographing back-in-stock notification A D V A N T A G E S O F C H O R E O G R A P H Y V S . O R C H E S T R A T I O N Registration request Registration API Registration request Product status check Product available Product available Notification orchestrator Fetch emails Email dispatch queue Email dispatcher Feedback events Feedback events Request handler Store registrations Amazon SES Email store Amazon EventBridge
16. Choreographing back-in-stock notification Registration request Registration API Email store Registration request Product status check Product available Product available Notification orchestrator Fetch emails Email dispatch queue Email dispatcher Feedback events Feedback events Amazon SES Amazon EventBridge
17. Orchestrating the notification process Product available
18. Email feedback event stream with batching Amazon SES Amazon S3 bucket Amazon Kinesis Data Firehose AWS Lambda function Feedback event Buffering events Storing events Grouping events Amazon EventBridge event bus Batched feedback event Success events Error events Reject, Bounce, Complaint, Send, Open, Click Event with an array of custom data Business insights Error feedback handling A single EventBridge event contains multiple feedback events
19. Email feedback event stream with batching Amazon SES S3 bucket Kinesis Data Firehose Lambda function Buffering events Storing events Business insights Error feedback handling Success events Error events Efficient data ingestion pattern Patterns help to accelerate your serverless adoption Batched feedback event Feedback event Grouping events EventBridge event bus
20. https://bit.ly/2FZ1b1a
21. Order placement flow Checkout Submit order Payment Authorize payment Order Process order Order submitted Order submitted Order canceled Payment authorized Payment canceled Payment authorized Order confirmed Order canceled Order confirmed Shipping Orders to SAP Amazon EventBridge
22. Orchestration inside order processing
23. • Use choreography to coordinate different microservices § For example, with Amazon EventBridge • Use orchestration inside microservices, or across different microservices if you need a distributed transaction § For example, using AWS Step Functions Choreography vs. orchestration
24. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Serverless team growth
25. Start of our serverless journey Serverless team Front-end team • Small focus team • Build skills and expertise • Expand incrementally
26. Feature-driven and full-stack squads payments rewards platform content insights new products checkout • Feature squads • Full stack • Distributed skills
27. • Feature squads with growing serverless knowledge • Engineers become architecture-aware • Engineers are part of the DevOps process • Adapting best practices, including Serverless Lens • Service ownership – we build, we run! • Sharing knowledge with the community
28. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Business value acceleration
29. Scalability gives business value too
30. • Faster feature development • Stakeholder trust in delivering value • Increased operational visibility • Scalable, serverless platform that serves more customers Business value acceleration
31. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sheen Brisals Twitter @sheenbrisals Danilo Poccia Twitter @danilop