Event Driven Architectures on AWS

This talk about event driven architectures was presented on the 26th of May 2020 for the AWS UG Novi Sad in Serbia.

Transcript

1. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS UG Novi Sad – Part 1 Event Driven Architectures Marek Kuczynski Serverless Specialist SA marekku@amazon.nl marekq@
2. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Evolution of compute on AWS • What got me interested in Lambda • Building and deploying serverless apps • 19.00 – short break • Building secure, serverless architectures • How to leverage serverless for security • Where to learn more • Q&A
3. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. We are witnessing a paradigm shift Level of abstraction Focus on business logic Serverless Physical machines Virtual machines Containerization AWS Lambda AWS Fargate • Continuous scaling • Fault tolerance built in • Pay for value • Zero maintenance • Focus on business value
5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon EC2 Instances, containers and functions Boots up in minutes Runs for days or much longer Full OS access AWS Fargate Starts in seconds Runs for minutes/hours Limited OS access AWS Lambda Starts in milliseconds Runs up to 15 minutes No OS access
6. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What got me interested in serverless Back in the day when I was still an AWS customer (2016)
7. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data collection in an EC2 based architecture (2015 and before) EC2 instance collecting data Data forwarder Search interface (Splunk) Application and OS logs Cost, usage, infrastructure and security logs
8. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS data collection around 2015, powered by EC2 https://github.com/marekq/list-ec2 and https://github.com/marekq/list-s3 Listing instances that were running or stopped Listing buckets and objects on S3 that had public access enabled
9. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data collection after discovering Lambda (~2016) Lambda Runs every hour Replaces the instance Bucket receives CSV report Replaces the forwarder Splunk on EC2 Remains unchanged, but easier to ingest historical and live data
10. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Less manual intervention, lower cost and higher availability
11. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. It got even easier afterwards (2016 onwards) AWS can deliver more data directly without requiring Lambda • Data S3 inventory about stored objects can be collected automatically • Infra AWS Config captures changes in your infrastructure • Cost AWS Budgets and billing alert you about cost increases • Security GuardDuty shares various findings through events A lot of my Lambda code became obsolete over night, and this is great!
12. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. You can create an S3 inventory report in less than a minute…
13. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. … or get notified whenever someone puts something public https://aws.amazon.com/blogs/security/how-to-use-aws-config-to-monitor-for-and-respond-to-amazon-s3- buckets-allowing-public-access/ You can monitor your environment top down and automatically remediate any issues. Blocking all public access to S3 is also a lot easier today.
14. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. My cost reporter Lambda has been running daily for ~4 years Source; https://github.com/marekq/s3-cost-explorer (will be updated soon, ping me fore more info!)
15. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building security automation for corporate networks Lambda function pings a cloud hosted or internal system to retrieve data Amazon CloudWatch Schedule AWS Lambda Ping function Instance or VM Alarm Archive output Update database AWS Cloud Corporate data center
16. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless architectures
17. How to process events using Lambda Synchronous (push) Asynchronous (event) Stream-based Amazon API Gateway AWS Lambda function Amazon DynamoDB Amazon SNS /order AWS Lambda function Amazon S3 reqs Amazon Kinesis changes AWS Lambda service function
18. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. You can combine services and build highly available architectures Sources: https://marek.rocks and https://github.com/marekq/marek.rocks
19. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monitoring event driven architectures using X-Ray
20. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 20 Events are the connective tissue of modern applications
21. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 21 Event-driven architectures drive reliability and scalability Asynchronous Events Improve responsiveness and reduce dependencies Event Routers Abstract producers and consumers from each other Event Stores Buffer messages until services are available to process
22. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EventBridge Architecture Partner event source Rules Default event bus Custom event bus SaaS event bus Amazon EventBridge
23. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 23 Lego uses an event-driven design Commerce platform Order & customer updates Event relay Amazon EventBridge Login Customer login Checkout Submit order Order Process order Shipping Send order to SAP Data sync Customer, VIP, wishlist sync Payment Authorize payment FIFO queue Customer login Invoke every minute Order complete Events Payment authorized Customer login Order complete Order submit
24. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 24 How can you build serverless applications?
25. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 25 developers services monitor release test build delivery pipelines monitor release test build monitor release test build monitor release test build monitor release test build monitor release test build Deployment: Microservice development lifecycle
26. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 26 AWS CloudFormation is a greatfoundation tobuildon AWS Code Commit Execute Deploy Write templates in YAML, or JSON, or generate code via macros/transforms or higher-level languages (AWS CDK) Upload templates directly via browser, from an S3 bucket, or ideally via CI/CD pipelines to leverage testing tools Create stacks from templates using the console, CLI, or AWS SDK, or create stack sets across multiple accounts and regions Stacks, stack sets, and resources are created, updated, and managed in a uniform way
27. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 27 However, these solutions can accelerate your development significantly; • AWS Amplify Console and CLI The fastest way to build mobile and web applications • Serverless Application Model (SAM) CLI Build serverless apps using a YAML template with simple, clean syntax • Cloud Development Kit (CDK) Define cloud resources in your favourite programming language Serverless framework options from AWS
28. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 28 AWS Amplify Console
29. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 29 AWS Amplify CLI
30. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 30 AWSTemplateFormatVersion: '2010-09-09’ Transform: AWS::Serverless-2016-10-31 Resources: GetProductsFunction: Type: AWS::Serverless::Function Properties: Handler: index.getProducts Runtime: nodejs10.x CodeUri: src/ Policies: - DynamoDBReadPolicy: TableName: !Ref ProductTable Events: GetResource: Type: Api Properties: Path: /products/{productId} Method: get ProductTable: Type: AWS::Serverless::SimpleTable AWS Cloud AWS SAM templates and CLI Amazon API Gateway Lambda function Table Role === To become this Allowing this
31. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 31 AWS Cloud Development Kit Software development framework for defining cloud infrastructure using familiar programming languages AWS CDK
32. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 32 CDK - from constructs to the cloud AWS CDK
33. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 33 The CDK is a great fit as you can include any AWS resource Source: https://github.com/marekq/sqs-fargate-poller
34. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 34 There are even more framework options… ZAPPA Chalice
35. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 35 AWS Amplify • Check the documentation and samples on https://aws.amazon.com/amplify/ • Follow Nader Dabit (dabit3@) on Twitter to find many awesome Amplify tutorials • Sign up for Amplify Days event on 10/11 June; https://awsamplifydays.splashthat.com/ Serverless Application Repository (SAM) • Find SAM examples on GitHub; https://github.com/awslabs/serverless-application-model • Deploy full serverless apps from the Serverless App Repo, this is a great way to learn!; https://aws.amazon.com/serverless/serverlessrepo/ Cloud Development Kit (CDK) • Run a sample CDK workshop to learn the basics; https://cdkworkshop.com/ • Various GitHub repo’s contain common CDK patterns; https://twitter.com/cdkpatterns Where to learn more
36. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Marek Kuczynski Serverless Specialist SA marekku@amazon.nl marekq@