Building Modern Applications on AWS with Persistent Storage

Development teams are modernizing their applications by adopting containers, serverless, and microservices-based architectures. Because containers are transient in nature, long-running applications can benefit from keeping state in durable storage. Distributed applications like machine learning training, and web serving, benefit from a shared storage layer. Amazon Elastic File System (Amazon EFS) is a simple, scalable, fully managed, elastic, cloud native shared file system, enabling you to build modern applications, persist and share data from your AWS containers and serverless applications, with zero management required.

🎙 Speaker

This presentation will be led by Ananth Vaidyanathan:

Ananth Vaidyanathan is a Sr. Product Manager at AWS. Having spent time on different teams across Amazon, he has business and technical expertise in helping customers migrate to, and innovate, and optimize with best practices on AWS. He loves thinking about new ideas and creating new products that bring value to customers on the cloud. In his spare time, he is an avid reader of world history and an amateur violin player.

https://www.youtube.com/watch?v=JVEeMoNB7xQ

https://www.meetup.com/AWS-Dubai/events/277995867/

Transcript

1. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Ananth Vaidyanathan Sr. Product Manager, Amazon EFS Persistent Storage for Modern Apps Simple, serverless, set-and-forget storage solution
2. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Agenda  Intro to Amazon Elastic File System (EFS)  Deep Dive and Use Cases  Modern Application Development with EFS  Containers  Lambda  Demos!  Security  Developer and Administrative best practices
3. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Amazon EFS meets you where you are today and tomorrow Migrate: lift and shift to AWS cloud without refactoring application Optimize: enable cost efficiency Modernize: build micro-services into application with common data platform Innovate: improve development efficiency, build new features, enter new markets Migrate Optimize Innovate Modernize Operate
4. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. EFS is serverless, shared file storage VPC VPC Inter-Region VPC peering Intra-Region VPC Peering Other AWS Regions On-premises servers AWS Direct Connect AWS VPN Amazon EC2 Amazon EKS Amazon ECS Amazon SageMaker NFS clients NFS clients Amazon EFS AWS Fargate AWS Lambda
5. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Performant and cost-optimized Serverless shared storage Full AWS compute integration EC2 Instances, containers, and serverless Supports 10,000s of connections Four storage classes Automatic lifecycle-based cost optimization Serverless and scalable No provisioning, scale capacity, connections, and IOPS Simple, serverless, set-and-forget, elastic file system for AWS compute Amazon EFS Simple and highly reliable Elastic Pay only for capacity used Performance built in, scales with capacity Highly durable and available Designed for 11 9s of durability 99.99% availability SLA Performant 10s of GB/s of throughput and 500,000+ IOPS
6. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Availability Zone Availability Zone Availability Zone Amazon EFS Highly durable and available – Amazon EFS Standard AWS Backup
7. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Availability Zone *Because EFS One Zone storage classes store data in a single AWS Availability Zone, data stored in these storage classes might be lost in the event of a disaster or other fault that affects all copies of the data within the Availability Zone, or in the event of Availability Zone destruction. Highly durable and available – Amazon EFS One Zone Amazon EFS AWS Backup
8. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. AWS IAM AWS KMS Amazon CloudWatch AWS CloudTrail AWS CloudFormation Amazon ECS Amazon EC2 Amazon SageMaker Amazon EKS AWS Backup AWS DataSync AWS Direct Connect AWS VPN AWS Fargate AWS Lambda AWS Systems Manager Sep ‘20 June ‘20 July ‘20* Amazon EFS Amazon VPC *Added EKS + Fargate support. ECS + Fargate has been supported since April ‘20 AWS Transfer Jan ‘21 AWS Integrations
9. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. No minimum commitments or upfront fees No need to provision storage Choice of storage classes, automated lifecycle management Optimized for all compute: EC2, Spot, containers, serverless Cost optimized
10. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Modern Application Development
11. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Why modernize? 3. Develop and deploy applications with greater efficiency 2. Increase agility by instantly scaling up according to demand 1. Save cost by reducing operations burden, underutilization of compute and storage
12. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Traditional storage is not designed for modern applications Lack of scalability Administrative overhead Lack of agility
13. Many containerized applications need persistent storage Long-running stateful applications Shared datasets Developer tools Content management Machine learning Shared notebooks WordPress Drupal JIRA Git Artifactory MXNet TensorFlow SageMaker Jupyter Jupyterhub
14. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Data science with EFS Home Directories Shared Project Folders Managed Containers Amazon EKS Amazon ECS AWS Fargate* Amazon EC2 Compute Automation Machine Learning Amazon SageMaker Jupyter SageMaker AWS Auto Scaling Amazon EFS *announced, not currently available AWS Lambda
15. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Amazon EFS for Lambda • Share data across 1000s of function invocations • Achieve high performance, highly available, durable storage with persistent volumes • Pay only for what you use Availability zone Availability zone EFS Mount Target EFS Mount Target Amazon EFS file system AWS Lambda
16. Enable new workloads on AWS Lambda with EFS Large file data manipulation Large scale media processing AI/ML analytics Realtime applications High res images HD VIDEOS Zip/Archives Git MXNet TensorFlow Content management Web apps Simplify application architecture Process files of any size Reduce costs
17. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Perform AI/ML and Analytics with AWS Lambda Import large AI models quickly Load extra code libraries Deploy Modeling and AI solutions with Lambda AWS Lambda Amazon EFS TensorFlow PyTorch, NumPy, Keras Pre trained AI Models C++ Libraries … /tmp
18. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Streamline Media Processing on the Serverless Platform One location for ALL files Simplify Application Architecture Process files of any size Reduce Costs File system AWS Lambda Amazon EFS Images Videos Audio Recordings
19. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. When should I use EFS vs EBS? • I need to share data between containers • I’d like to run across instances or AZs • I’d like to take advantage of spot pricing • I don’t need shared storage (e.g. Database) • I have high amount of small file IO for a transactional app Amazon Elastic Block Store Amazon Elastic File System
20. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. ” “ ” “ Asurion builds on-demand ML using AWS Lambda & Amazon EFS Perform real-time analysis of customer experience during support calls using ML. Call recordings didn’t fit in Lambda /tmp space. Asurion uses Amazon EFS to give extended storage space to their ML functions running in AWS Lambda. • ML inference infrastructure scales elastically with call volume • Reduced operational overhead compared to maintaining instances and auto-scaling. Solution Challenge Benefits We really wanted to use AWS Lambda to make our ML inference elastic, but thought we wouldn’t be able to because of the size of data the process required. With Amazon EFS, we were easily able to give our function all of the storage space it needs. – Jeff Tougas, Senior Principal Software Engineer, Asurion Company: Asurion Industry: Insurance Services Country: United States Employees: 10K+ Website:asurion.com About Asurion We are the go-to solution for all things tech – our Experts can repair, replace and resolve nearly any tech issue. We’re easy to reach via call, chat, and in-person, too – at one of our convenient uBreakiFix stores or pick a time and place and we’ll to come to you. With our passion for helping people stay connected to their tech, we’re making lives a little bit easier—and their tech a lot more amazing.
21. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. ” “ ” “ Acquia Modernizes Web Hosting With Amazon EKS & Amazon EFS Sought to further improve ability to elastically scale across compute and storage to improve end customers digital experience. Containerize hosting application and run on Amazon EKS, using Amazon EFS as persistent storage. • Dynamic scaling of customer environments • Lower TCO through improved storage and compute utilization. • Reduced administrative burden by leveraging fully managed services Solution Challenge Benefits By containerizing our hosting applications and running them on Amazon EKS and Amazon EFS we have improved our customer experience while considerably reducing our infrastructure and operational maintenance overhead. – Jake Farrell, Senior Director of Engineering, Acquia Company: Acquia Industry: IT Country: United States Employees: 1k+ Website:acquia.com About Acquia Acquia’s software and services were built around Drupal to give enterprise companies the ability to build, operate, and optimize websites, apps, and other digital experiences. Our products include: Acquia Cloud, Dev Studio, Site Studio, Edge CDN, Site Factory, Acquia Lightning, Cloud IDE, Acquia DAM, Personalization, Customer Data Platform, Campaign Studio, Campaign Factory, and several others for the developer experience. Get the most out of Drupal and future-proof your digital strategy.
22. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Setting up EFS for modern apps
23. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Goals for Security & Identity 1. File systems should only be mountable by the applications that need them 2. Apps that mount file systems should only have access to the data they need Amazon EFS File System $ cat /my_app/data ### SUCCESS THIS IS MY FILE ### $ cat /someone_elses_app/data cat: /someone_elses_app/data : Permission denied
24. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Handling EFS Authorization Using IAM Anonymous Task Task “semitrust” Task “fulltrust” “Effect” : “allow”, “Action” : “elasticfilesystem:ClientMount”, “Principal” : “*” “Effect” : “allow”, “Action” : [“elasticfilesystem:ClientMount”, “elasticfilesystem:ClientWrite”], “Principal” : { “AWS”: “semitrust” } “Effect” : “allow”, “Action” : [“elasticfilesystem:ClientMount”, “elasticfilesystem:ClientWrite”, “elasticfilesystem:ClientRootAccess], “Principal” : { “AWS”: “fulltrust” } Squashed to 65535
25. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Understanding Container Identity ECS Task Task Identity (IAM Role) AWS IAM Container Image App Identity User: Root Group: Root $ ls –l /efs/home drwx------ bob . BobHome drwx------ sally . SallyHome drwxrwx--- . biusers BI_Shared By default, POSIX identity comes from the container image, not the task/pod runtime.
26. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Understanding Function Identity Lambda Function Task Identity (IAM Role) AWS IAM $ ls –l /efs/home drwx------ bob . BobHome drwx------ sally . SallyHome drwxrwx--- . biusers BI_Shared By default, Lambda functions have no pre-determined identity
27. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Application-specific Access with EFS Access Points { “Name”: “MyApp”, "FileSystemId": ”fs-deadbeef", “PosixUser”: { “Uid”: 123 “Gid”: 123, “SecondaryGids”: [100, 200, 300] }, “RootDirectory”: { “Path”: “/apps/myapp”, “CreationInfo”: { “OwnerUid”: 123, “OwnerGid”: 123, “Permissions”: “0700” } } } Creates App-specific Directory & Permissions No EC2 instance required! Apps only see data they need Enforces File System Identity Root containers can’t escalate access Arbitrary users aren’t locked out ECS EKS Lambda
28. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. { “Name”: “MyApp”, “PosixUser”: { “Uid”: 123 “Gid”: 123, “SecondaryGids”: [100, 200, 300] }, “RootDirectory”: { “Path”: “/apps/myapp”, “CreationInfo”: { “OwnerUid”: 123, “OwnerGid”: 123, “Permissions”: “0700” } } } How EFS Access Points Work File System with POSIX Permissions “Effect” : “allow”, “Action” : “elasticfilesystem:Client*”, “Principal” : { “AWS”: “approle” }, “Condition”* : {“accessPointArn” : “fsap-1234” ECS EKS Lambda
29. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Amazon EFS with Amazon ECS Task Container 1 Container 2 Amazon ECS Amazon EC2 AWS Fargate Amazon EFS File system EFSVolumeConfiguration Amazon Elastic Container Service (Amazon ECS)
30. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Attaching Amazon EFS to a function 1. Create 1. File System 2. Mount Targets 3. Access Point 2. Configure function for VPC of file system 3. Add file system 1. Select File System 2. Select Access Point
31. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Security and Best Practices
32. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Security and compliance Control client access Using Amazon VPC security groups and network ACLs Using IAM identity and resource policies Control file and directory access Using POSIX permissions Using access points Encrypt data At rest and in transit Achieve compliance HIPAA GDPR PCI-DSS SOC ISO FedRAMP Control administrative (API) access Using AWS IAM, action- level and resource-level permissions, and identity-based policies
33. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Amazon EFS | Developer best practices Test in General Purpose performance mode Start with Bursting Throughput mode Linux kernel 4.3+ Large I/O size (aggregate I/O) Multiple threads Multiple instances Multiple directories Consider Provisioned Throughput mode for loading >2.1 TB EFS mount helper (NFSv4.1)
34. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Amazon EFS | Administrative best practices Create a backup plan to further protect your data Enable encryption at rest and in transit for sensitive workloads Enable lifecycle management to automatically save up to 92% Monitor throughput utilization, burst credits, and PercentIOLimit Simplify EFS client management with AWS Systems Manager Use AWS Budgets for cost management
35. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Resources to get started • EFS Documentation: https://docs.aws.amazon.com/efs/index.html • EFS reinvent 2020 sessions: https://aws.amazon.com/blogs/storage/8-cant-miss-reinvent-2020- sessions-with-amazon-efs/ • EFS blogs: • Containers Blog: https://aws.amazon.com/blogs/containers/category/storage/amazon- elastic-file-system-efs/ • Storage Blog: https://aws.amazon.com/blogs/aws/category/storage/amazon-elastic-file- system-efs/ • Storage Blog: https://aws.amazon.com/blogs/compute/category/storage/amazon-elastic- file-system-efs/
36. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Thank you!
37. © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark. Many customers use Amazon EFS