Amazon S3 - Portland Java User Group 2024-09-17

Portland Java User Group
Amazon S3
September 17, 2024

#aws
#awscloud

Transcript

1. Amazon S3 Portland Java User Group 2024-09-17 Sean Sullivan
2. Agenda Amazon S3 AWS SDK Code Bonus topics
3. Amazon S3 “Simple Storage Service” Launched on March 14, 2006
4. Amazon S3 S3 is an object storage service with an HTTP REST API https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
5. Amazon S3 “There is a frontend fl eet with a REST API, a namespace service, a storage fl eet that’s full of hard disks, and a fl eet that does background operations.” https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
6. S3 core concepts Buckets Objects
7. S3 core concepts An Amazon S3 object represents a fi le or collection of data Every object must reside within a bucket
8. S3 pricing https://aws.amazon.com/s3/pricing/ “You pay for storing objects in your S3 buckets. The rate you’re charged depends on your objects' size, how long you stored the objects during the month, and the storage class”
9. S3 storage classes https://aws.amazon.com/s3/storage-classes/ “Amazon S3 o ff ers a range of storage classes that you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.”
10. S3 storage classes https://aws.amazon.com/s3/storage-classes/ Standard Intelligent Tiering Express One Zone … and many others
11. S3 storage classes https://aws.amazon.com/s3/storage-classes/ You can con fi gure S3 storage classes at the object level, and a single general purpose bucket can contain objects stored across all storage classes except S3 Express One Zone
12. Storage class choice matters https://www.youtube.com/watch?v=RxgYNrXPOLw
13. S3 REST API 1917 pages
14. S3 operations Upload object List objects Download object Copy Move Delete
15. using S3 in a Java application
16. AWS SDK for Java v1 AWS SDK for Java v2 AWS SDK for Kotlin
17. AWS SDK for Java v1 https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/
18. Open source SDK’s
19. AWS SDK for Java v2 pom.xml
20. AWS SDK for Java v2 pom.xml
21. AWS SDK for Java v2 Apache Client Netty Client CRT Client
22. AWS SDK for Java v2: CRT client pom.xml
23. Di ff erent fl avors of S3 clients Async Sync
24. how to create an S3 bucket?
25. Creating an S3 bucket AWS Console UI AWS CLI AWS SDK CloudFormation AWS CDK Terraform Pulumi Infrastructure as Code Other
26. CloudFormation
27. Pulumi
28. Code https://github.com/sullis/s3-playground
29. s3-playground https://github.com/sullis/s3-playground
30. testing S3 locally Localstack Minio Adobe S3Mock Testcontainers Testcontainers Testcontainers
31. S3 with Localstack S3LocalstackTest.java
32. how to upload an object? PutObjectRequest
33. how to retrieve an object? GetObjectRequest
34. how to upload large objects? CreateMultipartUploadRequest
35. parallel uploads? S3TransferManager
36. Bonus Topics
37. Big Data analytics?
38. Iceberg @ Net fl ix 2018 June 2018 https://www.youtube.com/watch?v=nWwQMlrjhy0 S3
39. Apache Iceberg 2024 A table format is a method of structuring a dataset’s files to present them as a unified “table.”
40. Apache Iceberg 2024 In a data lake, all your data is stored as files in some storage solution (e.g. Amazon S3)
41. AWS Re:Invent 2023
42. AWS Re:Invent 2023
43. AWS Re:Invent 2023 S3
44. AWS Re:Invent 2023
45. AWS Re:Invent 2023
46. AWS Re:Invent 2023 “Too many small fi les are a problem”
47. Apache Iceberg project https://iceberg.apache.org/
48. Apache Iceberg project https://github.com/apache/iceberg
49. Iceberg pull request https://github.com/apache/iceberg/pull/10217 work-in-progress
50. zero disk architecture?
51. Zero Disk Architecture November 2023 September 2024
52. S3 Conditional Writes https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ Conditional writes can ensure there is no existing object with the same key name in you bucket during PUT operations
53. S3 Storage Lens https://aws.amazon.com/s3/storage-lens/ “S3 Storage Lens delivers organization-wide visibility into object storage usage, activity trends, and makes actionable recommendations to optimize costs ”
54. S3 bucket permissions https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleCon fi guration.html By default, all Amazon S3 resources are private, including buckets, objects, and related subresources
55. S3 performance considerations?
56. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 pre fi x”
57. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "There are no limits to the number of pre fi xes in a bucket. You can increase your read or write performance by using parallelization”
58. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html “While Amazon S3 is scaling to your new higher request rate, you may see some 503 (Slow Down) errors. These errors will dissipate when the scaling is complete.”
59. how can I migrate from AWS SDK v1 to AWS SDK v2 ?
60. https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-tool.html OpenRewrite recipe
61. https://repo1.maven.org/maven2/software/amazon/awssdk/v2-migration/ OpenRewrite recipe in Maven Central
62. https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-tool.html running the OpenRewrite recipe gradle rewriteRun \ --init-script init.gradle \ -Drewrite.activeRecipes=software.amazon.awssdk.v2migration.AwsSdkJavaV1ToV2
63. Questions
64. The End