Amazon S3 NYJavaSIG 2024-12-12

Amazon S3
NYJavaSIG
New York, NY

Transcript

1. Amazon S3 NYJavaSIG 2024-12-12 Sean Sullivan
2. AWS Developers
3. Agenda AWS re:Invent 2024 Amazon S3 AWS SDK Bonus topics
4. AWS re:Invent 2024 S3 Tables S3 Metadata
5. AWS re:Invent 2024 fully managed Iceberg tables
6. AWS re:Invent 2024 “Table buckets”
7. AWS re:Invent 2024 Iceberg tables as fi rst-class AWS resources
8. AWS re:Invent 2024
9. AWS re:Invent 2024 pom.xml
10. AWS re:Invent 2024 automatic generation of metadata that is captured when S3 objects are added or modi fi ed stored in fully managed Apache Iceberg tables
11. Amazon S3 “Simple Storage Service” Launched on March 14, 2006
12. Amazon S3 S3 is an object storage service with an HTTP REST API https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
13. Amazon S3 “There is a frontend fl eet with a REST API, a namespace service, a storage fl eet that’s full of hard disks, and a fl eet that does background operations.” https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
14. S3 core concepts Buckets Objects
15. S3 core concepts An Amazon S3 object represents a fi le or collection of data Every object must reside within a bucket
16. S3 bucket types General purpose buckets Directory buckets Table buckets
17. S3 bucket names an Amazon S3 bucket name is globally unique the namespace is shared by all AWS accounts
18. S3 pricing https://aws.amazon.com/s3/pricing/ “You pay for storing objects in your S3 buckets. The rate you’re charged depends on your objects' size, how long you stored the objects during the month, and the storage class”
19. S3 storage classes https://aws.amazon.com/s3/storage-classes/ “Amazon S3 o ff ers a range of storage classes that you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.”
20. S3 storage classes https://aws.amazon.com/s3/storage-classes/ Standard Intelligent Tiering Express One Zone … and many others
21. Storage class choice matters https://www.youtube.com/watch?v=RxgYNrXPOLw
22. S3 REST API 3314 pages
23. S3 operations Upload object List objects Download object Copy Move Delete
24. using S3 in a Java application
25. AWS SDK for Java v1 AWS SDK for Java v2 AWS SDK for Kotlin
26. AWS SDK for Java v1 https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/
27. Open source SDK’s
28. AWS SDK for Java v2 pom.xml
29. AWS SDK for Java v2 pom.xml
30. AWS SDK for Java v2 : HTTP clients Apache Client Netty Client CRT Client
31. AWS SDK for Java v2: CRT client pom.xml
32. CRT @ AWS re:Invent 2024 https://youtu.be/2DSVjJTRsz8?t=833
33. Di ff erent fl avors of S3 clients Async Sync
34. how to create an S3 bucket?
35. Creating an S3 bucket AWS Console UI AWS CLI AWS SDK CloudFormation AWS CDK Terraform Pulumi Infrastructure as Code Other
36. CloudFormation
37. Pulumi
38. s3-playground https://github.com/sullis/s3-playground
39. s3-playground https://github.com/sullis/s3-playground
40. testing S3 locally Localstack MinIO Adobe S3Mock Testcontainers Testcontainers Testcontainers
41. S3 with MinIO S3MinioTest.java
42. how to upload an object? PutObjectRequest
43. how to retrieve an object? GetObjectRequest
44. how to upload large objects? CreateMultipartUploadRequest
45. parallel uploads? S3TransferManager
46. Big Data analytics?
47. Iceberg @ Net fl ix 2018 June 2018 https://www.youtube.com/watch?v=nWwQMlrjhy0 S3
48. Apache Iceberg 2024 A table format is a method of structuring a dataset’s files to present them as a unified “table.”
49. Apache Iceberg 2024 In a data lake, all your data is stored as files in some storage solution (e.g. Amazon S3)
50. AWS re:Invent 2023 Ryan Blue
51. AWS re:Invent 2023 S3
52. AWS re:Invent 2023
53. AWS re:Invent 2023 “Too many small fi les are a problem”
54. Apache Iceberg project https://github.com/apache/iceberg
55. Iceberg pull request https://github.com/apache/iceberg/pull/11349
56. S3 Conditional Writes https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ Conditional writes can ensure there is no existing object with the same key name in you bucket during PUT operations
57. S3 bucket permissions https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleCon fi guration.html By default, all Amazon S3 resources are private, including buckets, objects, and related subresources
58. S3 performance considerations?
59. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 pre fi x”
60. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "There are no limits to the number of pre fi xes in a bucket. You can increase your read or write performance by using parallelization”
61. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html “While Amazon S3 is scaling to your new higher request rate, you may see some 503 (Slow Down) errors. These errors will dissipate when the scaling is complete.”
62. KubeCon November 2024
63. re:Invent December 2024
64. The End
65. Bonus content
66. AWS re:Invent 2024
67. AWS re:Invent 2024
68. AWS re:Invent 2024
69. AWS re:Invent 2024
70. AWS re:Invent 2024
71. AWS re:Invent 2024