S3AFileSystem Updates

Lightning Talk in Open Table Format Study Group (OTFSG) Osaka #2: https://otfsg-tokyo.connpass.com/event/395739/

Transcript

1. S3AFileSystem Updates OTFSG Osaka #2 Akira Ajisaka
2. How to access Amazon S3 from Big Data Stack • Hadoop S3AFileSystem (S3A) ◦ Spark ◦ Hive ◦ Hudi ◦ Delta • Iceberg S3FileIO • Trino S3 connector • Presto S3 connector • Flink S3 native connector
3. EMRFS? EMRFS is proprietary implementation in AWS, very similar to S3A S3A is now default in • EMR 7.10.0 and upper • Glue 5.1 Most of the features in EMRFS are also available in S3A • There are some differences between OSS S3A and AWS-forked S3A ◦ AIMD retry policy is currently not available in OSS • If you use OSS, you need to check the OSS documentation
4. Upgrade to AWS SDK for Java 2.x AWS SDK for Java 1.x is now EOL New features after the upgrades • S3 Express One Zone support • Checksum algorithm support ◦ 'CRC32', 'CRC32C', 'SHA1', and 'SHA256' ◦ Latest algorithms are not included for now (e.g. XXHASH3, SHA-512) ▪ Amazon S3 now supports five additional checksum algorithms
5. fs.s3a.performance.flags create • No checks are made for an object being written above a path containing other objects (i.e. a “directory”) • No checks are made for a parent path containing an object which is not a directory marker (i.e. a “file”) • No overwrite checks are made when creating a file, S3 conditional creation will be used when the file write completes. mkdir • No check whether the parent is directory or file
6. analytics-accelerator-s3 https://github.com/awslabs/analytics-accelerator-s3 Provides custom InputStream, optimized for S3 and Parquet files • Vectorized reads • Pre-fetch small files • Parquet footer caching • etc. Can be used with S3A and Iceberg S3FileIO • Enabled by default in S3A since Hadoop 3.4.3