Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Snowflake Horizon Catalog と Apache Iceberg で作る ...

Avatar for キタガワ キタガワ
September 29, 2026

Snowflake Horizon Catalog と Apache Iceberg で作る オープンなデータ基盤

DevelopersIO 2026 Osaka #devio2026での登壇資料

Avatar for キタガワ

キタガワ

September 29, 2026

More Decks by キタガワ

Other Decks in Technology

Transcript

  1. Iceberg Iceberg Iceberg Catalog metadata file metadata file s0 s1

    manifest list manifest file manifest file data files Parquet https://iceberg.apache.org/spec/#overview 09
  2. metadata file metadata file db1.table1 00003-<uuid>.metadata.json db1.table2 00012-<uuid>.metadata.json Amazon S3

    db1/table1/metadata/ 00001-<uuid>.metadata.json 00002-<uuid>.metadata.json 00003-<uuid>.metadata.json db1.table3 00007-<uuid>.metadata.json metadata file 00003 00004-<uuid>.metadata.json metadata file 00004 00004 https://iceberg.apache.org/spec/#metastore-tables 10
  3. Iceberg REST Catalog API HTTP Spark Trino API DuckDB PyIceberg

    Snowflake Iceberg REST Catalog API HTTP Horizon Catalog Apache Polaris AWS Glue 11
  4. Snowflake Horizon Catalog Snowflake 1 Snowflake SQL API REST API

    Iceberg REST Catalog API MCP Apache Polaris https://www.snowflake.com/en/product/features/horizon/ 14
  5. Snowflake Horizon Catalog Snowflake Polaris REST Spark Snowflake Horizon Catalog

    Trino Snowflake SQL DuckDB PyIceberg GRANT Iceberg REST /polaris/api/catalog vended credentials S3 Iceberg S3 15
  6. Snowflake Iceberg Snowflake Snowflake Managed Horizon Catalog Snowflake storage AWS

    Glue S3 Snowflake S3 Catalog Snowflake Horizon REST Snowflake catalog-linked database Snowflake cataloglinked database S3 17
  7. Snowflake DuckDB Athena Spark Snowflake Athena DuckDB Snowflake Horizon Catalog

    AWS Amazon Athena PC DuckDB PAT PySpark INSERT SELECT Snowflake Snowsight vended credentials INSERT SELECT vended credentials Amazon S3 Snowflake Iceberg Athena PySpark engine version 3 DuckDB 1.5.4 https://dev.classmethod.jp/articles/snowflake-managed-iceberg-tables-write-from-athena/ https://dev.classmethod.jp/articles/snowflake-tables-iceberg-internal-storage-query-try/ 18
  8. Athena Horizon Catalog REST type = rest Snowflake URL PAT

    warehouse uri = https://<account>.snowflakecomputing.com/polaris/api/catalog credential = <PAT> scope = session:role:<ROLE> warehouse = <DATABASE> S3 Athena header.X-Iceberg-Access-Delegation = vended-credentials Spark SELECT spark.sql.iceberg.vectorization.enabled = false 20
  9. DuckDB PAT S3 Horizon Catalog /v1/oauth/tokens vended credentials TOKEN INSTALL

    iceberg; LOAD iceberg; CREATE SECRET horizon (TYPE iceberg, TOKEN '<access_token>'); ATTACH '<DATABASE>' AS horizon ( TYPE iceberg, SECRET horizon, ENDPOINT 'https://<account>.snowflakecomputing.com/polaris/api/catalog' ); SELECT * FROM horizon.<SCHEMA>.<TABLE>; scope session:role:<ROLE> PAT CLIENT_SECRET DuckDB Snowflake 21
  10. Snowflake 2026 9 Snowflake scope session:role:<ROLE> GA vended credentials Snowflake

    Spark S3 Scan Plan API Iceberg 1.11 https://docs.snowflake.com/en/user-guide/tables-iceberg-enforce-access-policies-scan-plan-api 22
  11. IAM { "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": ["arn:aws:s3:::example-iceberg-bucket",

    "arn:aws:s3:::example-iceberg-bucket/*"], "Condition": {"ArnNotLike": {"aws:PrincipalArn": [ "arn:aws:iam::111122223333:role/snowflake-volume", "arn:aws:iam::111122223333:role/your-admin" ]}} } 24
  12. Snowflake AI Snowflake Snowflake Snowflake Cortex Agents Cortex Analyst Cortex

    Analyst Spark Trino DuckDB SQL Horizon Catalog Snowflake Iceberg REST Catalog API Iceberg S3 Snowflake 26
  13. Snowflake S3 Open Data Sharing Iceberg REST Catalog API Snowflake

    Iceberg S3 external consumer PAT Snowflake S3 Iceberg REST Catalog API PAT Iceberg REST 27
  14. Iceberg GA 2026 9 26 GA GA Snowflake Iceberg 2024-06

    catalog-linked database 2025-10 Horizon 2026-02 Horizon 2026-05 Iceberg v3 2026-05 Snowflake storage 2026-06 Open Data Sharing Horizon Scan Plan API 2026-08 2026-08 2026-09 28