Using Metrics Reporting in Apache Iceberg

Transcript

1. © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Iceberg MetricsReporter 活⽤のススメ O T F S G T O K Y O M E E T U P # 5 Tomohiro Tanaka Senior Cloud Support Engineer, AWS
2. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tomohiro Tanaka Senior Cloud Support Engineer, AWS Support Amazon Web Services • Responsible for solving most complex troubles and guiding best practices with Iceberg • Contributing to Apache Iceberg OSS project 2
3. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics Reporting in Iceberg Built-in metrics reporters Implement metrics reporters Demo Agenda 3
4. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4
5. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metrics Reporting in Iceberg A feature reporting manifest related metrics to any resource The metrics are published when an Iceberg table is scanned or committed. The metrics can be used to configure triggering some actions. Two APIs: MetricsReporter and MetricsReport By default, the metrics are recorded by LoggingMetricsReporter*. 5 *For Spark, InMemoryMetricsReporter is used, and the metrics are added to Spark event logs, and displayed on Spark UI
6. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. MetricsReporter API The core interface defining how Iceberg table metrics are reported. 6 https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/metrics/MetricsReporter.java
7. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. MetricsReport API Two types of MetricsReport: ScanReport and CommitReport ScanReport: Read metrics such as planning duration, scanned manifests etc. CommitReport: Write metrics such as added data files, added records etc. 7
8. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. ScanReport 8 https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/metrics/ScanMetrics.java
9. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. CommitReport 9 https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/metrics/CommitMetricsResult.java
10. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Built-in metrics reporters LoggingMetricsReporter InMemoryMetricsReporter (by default, for Spark) RESTMetricsReporter (to enable this, set rest-metrics-reporting-enabled to true) 10 Ref: https://iceberg.apache.org/docs/latest/metrics-reporting/#loggingmetricsreporter
11. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Implement custom MetricsReporter Implement MetricsReporter interface Properties are handled via initialize method Send metrics to the destination via reporter method And, safely close it 11
12. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use MetricsReporter 12 Configure your MetricsReporter class to metrics-reporter-impl: For multiple reporters:
13. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo: CloudWatchMetricsReporter Publishes the metrics to Amazon CloudWatch Metrics Some alarms (CloudWatch Alarms) are configured for metrics, and they trigger Iceberg compaction jobs based on its condition. 13 Iceberg table Processing job (Apache Spark) CloudWatchMetrics Reporter Amazon CloudWatch Metrics Compaction job (Apache Spark) Scan or commit Publish If the condition is met Trigger running compaction Compaction
14. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo: CloudWatchMetricsReporter 14 Spark config:
15. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. • For every scan/commit, relevant metrics are reported via MetricsReporter. • There are two types of metrics report such as ScanReport and CommitReport. • They can be used to get insights of Iceberg table status • In advanced use-cases, they can be used to enable adaptive table maintenance. 15 Key takeaways
16. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! 16