Serverless Data Processing with AWS Lambda and Apache Kafka

Slide 1

Slide 1 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 1 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. with AWS Lambda and Apache Kafka Kensuke Shimokawa Amine Tei J A W S - U G D I V E R S I T Y , E Q U I T Y , A N D I N C L U S I O N Serverless data processing

Slide 2

Slide 2 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 2 スタッフの紹介 Amine Tei Snr. Serverless Specialist Kensuke Shimokawa Manufacturing Solutions Architect

Slide 3

Slide 3 text

Slide 4

Slide 4 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 11 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. ワークショップについて

Slide 5

Slide 5 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 12 ワークショップのゴール • Kafka を利用したストリーミングデータに基づきサーバーレスのリアルタイムサービスによる位置追跡を構築する • Amazon Managed Streaming for Apache Kafka (MSK) • AWS Lambda • AWS AppSync Uberの配車サービスみたいなものをサクッとつくってみる（車両の位置情報収集部分のみ）

Slide 6

Slide 6 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 13 作るアーキテクチャ Amazon Managed Streaming for Apache Kafka (Amazon MSK) AWS Lambda AWS AppSync Dashboard Unicorns AWS Cloud

Slide 7

Slide 7 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 14 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. ワークショップに登場するサービス

Slide 8

Slide 8 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 15 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. AWS AppSync

Slide 9

Slide 9 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 16 • OSS QUERY ⾔語型指定された Schema type Query { getTodos: [Todo] } type Todo { id: ID! name: String! description: String priority: Int duedate: String } クライアントがレスポンス形式を指定 query { getTodos { id name priority } } type Query { getTodos: [Todo] } type Todo { id: ID! name: String! description: String priority: Int duedate: String } ３つの処理形態 Query Mutation Subscription https://graphql.org/ Over/Under-fetching の是正 A query language for your API GraphQL とは !?

Slide 10

Slide 10 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 17 { "id": "1", "name": "Study", “priority": "1" }, { "id": “2", "name": “Go to gym", “priority": “5" },… type Query { getTodos: [Todo] } type Todo { id: ID! name: String description: String priority: Int duedate: String } query { getTodos { id name priority } } Schema とモデルデータ (Type) クライアントが必要なものだけをリクエストリクエストしたデータのみが返されるクライアントがレスポンス形式を指定

Slide 11

Slide 11 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 18 Schema Query Mutation Subscription type Subscription { onComments (eventId: String!): Comment @aws_subscribe(mutations:["commentOnEvent") } type Mutation { commentOnEvent ( eventId: ID!, content: String!, createdAt: String! ): Comment } ３つの処理形態

Slide 12

Slide 12 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 19 S I M P L E S T W A Y T O B U I L D S C A L A B L E A P I S T H A T C O N N E C T A P P L I C A T I O N S T O D A T A 100% Serverless Code 減少に伴い複雑さも減少安全性と⾼パフォーマンス TLS 暗号化、安全な API、Custom domain、Cache 管理、認証/認可、 WAF、CloudTrail 等による⾼い Resiliency をBuilt-in Full-managed の GraphQL API と Pub/Sub API のセットアップ、管理、メンテナンス、⾼可⽤性を実現するためのインフラプロビジョニング他 AWS Services との統合により AppSync ⾃体は低 Layer での Utility code を処理し、マニュアルエラーを削減 AWS AppSync とは

Slide 13

Slide 13 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 20 Data Source 他 Services etc ... AppSync を BFF としたアーキテクチャの場合 Resolver、Data Source、 Back-end services の組合せは開発者の⾃由開発における選択肢が多⽤で拡張性が⾮常に⾼い反⾯、Workload に沿ったアーキテクチャデザインが重要となる Resolver - Unit Resolver - Pipeline Resolver AWS AppSync アーキテクチャ AWS AppSync

Slide 14

Slide 14 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 21 Pub/Sub APIs AWS AppSync Use cases Front-end developer は Serverless な WebSocket 接続を介してSubscribe している API Client に Real-time のデータ更新を発⾏

Slide 15

Slide 15 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 22 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Amazon Managed Streaming for Apache Kafka / MSK

Slide 16

Slide 16 text

Slide 17

Slide 17 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 24 クラスター容量の初期適正見積りなく、 Apache Kafka クラスターを簡単に実行容量の拡大縮小、パーティション再割り当てを気にすることなく、I/Oを即座に拡張ストリーミング量と保持するデータ量に応じた課金 Amazon MSK Serverless

Slide 18

Slide 18 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 25 Kafka パーティション / Kinesis シャード old new 1 4 5 6 7 8 3 2 9 Partition 0/ shard 1 1 4 5 6 7 8 3 2 10 9 Partition 1/ shard 2 1 4 5 6 7 8 3 2 Partition 2/ shard 3 新規データ書き込み Kafka トピック / Kinesis ストリーム

Slide 19

Slide 19 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 26 パーティションキー Producer Producer Producer PK1 PK1 PK1 PK1 Pk4 PK4 PK4 PK4 PK2 PK2 PK3 PK3 MD5 hash function PK1 PK1 PK1 PK1 Partition 0 / shard 1 PK2 PK3 PK3 PK2 Partition 1 / shard 2 PK4 PK4 PK4 PK4 Partition 2 / shard 3 Kafka partition/ Kinesis shard

Slide 20

Slide 20 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 27 Kafka オフセット / Kinesis シーケンス番号コンシューマがストリーム内の処理済み位置を追跡するために保持 1 4 5 6 7 8 3 2 10 9 Partition New data Old Now Time put Producer Consumer-1 offset 4/ sequence number 4 Consumer-2 offset 7/ sequence number 7

Slide 21

Slide 21 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 28 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. ストリームデータ処理

Slide 22

Slide 22 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 29 EventBridge Pipes: AWSのサービスをパイプで繋ぐ Amazon SQS Amazon Kinesis Data Streams Amazon DynamoDB Amazon MSK Amazon MQ Source AWS service Amazon EventBridge Pipes Filter, enrich, and transform your events Filter Choose which events to keep Enrich and transform Restructure, modify, or pull additional data into your event Target AWS services Amazon EventBridge Amazon Kinesis Data Streams AWS Step Functions Amazon API Gateway Amazon SQS Amazon API Gateway AWS Step Functions AWS Lambda Amazon EventBridge API Destinations ※ Amazon EventBridge バス側にKafka Connect⽤のオープンソースコネクタが利⽤可能

Slide 23

Slide 23 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 30 Kinesis ユースケース: データストリームの分割コードを書かずに多重化されたデータストリームを複数のドメインストリームに分割 Amazon EventBridge Pipes Amazon Kinesis Data Streams Filter Amazon EventBridge Pipes Filter Amazon Kinesis Data Streams Amazon Kinesis Data Streams

Slide 24

Slide 24 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 31 Kafka ユースケース: API destination コードを書かずに、セルフマネージド Kafka クラスターを AWS サービスや HTTP API に接続 VPC Self-managed Kafka Amazon EventBridge Pipes Filter Amazon EventBridge Pipes Filter Amazon SageMaker API Destination Enrichment

Slide 25

Slide 25 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 32 Kafka Lambda consumer のオプション AWS Lambda service Event source mapping AWS Lambda function Kafka sink connector AWS Lambda function

Slide 26

Slide 26 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 34 Lambda event source mapping / ESM イベントソースから読み取り、Lambda 関数を呼び出す Lambda リソース Event Source Amazon Kinesis DynamoDB Streams Amazon SQS Amazon MSK Amazon MQ Apache Kafka Filter Event pattern content filtering Batch AWS Lambda service pollers Target AWS Lambda function Synchronous invoke

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 36 At-least once 配信イベントは、ターゲットに複数回配信されることも。処理されたイベントの状態を追跡することによって重複イベントを検出するロジックを含める (冪等性) 配信セマンティクス Exactly-once 配信* AWSサービスに重複排除に使用する識別子を渡すことで重複排除 * Lambdaでリトライ設定している場合は、At-least once • Amazon EventBridge • Amazon SNS 標準トピック • Amazon SQS 標準キュー • Amazon Kinesis • Amazon SNS FIFO トピック • Amazon SQS FIFO キュー • Kafka Topic

Slide 29

Slide 29 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 37 順序保証なしイベント/メッセージが順序保証なしで配信順序セマンティクス* 順序保証イベントは、パーティション、メッセージグループなどの中で順番に配信（グローバルな順序保証はありません） • Amazon EventBridge • Amazon SNS 標準トピック • Amazon SQS 標準キュー • Amazon Kinesis • Amazon SNS FIFO トピック • Amazon SQS FIFO キュー • Kafka Topic * 順不同のイベント処理ロジックが受⼊可能かは、ビジネス要件次第となります。順不動なイベントを処理するようにアプリケーションを設計できない場合は、代わりにオーケストレーションも検討

Slide 30

Slide 30 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 50 © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Kafka ESM のプロビジョニングモード

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 53 パフォーマンスの違い • BatchSize = 50 • Batching Window = 1 sec • Mean duration = 200ms • Min pollers = 5 Producers • MSK Cluster • 2 brokers • 1 topic • 100 partitions • Record size 1.5KB • Random Partition Key • Initial traffic – 3,000 records / second • Traffic spike - 9,000 records / second

Slide 34

Slide 34 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 54 パフォーマンスの違い • BatchSize = 50 • Batching Window = 1 sec • Mean duration = 200ms • Min pollers = 5 Producers • MSK Cluster • 2 brokers • 1 topic • 100 partitions • Record size 1.5KB • Random Partition Key • Initial traffic – 3,000 records / second • Traffic spike - 9,000 records / second

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 59 オンデマンドとプロビジョニング・ ESM のパフォーマンス ~7 minutes ~15 minutes 9000 records/sec Kafka ESM – on-demand mode

Slide 40

Slide 40 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 60 オンデマンドとプロビジョニング・ ESM のパフォーマンス Kafka ESM – on-demand mode Kafka ESM – Provisioned mode (min=5)

Slide 41

Slide 41 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 61 オンデマンドとプロビジョニング・ ESM のパフォーマンス Kafka ESM – on-demand mode Kafka ESM – Provisioned mode (min=5) ~60,000 at peak ~3,000,000 at peak

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 64 Thank you! © 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.