Hurdles in Implementing CDC with Aurora during Operation Kazuki Maeda(kzk_maeda) AWS Community Builder

Who am I Kazuki Maeda 𝕏 @kzk_maeda SRE/DRE/Enabling/EM @atama plus AWS Community Builders AWS Startup Community Core Member 2

Main Topic Consider about architecture and organizational design of the product and data platform Dive Deep and provide my own guidelines, especially in cases where responsibilities and dependencies can be complex. 3

Agenda 1. Product and Data Organization 2. Data Platform Architecture and Technology 3. Product System Operation and Data Platform w/ CDC 4. Case Pattern for Architecture & Organization 5. Conclusion 4

Product and Data Organization This is sample text. 5

Data Platform & Organization User Facing Product Core Value 6 Internal Use Feedback & Analysis Product Platform Data Platform Transfer

Roles of Product & Data Organization 7 Including … Product Manager, Developer, Operation SRE Product Platform Data Platform Transfer Including … Data Manager, Data Engineer, Analyst DRE

Need for Responsibility Separation 8 Focus on … Delivering Values Maintaining Stability Product Platform Data Platform Transfer Focus on … Gathering Data Managing Data Quality

Product Team Organization Pattern Example 9 1. 2. 3. Product Platform Data Platform Transfer SRE / Infrastructure Team SRE / Infra Team Data Team Product Team Embedded SRE Team Data Team

Product Team Organization Pattern Example 10 1. 2. 3. Product Platform Data Platform Transfer SRE / Infrastructure Team SRE / Infra Team Data Team Product Team Product Team Data Team Direction of Dependence? Separation of Responsibility?

Data Platform Architecture and Technology This is sample text. 11

Data Platform Architecture Batch Transfer Architecture 12 Product Platform Data Platform Batch Transfer ● Transfer data in periodically ● For example, data are all sent and full replaced in data platform ● Or, data are sent in chunk periodically

Data Transfer Technology Batch Transfer Technology 2 way technologies for Batch Transfer 13 Snapshot Transfer ● Transfer full data via Database snapshot Query Transfer ● Capture changes using SQL via JDBC connection ● Use ETL Service or Broker Service

Data Transfer Technology Batch Transfer Technology 14 Amazon Athena Amazon S3 Amazon Aurora Reader AWS Glue Amazon Redshift Amazon MSK Snapshot Snapshot Query chunk data JDBC Connector

Dependence / Responsibility Batch Transfer Technology 15 Amazon Athena Amazon S3 Amazon Aurora Reader AWS Glue Amazon Redshift Amazon MSK Data Platform Product Platform Product Platform Data Platform Weak Dependence Little Dependence Snapshot Query chunk data

Data Platform Architecture Streaming Transfer Architecture 16 Product Platform Data Platform Streaming Transfer ● Transfer data in streaming ● Data in data platform are updated in near real-time ● Generally, Change data capture (CDC) is used

Data Transfer Technology Change Data Capture 17 Amazon Aurora Writer Amazon MSK AWS DMS Amazon Athena Amazon Redshift Debezium Connector Zero-ETL Replication Logs Amazon S3

Data Transfer Technology Change Data Capture 18 Amazon Aurora Writer Amazon MSK AWS DMS Amazon Athena Amazon Redshift Debezium Connector Zero-ETL Replication Logs Product Platform Amazon S3 Data Platform Strong Dependence

Dependence / Responsibility Why is it so complex? ● Most Database Engine in recent version, it is necessary to gather replication logs from writer instance ● In addition, database user or settings for replication logs should be configured ● So, boundary between both platform is ambiguous 19

Data Platform Architecture Lambda Architecture 20 Product Platform Data Platform Streaming Transfer ● Transfer data using both of batch and streaming ● Benefit from both the certainty data of batch transfers and the near real-time data of streaming transfers ● Use in carefully because of its complexity Batch Transfer

Product System Operation and Data Platform w/ CDC This is sample text. 21

Product System Operation Product operation cases or issues affecting Data Platform ● Manual Table Maintenance ○ In general, operating with manual failover or service maintenance ● Suddenly Failover ○ Hard to predict ● Database Upgrade Operation ○ Recently, operating with Aurora Blue/Green Deployment is recommended to avoid service down. 22

Product System Operation Product operation cases or issues affecting Data Platform ● Manual Table Maintenance ○ In general, operating with manual failover or service maintenance ● Suddenly Failover ○ Hard to predict ● Database Upgrade Operation ○ Recently, operating with Aurora Blue/Green Deployment is recommended to avoid service down. 23 Dive Deep into this pattern

Amazon Aurora Cluster Case Architecture Change Data Capture using DMS to Redshift serverless 24 Amazon Redshift Serverless binlog Writer instance AWS DMS anecdote: Oh no, using DMS Serverless transferring to Redshift Serverless is not supported!! v5.7 Operation Case: We’d like to upgrade Database Engine version to MySQL 8.0 using Aurora Blue / Green Deployment

Operation affected to Data Platform Database Upgrade Operation using Blue/Green Deploy 25 Amazon Aurora Cluster (blue) Amazon Redshift Serverless binlog Writer instance AWS DMS Amazon Aurora Cluster (green) Writer instance binlog v5.7 v8.0 binlog 2 binlogs with different offset values

Operation affected to Data Platform Database Upgrade Operation using Blue/Green Deploy 26 Amazon Aurora Cluster (blue) Amazon Redshift Serverless binlog Writer instance AWS DMS Amazon Aurora Cluster (green) Writer instance binlog v5.7 v8.0 binlog Switch Cannot follow binlog with different offset

Operation affected to Data Platform Database Upgrade Operation using Blue/Green Deploy 27 Amazon Aurora Cluster (blue) Amazon Redshift Serverless binlog Writer instance AWS DMS Amazon Aurora Cluster (green) Writer instance binlog v5.7 v8.0 binlog Switch Cannot follow binlog with different offset

Considering Point for above Problem Team Collaboration ● If product team and data team are different, it is necessary to address those problems during operation ● Direct Communication, maintain runbook, and so on Direction of Dependence and Separation of Responsibility ● If team collaboration is difficult, it is important to design well direction of dependence and separation of responsibility 28

Case Pattern for Architecture & Organization This is sample text. 29

Case pattern Small amount of data and small team size ● Choose simple data platform and technology ● Operate both platform by the same team 30 Amazon Aurora Cluster Writer instance Reader instance Amazon Athena Amazon S3 Cluster Snapshot (parquet file) Product and Data Platform Team

Case pattern Medium amount of data and medium team size ● If data freshness is important, create CDC platform using simple managed, serverless services ● Collaborate with product and data team 31 Amazon Redshift Serverless replication log AWS DMS Amazon Aurora Cluster Writer instance Reader instance Product Platform Team Data Platform Team Dependence

Case pattern Large amount of data, several products, large team size ● Design dependency direction that data platform refers to the product platform 32 Amazon Redshift Serverless Amazon Aurora Cluster Writer instance Reader instance Amazon MSK JDBC Connector Product Platform Team Data Platform Team Dependence SQL

Conclusion This is sample text. 33

Conclusion Proper solution depend on the several factors; ● Characteristics of product ● Size and capability of team ● Technology stack that can be adopt It is important to; ● design the team and the platform together ● be aware of the separation of responsibilities and the direction of dependencies. 34

Thank you! LinkedIn 35 X