Share
Embed
AWS re:Invent 2023 でのアップデートを話してと言われた / Fusic Tech Live Vol 17 Makihara
Fusic Events
ハイブリッドでFusic Tech Live Vol.17 AWS re:Invent 振り返り会
https://fusic.connpass.com/event/299160/
Ryunosuke Makihara
December 11, 2023
Transcript
37 slides
-
ࣗݾհ ᴳݪ ཽ೭ี Makihara Ryunosuke - From - ࠤլݝ -
Work at - Fusic@Ԭ(2020~) - ܯඋձࣾͷITࢠձࣾ@౦ژ(2018~2019) - Skill - લ৬ͰLinuxαʔόԾج൫ͷӡ༻ཧ - FusicͰPHPͰͷγεςϜ։ൃ / AWSͷઃܭߏங - Like - ອըɺকعɺөը - SNS - @ryu022304 2 -
ͲΜͳηογϣϯʹࢀՃ͔ͨ͠ 4 • جຊશͯϫʔΫγϣοϓ - ηογϣϯޙ৴Ͱ؍ΕΔͨΊ - Keynoteݱ or ৴ͰϦΞϧλΠϜͰݟͨ
- ۭ͍ͨ࣌ؒΛηογϣϯͰຒΊͨΓͨ͠ • DevOpsܥ + ͦͷଞ - ӡ༻ίϯςφɺCDKɺӴσʔλΛ ѻ͏ͷʹࢀՃ͍ͯͨ͠ - CDK, CloudWatch, EKS, DevOps Guru, Guard Duty -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 7 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 8 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) 1ඵ͋ͨΓඦສ݅ͷॻ͖ࠐΈτϥϯβΫγϣϯʹ εέʔϧՄೳ ϖλόΠτ୯ҐͷσʔλཧՄೳʹ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 9 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ਨɾਫฏํͷεέʔϦϯά γϟʔυϨϓϦΧ͕ઃఆෆཁͰɺεϖοΫͷ ্ݶͷΈࢦఆ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 10 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ߴසΞΫηεɾߴϨΠςϯγ༻ͷετϨʔδΫϥε Standardͱൺͯ10ഒߴɺϦΫΤετඅ༻Λ60%ݮ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 11 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) Graviton 3ΑΓ30%ߴ͍ॲཧೳྗɺ50%ଟ͍ίΞɺ 75%ߴͳϝϞϦ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 12 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ػցֶश༻ΞΫηϥϨʔλ Trainiumͱൺֱͯ͠4ഒͷ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 13 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ͜ͷޙͷൃදͰৄࡉ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 14 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ੜܥAIʹΑΔνϟοτϘοτ AWSϚωίϯΛ։͘ͱӈʹϓϨϏϡʔ൛͕͋Δ ݱঢ়ӳޠͷΈରԠ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 15 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ֤σʔλιʔεʹରͯ͠ETLॲཧແ͠Ͱ Redshiftͷ౷߹ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 16 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) DynamoDB͔ΒETLॲཧແ͠Ͱ OpenSearchͷ౷߹ PIPTͱStreamΛ༗ޮʹ͢Δඞཁ͋Γ -
ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 17 • Aurora Limitless Database • ElastiCache Serverless •
S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) -
Amazon S3 Express One Zone 19 • සൟʹΞΫηε͞ΕΔσʔλߴϨΠςϯγʔରԠ༻ͷ ୯ҰAZͷS3ετϨʔδΫϥε
• S3 Standardͱൺֱͯ͠ΞΫηε͕10ഒ্ɺ ϦΫΤετίετ͕50%ݮ • SageMakerɺAthenaɺEMRɺGlue Data Catalogͷ αʔϏεͱͷซ༻ʹΑΓMLϫʔΫϩʔυͷߴԽ͕Մೳ -
ݕূҙϙΠϯτ 22 • IAM Policy͕S3FullAccess͚ͩͰΓͳ͍ - ʮs3expressʯͱ͍͏αʔϏε໊લۭ͕ؒ͋Δ • S3 Express༻ͷVPCΤϯυϙΠϯτ͕͋Δ
- ʮcom.amazonaws.{region}.s3expressʯ • AmazonLinux 2023σϑΥϧτͷAWS CLI࠷৽Խ͕ඞཁ - σϑΥϧτͷόʔδϣϯ2.9.19(2023/12/10ݱࡏ) - 2ܥ2.14.2Ҏ͔߱ΒExpress͕αϙʔτ͞Ε͍ͯΔ -
XSJUF .# XSJUF (# SFBE .# SFBE
(# TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT ฏۉ -
XSJUF .# XSJUF (# SFBE .# SFBE
(# TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT ฏۉ • όέοτʹର͢ΔϑΝΠϧͷupload/downloadʹؔͯ͠ɺ %ʙ30ˋఔͷൣғͰશൠతʹExpressͷੑೳ͕ྑ͍݁Ռ • Athena20%ఔExpressͷํ͕ੑೳ͕ྑ͔ͬͨ • খ͍͞αΠζ(128MBҎԼ)ͷϑΝΠϧ͕ଟ͗͢Δͱ ಡΈऔΓϨΠςϯγ͕େ͖͘ͳΔͷͰ࠷ద͡Όͳ͔ͬͨՄೳੑ͕͋Δ https://aws.amazon.com/jp/blogs/news/top-10-performance-tuning-tips-for-amazon-athena/ -
Zero ETL 26 • ETL(Extract, Transformation, Load)ॲཧ - σʔλιʔε͔Βͷσʔλͷநग़ɾมɾॻ͖ग़͠ॲཧ -
σʔλͷඋΛߦ͍ɺੳɾMLͷϫʔΫϩʔυʹ Ԡͨ͡ॲཧΛߦ͏ ETL Zero-ETL -
Zero ETL 27 • ETL(Extract, Transformation, Load)ॲཧ - σʔλιʔε͔Βͷσʔλͷநग़ɾมɾॻ͖ग़͠ॲཧ -
σʔλͷඋΛߦ͍ɺੳɾMLͷϫʔΫϩʔυʹ Ԡͨ͡ॲཧΛߦ͏ ETL Zero-ETL T(ม)ʁ 🤔 -
Zero ETL 28 • ETL(Extract, Transformation, Load)ॲཧ - σʔλιʔε͔Βͷσʔλͷநग़ɾมɾॻ͖ग़͠ॲཧ -
σʔλͷඋΛߦ͍ɺੳɾMLͷϫʔΫϩʔυʹ Ԡͨ͡ॲཧΛߦ͏ • θϩETLʹΑΔར - ΞʔΩςΫνϟͷ؆қԽ - ίετޮԽ - ϦΞϧλΠϜ / χΞϦΞϧλΠϜॲཧ -
ݕূҙϙΠϯτ 33 • RedshiftAuroraϓϨϏϡʔ൛&ΦϋΠΦϦʔδϣϯʹ͢Δ ඞཁ͕͋Δ - Aurora MySQL-RedshiftͷZero-ETLطʹGA͕ͩɺ
ࠓճ৽نʹൃද͞ΕͨରαʔϏε·ͩϓϨϏϡʔ • RedshiftAuroraύϥϝʔλάϧʔϓͷमਖ਼͕ඞཁ • ҉߸Խ͕ඞਢ • ϓϥΠϚϦΩʔ͕ແ͍ͱ࿈ܞʹࣦഊ͢Δ - ݕূ͔ͩΒͱదʹςʔϒϧΛ࡞Δͷྑ͘ͳ͍ -
·ͱΊ 36 • re:Invent 2023Ͱൃද͞Εͨ༰ͷৄࡉ͝հ • ৽αʔϏεݕূ - S3 Express
One Zone ➡ StandardͱൺͯύϑΥʔϚϯεͷ্Λ֬ೝ - Zero-ETL ➡ Aurora(PostgreSQL)͔ΒRedshiftͷZero ETLΛ֬ೝ • ॴײ - σʔλ࿈ܞͷ͠͞Λײ͍ͯͨ͡λΠϛϯάͩͬͨͷͰɺ Zero-ETLͷະདྷඇৗʹؾʹͳΔ - ৽αʔϏεͷ༰Λݕূ͢Δͷେม͚ͩΕͲେࣄ - re:Inventָ͔ͬͨ͠ͷͰٕज़ྗͱӳޠྗΛ্͛ͯޙ·ͨߦ͖͍ͨ