AWS re:Invent 2023 でのアップデートを話してと言われた / Fusic Tech Live Vol 17 Makihara

Fusic Tech Live Vol.17～ハイブリッドでFusic Tech Live Vol.17 AWS re:Invent 振り返り会〜

Transcript

1. re:Invent 2023ͰͷΞοϓσʔτΛ ࿩ͯ͠ͱݴΘΕͨ 2023.12.11 ᴳݪ ཽ೭ี 1
2. ࣗݾ঺հ ᴳݪ ཽ೭ี Makihara Ryunosuke - From - ࠤլݝ - Work at - Fusic@෱Ԭ(2020~) - ๭ܯඋձࣾͷITࢠձࣾ@౦ژ(2018~2019) - Skill - લ৬Ͱ͸Linuxαʔό΍Ծ૝ج൫ͷӡ༻؅ཧ౳ - FusicͰ͸PHPͰͷγεςϜ։ൃ / AWSͷઃܭߏங - Like - ອըɺকعɺөը౳ - SNS - @ryu022304 2
3. 01 re:Invent 2023
4. ͲΜͳηογϣϯʹࢀՃ͔ͨ͠ 4 • جຊશͯϫʔΫγϣοϓ - ηογϣϯ͸ޙ೔഑৴Ͱ؍ΕΔͨΊ - Keynote͸ݱ஍ or ഑৴ͰϦΞϧλΠϜͰݟͨ - ۭ͍ͨ࣌ؒΛηογϣϯͰຒΊͨΓ͸ͨ͠ • DevOpsܥ + ͦͷଞ - ӡ༻΍ίϯςφɺCDKɺӴ੕σʔλ౳Λ 
 ѻ͏΋ͷʹࢀՃ͍ͯͨ͠ - CDK, CloudWatch, EKS, DevOps Guru, Guard Duty
5. re:Invent 2023ͷࢥ͍ग़(Ұ෦) 5
6. 02 ओͳൃද͞Εͨ৽αʔϏε
7. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 7 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB)
8. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 8 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) 1ඵ͋ͨΓ਺ඦສ݅ͷॻ͖ࠐΈτϥϯβΫγϣϯʹ εέʔϧՄೳ ϖλόΠτ୯Ґͷσʔλ؅ཧ΋Մೳʹ
9. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 9 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ਨ௚ɾਫฏํ޲΁ͷεέʔϦϯά γϟʔυ਺΍ϨϓϦΧ਺͕ઃఆෆཁͰɺεϖοΫͷ ্ݶͷΈࢦఆ
10. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 10 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ߴස౓ΞΫηεɾߴϨΠςϯγ༻ͷετϨʔδΫϥε Standardͱൺ΂ͯ10ഒߴ଎ɺϦΫΤετඅ༻Λ60%࡟ݮ
11. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 11 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) Graviton 3ΑΓ30%ߴ͍ॲཧೳྗɺ50%ଟ͍ίΞ਺ɺ 75%ߴ଎ͳϝϞϦ
12. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 12 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ػցֶश༻ΞΫηϥϨʔλ Trainiumͱൺֱͯ͠4ഒͷ଎౓
13. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 13 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ͜ͷޙͷൃදͰৄࡉ
14. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 14 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ੜ੒ܥAIʹΑΔνϟοτϘοτ AWSϚωίϯΛ։͘ͱӈʹϓϨϏϡʔ൛͕͋Δ ݱঢ়ӳޠͷΈରԠ
15. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 15 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) ֤σʔλιʔεʹରͯ͠ETLॲཧແ͠Ͱ ௚઀Redshift΁ͷ౷߹
16. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 16 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB) DynamoDB͔ΒETLॲཧແ͠Ͱ௚઀ OpenSearch΁ͷ౷߹ PIPTͱStreamΛ༗ޮʹ͢Δඞཁ͋Γ
17. ओͳൃද͞Εͨ৽αʔϏεɾߋ৽ 17 • Aurora Limitless Database • ElastiCache Serverless • S3 Express One Zone • Graviton 4 • Trainium 2 • Agents / Knowledge base for Bedrock • Amazon Q • zero-ETL integration with Redshift 
 (Aurora(PostgreSQL), DynamoDB, RDS(MySQL)) • zero-ETL integration with OpenSearch(DynamoDB)
18. 03 (ͪΐͬͱ)ৄࡉ
19. Amazon S3 Express One Zone 19 • සൟʹΞΫηε͞ΕΔσʔλ΍ߴϨΠςϯγʔରԠ༻ͷ 
 ୯ҰAZͷS3ετϨʔδΫϥε • S3 Standardͱൺֱͯ͠ΞΫηε଎౓͕10ഒ޲্ɺ 
 ϦΫΤετίετ͕50%࡟ݮ • SageMakerɺAthenaɺEMRɺGlue Data Catalog౳ͷ 
 αʔϏεͱͷซ༻ʹΑΓMLϫʔΫϩʔυͷߴ଎Խ͕Մೳ
20. S3 Express One ZoneΛݕূͯ͠ΈΔߏ੒ 20 ԼهΛStandardͱExpress 1zoneͦΕͧΕʹ࣮ߦ ɾEC2͔Β1MB*1000ϑΝΠϧͷread/write ɾEC2͔Β1GB*1ϑΝΠϧͷread/write ɾAthenaͰͷcount(10KB*10ສϑΝΠϧ)
21. Ϛωʔδϝϯτίϯιʔϧ͔Β࡞੒ 21
22. ݕূ஫ҙϙΠϯτ 22 • IAM Policy͕S3FullAccess͚ͩͰ͸଍Γͳ͍ - ʮs3expressʯͱ͍͏αʔϏε໊લۭ͕ؒ͋Δ • S3 Express༻ͷVPCΤϯυϙΠϯτ͕͋Δ - ʮcom.amazonaws.{region}.s3expressʯ • AmazonLinux 2023σϑΥϧτͷAWS CLI͸࠷৽Խ͕ඞཁ - σϑΥϧτͷόʔδϣϯ͸2.9.19(2023/12/10ݱࡏ) - 2ܥ͸2.14.2Ҏ͔߱ΒExpress͕αϙʔτ͞Ε͍ͯΔ
23. Standard Express
24. XSJUF .#  XSJUF (#  SFBE .#  SFBE (#  TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT                                              ฏۉ        
25. XSJUF .#  XSJUF (#  SFBE .#  SFBE (#  TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT TUBOEBSE FYQSFTT                                              ฏۉ         • όέοτʹର͢ΔϑΝΠϧͷupload/downloadʹؔͯ͠ɺ 
 ਺%ʙ30ˋఔͷൣғͰશൠతʹExpressͷੑೳ͕ྑ͍݁Ռ • Athena΋20%ఔExpressͷํ͕ੑೳ͕ྑ͔ͬͨ • খ͍͞αΠζ(128MBҎԼ)ͷϑΝΠϧ਺͕ଟ͗͢Δͱ 
 ಡΈऔΓϨΠςϯγ͕େ͖͘ͳΔͷͰ࠷ద͡Όͳ͔ͬͨՄೳੑ͕͋Δ https://aws.amazon.com/jp/blogs/news/top-10-performance-tuning-tips-for-amazon-athena/
26. Zero ETL 26 • ETL(Extract, Transformation, Load)ॲཧ - σʔλιʔε͔Βͷσʔλͷநग़ɾม׵ɾॻ͖ग़͠ॲཧ - σʔλͷ੔උΛߦ͍ɺ෼ੳɾML౳ͷϫʔΫϩʔυʹ 
 Ԡͨ͡ॲཧΛߦ͏ ETL Zero-ETL
27. Zero ETL 27 • ETL(Extract, Transformation, Load)ॲཧ - σʔλιʔε͔Βͷσʔλͷநग़ɾม׵ɾॻ͖ग़͠ॲཧ - σʔλͷ੔උΛߦ͍ɺ෼ੳɾML౳ͷϫʔΫϩʔυʹ 
 Ԡͨ͡ॲཧΛߦ͏ ETL Zero-ETL T(ม׵)͸ʁ 🤔
28. Zero ETL 28 • ETL(Extract, Transformation, Load)ॲཧ - σʔλιʔε͔Βͷσʔλͷநग़ɾม׵ɾॻ͖ग़͠ॲཧ - σʔλͷ੔උΛߦ͍ɺ෼ੳɾML౳ͷϫʔΫϩʔυʹ 
 Ԡͨ͡ॲཧΛߦ͏ • θϩETLʹΑΔར఺ - ΞʔΩςΫνϟͷ؆қԽ - ίετޮ཰Խ - ϦΞϧλΠϜ / χΞϦΞϧλΠϜॲཧ
29. Zero-ETL integration with Redshift(Aurora PostgreSQL)Λݕূͯ͠ΈΔߏ੒ 29 ɾAuroraͱRedshiftͷZero-ETL integrationΛઃఆ ɾEC2͔ΒAuroraʹσʔλ౤ೖ ɾRedshiftଆͰσʔλ֬ೝ
30. Zero ETL ݕূ 30
31. Zero ETL ݕূ 31
32. Zero ETL ݕূ 32
33. ݕূ஫ҙϙΠϯτ 33 • Redshift΋Aurora΋ϓϨϏϡʔ൛&ΦϋΠΦϦʔδϣϯʹ͢Δ 
 ඞཁ͕͋Δ - Aurora MySQL-RedshiftͷZero-ETL͸طʹGA͕ͩɺ 
 ࠓճ৽نʹൃද͞Εͨର৅αʔϏε͸·ͩϓϨϏϡʔ • Redshift΋Aurora΋ύϥϝʔλάϧʔϓͷ஋मਖ਼͕ඞཁ • ҉߸Խ͕ඞਢ • ϓϥΠϚϦΩʔ͕ແ͍ͱ࿈ܞʹࣦഊ͢Δ - ݕূ͔ͩΒͱద౰ʹςʔϒϧΛ࡞Δͷ͸ྑ͘ͳ͍
34. ݁Ռ 34 Aurora(PostgreSQL) Redshift • େମ10ඵҎ಺ʹ͸࿈ܞ 
 ͞Εͨ
35. 04 ·ͱΊ
36. ·ͱΊ 36 • re:Invent 2023Ͱൃද͞Εͨ಺༰ͷৄࡉ͝঺հ • ৽αʔϏεݕূ - S3 Express One Zone ➡ Standardͱൺ΂ͯύϑΥʔϚϯεͷ޲্Λ֬ೝ - Zero-ETL ➡ Aurora(PostgreSQL)͔ΒRedshift΁ͷZero ETLΛ֬ೝ • ॴײ - σʔλ࿈ܞͷ೉͠͞Λײ͍ͯͨ͡λΠϛϯάͩͬͨͷͰɺ 
 Zero-ETLͷະདྷ͸ඇৗʹؾʹͳΔ - ৽αʔϏεͷ಺༰Λݕূ͢Δͷ͸େม͚ͩΕͲ΋େࣄ - re:Invent͸ָ͔ͬͨ͠ͷͰٕज़ྗͱӳޠྗΛ্͛ͯ਺೥ޙ·ͨߦ͖͍ͨ
37. ͝ਗ਼ௌ͍͖ͨͩ͋Γ͕ͱ͏͍͟͝·ͨ͠ Thank You We are Hiring ! https://recruit.fusic.co.jp/