5年ぶりのEMR、進化しすぎて別物だった件

Transcript

1. 5೥ͿΓͷEMRɺ ਐԽ͗ͯ͢͠ผ෺ͩͬͨ݅ ͡ΎΜ͡ΎΜʢ@yoshiyama_hanaʣ
2. ͡ΎΜ͡ΎΜ ܦྺ 2018೥ 4݄ ंࡌثϝʔΧʔ ೖࣾ 2024೥ 9݄ ָثϝʔΧʔ ೖࣾ ※ຊൃද͸ݸਓͷݟղʹجͮ͘΋ͷͰ͋Γɺॴଐ૊৫ͱ͸ؔ܎͋Γ·ͤΜɻ ৬ۀ ΤϯδχΞ ʢΫϥ΢υɺόοΫΤϯυɺSREɺIoTɺσʔλɺ෼ੳʣ ޷͖ͳAWSαʔϏεɿAthena, QuickSight @yoshiyama_hana 4/4౳͸ͪ͜Β
   ͓ؾܰʹͲ͏ͧ
3. એ఻
4. Tech×Rock Festival 2025 ΍Γ·͢ʂʢ9݄ʙ11݄ʣ ٕज़ษڧձͷϊϦͰΤϯδχΞಉ࢜ͰόϯυΛ૊ΜͰϥΠϒ͠Α͏ ͥʂͬͯΠϕϯτͰ͢ɻʢࠓ೥͸70ਓҎ্ͷΤϯδχΞ͕ग़ԋ༧ఆʣ
5. Tech×Rock Festival 2025 ΍Γ·͢ʂ conpassϖʔδ͋Γ·͢
6. ؓ࿩ٳ୊
7. ࠓ೔͸ɺ
8. ໿5೥ͿΓʹEMRΛ࢖ͬͯΈ ͯײͨ͡ਐԽΛ ڞ༗͍ͨ͠ʂ
9. ͱ͜ΖͰɺ
10. Έͳ͞Μ͕ॳΊͯ࢖ͬͨ AWSαʔϏε͸ͳΜͰ͔͢ʁ
11. ࢲ͸EMRͰͨ͠
12. EMRʢAmazon Elastic MapReduceʣͱ͸ʁ Amazon EMR (Ҏલ͸ Amazon Elastic MapReduce ͱݺ͹Ε͍ͯ·ͨ͠) ͸ɺ Ͱͷ Apache Hadoop ΍ Apache Spark ͳͲͷϏοά σʔλϑϨʔϜϫʔΫͷ࣮ߦΛ؆ૉԽ AWS ͠ɺେྔͷσʔλΛॲཧ͓Αͼ෼ੳ͢ΔϚωʔδυΫϥελʔϓϥοτϑΥʔϜͰ͢ɻ ͜ΕΒͷϑϨʔϜϫʔΫͱɺؔ࿈͢ΔΦʔϓϯιʔεϓϩδΣΫτΛ࢖༻͢Δ͜ͱͰɺ෼ੳ༻ͷσʔλ΍ϏδωεΠϯςϦδΣϯε ϫʔΫϩʔυΛॲཧͰ͖·͢ɻAmazon EMR Λ࢖༻ͯ͠ɺେྔͷσʔλΛม׵͠ɺAmazon Simple Storage Service (Amazon S3) ΍ Amazon DynamoDB ͳͲͷଞͷ AWS σʔλετΞ΍σʔλϕʔεʹσʔλΛग़͠ೖΕ͢Δ͜ͱ΋Ͱ͖·͢ɻ Ҿ༻ɿhttps://docs.aws.amazon.com/ja_jp/emr/latest/ManagementGuide/emr-what-is-emr.html ΋ͷ͘͢͝؆୯ʹ͍͏ͱ େྔͷσʔλॲཧʹదͨ͠؀ڥΛఏڙͯ͘͠ΕΔαʔϏε ϢʔεέʔεɿσʔλͷલॲཧɺETLॲཧɺσʔλ෼ੳɺػցֶश
13. ͸͡ΊͯͷAWS ʙઌഐʹEMRΛڭΘΖ͏ʙ
14. ͸͡ΊͯͷAWS ʙઌഐʹEMRΛڭΘΖ͏ʙ ʮ·ͣɺVPCͱαϒωοτͷઃఆͯ͠ʯ ʮͰɺIAMϩʔϧ࡞ΔɻEC2༻ͱEMR༻Ͱ෼͚ͯɺඞཁͳϙϦγʔ͚ͭͱ͍ͯʯ ʮ࣍ɺS3όέοτɻσʔλஔ͖৔΋ϩάஔ͖৔΋͍Δ͔Βɺόέοτ2ͭ͸༻ҙ͠ͱ͍ͯɻʯ ʮͰɺηΩϡϦςΟάϧʔϓɻϚελʔͱίΞϊʔυ༻ʹͪΌΜͱϙʔτ։͚ͯɻSSH͍ͨ͠ͳΒ22൪Ͷɻʯ ʮͦͷ͋ͱɺΫϥελʔͷ࡞੒ʹਐΉΜ͚ͩͲɺ͜͜ͰϦϦʔεόʔδϣϯʢemr-6.xͱ͔ʣબͿͷ஍ຯʹॏཁͩ ͔Β஫ҙͶɻʯ ʮΞϓϦέʔγϣϯ͸SparkͱHadoopͰOKɻඞཁͳΒHive΋͚ͭͱ͍ͯɻʯ ʮϒʔτετϥοϓΞΫγϣϯ࢖ͬͯґଘϥΠϒϥϦͱ͔΋ೖΕͱ͘ͱ͍͍Αɻʯ ʮ͋ͱɺεςοϓΛొ࿥͓ͯ͘͠ͱɺΫϥελʔ্ཱͪ͛ͯଈॲཧͰ͖ΔɻόονॲཧͳΒಛʹศརɻʯ ʮͰɺ࠷ޙʹAuto-terminationʢࣗಈऴྃʣ๨Εͣʹɻ͡Όͳ͍ͱ͔͔ۚΔΑʁʯ
15. Θ͔Γ·ͨ͠ʂ😊
16. Θ͔Μͳ͍Α…ɻΘ͔ͬΜͳ͍Αʂʂʂ ઌഐ͕ݴͬͯΔ͜ͱɺͻͱͬͭ΋Θ͔Μͳ͍Αʂʂ ʮ·ͣɺVPCͱαϒωοτͷઃఆͯ͠ʯͬͯԿʂʁͲ͜ʹ͋ΔͷʂʁɹͳΜͰωοτϫʔΫͷ͜ͱΛࢲ͕ߟ͑ͳ͖Ό͍͚ͳ͍ͷʂʁ ʮͰɺIAMϩʔϧ࡞Δʯͬͯʂʁɹϩʔϧͬͯͳʹʂʁԋܶͳͷʂʁɹ഑໾ͳͷʂʁɹͳΜͰࢲʹ2໾΍ΒͤΔͷʂʁ ʮϙϦγʔ͚ͭͱ͍ͯʯͬͯɺͲͷʂʁɹͲΕ͕ਖ਼͍͠ͷʂʁ͋ͷνΣοΫϘοΫε஍ࠈɺා͗͢ΔΑʂʂ ʮS3όέοτ͸2ͭ༻ҙʯͬͯɺͳΜͰ2ͭʂʁҰॹ͡Όμϝͳͷʂʁɹʮϩά༻ʯͬͯ…Ͳ͔͜Βग़Δϩάʂʁ ʮηΩϡϦςΟάϧʔϓͪΌΜͱ։͚ͯʯͬͯɺͪΌΜͱͬͯԿʂʁ 22൪ʂʁ ୭͕ܾΊͨͷʂʁดͨ͡ΒౖΒΕΔͷʂʁɹ։͚ͨΒౖΒΕ Δͷʂʁ ʮϦϦʔεόʔδϣϯબΜͰʯͬͯɺ6.13ͱ6.15ͱ6.17ͷҧ͍ͳΜͯΘ͔ΔΘ͚ͳ͍͡ΌΜʂʂʂʂʂͪΐͬͱͣͭมΘͬͯΔͱ͔ɺ஌Β ͳ͍Αʂʂ SparkͱHadoopͰOKͬͯɺOKͬͯݴΘΕͯ΋Ͳ͕ͬͪԿͯ͠Δͷ͔ɺ1ϛϦ΋Θ͔Μͳ͍Αʂʂ ʮϒʔτετϥοϓΞΫγϣϯͰϥΠϒϥϦೖΕͱ͍ͯʯͬͯɺԿͦΕࣄલ४උΠϕϯτʂʁ ຐ๏͔Կ͔ʂʁ S3ʹΞοϓ͠ͱ͚ͬͯɺS3͕΋͸΍໎ٶͳΜͩΑʂ εςοϓొ࿥ͬͯɺδϣϒʁ εςοϓʁ ΧλΧφ͹͔ͬΓͰΘ͔Μͳ͍Αʂʂ Auto-termination๨Εͣʹͬͯɺ๨ΕͨΒ͓ۚඈͿͬͯͦΕ΋ͬͱૣ͘ݴͬͯΑʂʂ CloudWatch͔S3ʹϩάग़͠ͱ͚ͬͯɺ΋͏Ͳͬͪʹ΋ग़ͯͯݟ͔ͭΒͳ͍ΜͩΑʂ ࢲʹ͸ͥΜͬͥΜΘ͔Μͳ͍ͷʂʂʂ
17. શવΘ͔Γ·ͤΜͰͨ͠
18. 5೥લ͸EMRΛ؆୯ʹಈ͔͚ͩ͢Ͱ΋ϋʔυϧ ͕ߴ͔ͬͨ ✓EMRͷ஌ࣝ ✓EC2ͷ஌ࣝ ✓Spark౳ͷ෼ࢄॲཧϑϨʔϜϫʔΫͷ஌ࣝ
19. ࣌͸ྲྀΕͯ5೥ޙ...
20. EMR ΋ਐԽ͍ͯ͠Δ ✓Amazon EMR StudioͷҰൠఏڙ։࢝ ✓Amazon EMR ServerlessͷҰൠఏڙ։࢝ ✓EMRϊʔτϒοΫͰͷJupyterLabαϙʔτ ✓AWS Step Functionsͱͷ౷߹
21. Ҏલͱൺ΂ͯEMRΛ؆୯ʹಈ͔͚ͩ͢ͳΒϋʔ υϧ͕Լ͕ͬͨ ✓EMRͷ஌ࣝ ✓EC2ͷ஌ࣝ ✓Spark౳ͷ෼ࢄॲཧϑϨʔϜϫʔΫͷ஌ࣝ
22. demo
23. ৄ͘͠͸webͰ
24. ·ͱΊ ❖ AWSͷΞοϓσʔτʹΑͬͯEC2ͷ؅ཧΛؾʹ͢Δඞཁ͕ͳ͘ͳͬͨ ͨΊEMRͰ͓खܰʹσʔλॲཧ͕Ͱ͖ΔΑ͏ʹͳΓ·ͨ͠ ͜ΕΛ͖͔͚ͬʹؾܰʹνϟϨϯδ͢Δ͖͔͚ͬʹͳΔͱخ͍͠Ͱ͢ɻ
25. 5೥ͿΓͷEMRɺ ਐԽ͗ͯ͢͠ผ෺ͩͬͨ݅ ͡ΎΜ͡ΎΜʢ@yoshiyama_hanaʣ END