Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWS Step Functions × AWS SAMで実現する家族ノートの低運用コストETL基盤/ kazokunote-stepfunctions-awssam-etl

AWS Step Functions × AWS SAMで実現する家族ノートの低運用コストETL基盤/ kazokunote-stepfunctions-awssam-etl

2021/09/29 AWS Devday Online Japan 2021の登壇資料です

Shoichiro Nagai(shnagai)

September 29, 2021
Tweet

More Decks by Shoichiro Nagai(shnagai)

Other Decks in Technology

Transcript

  1. © 2021, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AWS Step Functions × AWS Serverless Application Model (AWS SAM) Ͱ࣮ݱ͢ΔՈ଒ϊʔτͷ௿ӡ༻ίετETLج൫ ίωώτגࣜձࣾ ӬҪউҰ࿠(@shnagai) A W S D e v D a y O n l i n e J a p a n 2 0 2 1 2 0 2 1 / 0 9 / 2 9 E - 3
  2. © 2021, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. ίωώτͰͷETLج൫Λ࡞Δʹࢸͬͨ σʔλपΓͷ՝୊
  3. σʔλͷ׆༻ํ๏͸ݸผ࠷దԽ͞Ε͍ͯΔ w ػցֶश1+Ͱ͸ w "84(MVFͰ%#ͷσʔλΛநग़ͯ͠$47Խ w 1ZUIPOͷόονͰߦಈϩάΛ֎෦%8)͔Βநग़͠$47Խ w 1BOEBTͰͭΛ݁߹ͯͭ͠ͷσʔλͱͯ͠ѻ͏ w

    σʔλநग़Ͱ͸ w #*πʔϧ্Ͱ%#ͱ֎෦%8)ͷߦಈϩάΛ݁߹͢Δύλʔϯ w όονॲཧͰ%#ͱ֎෦%8)ͷߦಈϩάΛ݁߹͢Δύλʔϯ ͍͍ͩͨಉ͡Α͏ͳॲཧ͕ ࢸΔͱ͜ΖͰߦΘΕ͍ͯΔঢ়گ
  4. σʔλऩू͸ߟ͑Δ͜ͱ͕ଟ͍ɻɻ w Ͳͷ%#ʹͲͷݖݶͰΞΫηε͢Ε͹͍͍ͷ͔  w ऩूॲཧͰҙਤͤ͵σʔλϩετ͕ى͖͍ͯͳ͍͔  w ऩूͨ͠σʔλ͸৴པग़དྷΔͷ͔ 

    w σʔλ͕େྔͷ৔߹͸େ͖ͳίϯϐϡʔτϦιʔε΋ඞཁ w ଞͰಈ͍͍ͯΔॲཧͱόοςΟϯάͯ͠%#ʹେ͖ͳෛՙΛ͔͚ͳ͍͔  w %#෼ੳ%#ͱ͍͏αʔϏε͔Β͸੾Γ཭ͨ͠%#Λ༻ҙ͍ͯ͠Δ
  5. σʔλऩू͸ߟ͑Δ͜ͱ͕ଟ͍ɻɻ w Ͳͷ%#ʹͲͷݖݶͰΞΫηε͢Ε͹͍͍ͷ͔  w ऩूॲཧͰҙਤͤ͵σʔλϩετ͕ى͖͍ͯͳ͍͔  w ऩूͨ͠σʔλ͸৴པग़དྷΔͷ͔ 

    w σʔλ͕େྔͷ৔߹͸େ͖ͳίϯϐϡʔτϦιʔε΋ඞཁ w ଞͰಈ͍͍ͯΔॲཧͱόοςΟϯάͯ͠%#ʹେ͖ͳෛՙΛ͔͚ͳ͍͔  w %#෼ੳ%#ͱ͍͏αʔϏε͔Β͸੾Γ཭ͨ͠%#Λ༻ҙ͍ͯ͠Δ ݸผͰ࣮૷͢Δͷ͸ޮ཰͕ѱ͍ͷ͸ཧղͭͭ͠΋ ઐ೚ͷσʔλΤϯδχΞ΋͓Βͣ༏ઌ౓্͕͕Βͣ
  6. ͷσʔληοτ࡞੒ΛͲͷํࣜʹ͢Δͷ͔ݕ౼ ɹɹɹɹ ֎෦ DWH όον ॲཧ ֎෦DWH όον ॲཧ ಠࣗͰ

    ऩू/݁߹ σʔλϚʔτ σʔλϨΠΫ ऩू ݁߹ औಘ ᶃόονͰσʔλͷऩू/݁߹Λߦ͍৽͍͠σʔληοτΛ࡞Δύλʔϯ ᶄΞϓϦέʔγϣϯ͸σʔλऩू/݁߹͸ҙࣝͤͣʹ σʔλϚʔτΛར༻͢Δ͚ͩʹ͢Δύλʔϯ
  7. ͷσʔληοτ࡞੒ΛͲͷํࣜʹ͢Δͷ͔ݕ౼ ɹɹɹɹ ֎෦ DWH όον ॲཧ ֎෦DWH όον ॲཧ ಠࣗͰ

    ऩू/݁߹ σʔλϚʔτ σʔλϨΠΫ ऩू ݁߹ औಘ ᶃόονͰσʔλͷऩू/݁߹Λߦ͍৽͍͠σʔληοτΛ࡞Δύλʔϯ ᶄΞϓϦέʔγϣϯ͸σʔλऩू/݁߹͸ҙࣝͤͣʹ σʔλϚʔτΛར༻͢Δ͚ͩʹ͢Δύλʔϯ ϓϩμΫτʹ͓͚Δσʔλͷॏཁ౓ͱ σʔλ׆༻ͷαΠϩԽΛ๷͙ͱ͍͏͜ͱΛ໨తʹ ᶄΛ࣮ݱ͢ΔͨΊʹσʔλج൫Λ࡞Δ͜ͱΛܾఆ
  8. © 2021, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. ETLج൫ͷٕज़બఆ
  9. ίωώτͷσʔλج൫ ϚϚϦ಺QAσʔλ Ϣʔβσʔλ ೔࣍ͰඞཁͳσʔλΛ֎෦DWHʹసૹ DataLake ߦಈϩά ϚϚϦ಺QAσʔλ Ϣʔβσʔλ DataMart ΞϓϦέʔγϣϯ͔Β

    ࢀর͢ΔϏϡʔ DataWarehouse ूܭσʔλ Ո଒ϊʔτͷΞϓϦέʔγϣϯ ֎෦DWH BIπʔϧ Ո଒ϊʔτ σʔλΛࢀর
  10. ࠓճͷ࿩ͷείʔϓ ϚϚϦ಺QAσʔλ Ϣʔβσʔλ ೔࣍ͰඞཁͳσʔλΛ֎෦DWHʹసૹ DataLake ߦಈϩά ϚϚϦ಺QAσʔλ Ϣʔβσʔλ DataMart ΞϓϦέʔγϣϯ͔Β

    ࢀর͢ΔϏϡʔ DataWarehouse ूܭσʔλ Ո଒ϊʔτͷΞϓϦέʔγϣϯ ֎෦DWH BIπʔϧ Ո଒ϊʔτ σʔλΛࢀর
  11. ϫʔΫϑϩʔΤϯδϯൺֱ 4UFQ'VODUJPOTͱ%JHEBH ߲໨/ϫʔΫϑϩʔछྨ Step Functions Digdag جຊػೳ δϣϒ࣮ߦ/ϦτϥΠ/ Τϥʔॲཧ ˕

    ˕ ؅ཧը໘ ̋ ˕ FargateλεΫͷ࣮ߦ ˕ ̋ ίʔυ؅ཧ ̋ ※AWS SAMͰΧόʔ ˕ ϫʔΫϑϩʔϓϩηε ͷ؅ཧ ˕ ✕
  12. ϫʔΫϑϩʔΤϯδϯൺֱ 4UFQ'VODUJPOTͱ%JHEBH ߲໨/ϫʔΫϑϩʔछྨ Step Functions Digdag جຊػೳ δϣϒ࣮ߦ/ϦτϥΠ/ Τϥʔॲཧ ˕

    ˕ ؅ཧը໘ ̋ ˕ FargateλεΫͷ࣮ߦ ˕ ̋ ίʔυ؅ཧ ̋ ※AWS SAMͰΧόʔ ˕ ϫʔΫϑϩʔϓϩηε ͷ؅ཧ ˕ ✕ ػೳ໘͸ͲͪΒ΋ຬ͍͕ͨͯͨ͠ ӡ༻ίετͷ௿͞ͰStep FunctionsΛ࠾༻
  13. ϫʔΫϑϩʔΤϯδϯࣗମͷӡ༻ίετ͕ܾΊखʹͳͬͨ w %JHEBHϓϩηεΛͲ͔͜Ͱৗறͤ͞ͳ͍ͱ͍͚ͳ͍఺͕ωοΫʹͳͬͨ w &$4͔&$Ͱ͍͘ͷ͔ͷબఆ΋͍Δ͠ӡ༻΋ඞཁ w %JHEBHϓϩηεͷ؂ࢹ΋ඞཁ w %JHEBHࣗମͷӡ༻΋ඞཁ όʔδϣϯΞοϓରԠ΍ϓϥάΠϯௐࠪ

     w %JHEBH಺ͷδϣϒͷ࣮ߦ݁Ռ΋΋ͪΖΜ؂ࢹ͢Δ w ΋ͪΖΜ%JHEBH͕༏Ґͳ఺΋͋Δ͕ࠓճ͸௿ӡ༻ίετ͕伴ͩͬͨ w 4UFQ'VODUJPOTͩͱδϣϒͷ࣮ߦ݁Ռͷ؂ࢹ͚ͩͰ0, w ͲΜͳॲཧΛͲΜͳϑϩʔͰಈ͔͔͚ͩ͢ʹઐ೦ͯ͠։ൃग़དྷΔ w ͜ΕͧαʔόϨεͷڧ͞
  14. © 2021, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Step FunctionsͰ૊Ή ECS/Fargate, LambdaͷϫʔΫϑϩʔࣄྫ
  15. 4UFQ'VODUJPOTͷओͳొ৔ਓ෺ w 4UBUFT w ೖྗʹج͍ͮͯΞΫγϣϯΛ࣮ߦͯ͠ग़ྗΛଞͷঢ়ଶʹ౉͢୯Ґ w ͭͷύλʔϯ͕༻ҙ͞Ε͍ͯΔ w 5BTL ͳΜΒ͔ͷॲཧ࣮ߦ

     w $IPJDF ೖྗʹج͖ͮॲཧ෼ذ  w 'BJMPS4VDDFTT ࣦഊPS੒ޭͰ࣮ߦఀࢭ  w 1BTT ೖྗΛग़ྗʹ౉͢  w 8BJU Ұఆ࣌ؒεϦʔϓ  w 1BSBMMFM ฒྻ࣮ߦ  w .BQ ಈతʹ൓෮࣮ߦ
  16. "VSPSBͷςʔϒϧΛ֎෦%8)ʹҠૹ͢Δ&NCVMLͷ4UBUFT w &NCVMLͷॲཧͱ'BSHBUFλεΫΛ w 5ZQF5BTL w 3FTPVSDFTZODͰऴྃ଴ͪ w 1BSBNFUFSTͰ'BSHFUFͷఆٛ w

    1BSBNFUFST5BTL%FGJOJUJPO͸ڞ௨ w ڞ௨ͷ%PDLFSΠϝʔδΛ࢖͍ͬͯΔͷ Ͱɺ$POUBJOFS0WFSSJEFTΛ࢖͍$.%۟Λ ্ॻ͖͢Δ͜ͱͰ&NCVMLͷ࣮ߦॲཧΛ෼ ͚͍ͯΔ
  17. 4UFQ'VDUJPOTͷ։ൃεςʔδϯά؀ڥΛͲ͜·Ͱ༻ҙ͢Δ΂͖͔ w ௨ৗͷ։ൃͱಉ͘͡ຊ൪εςʔδϯά؀ڥ͕͋Δ͜ͱ͕ཧ૝Ͱ͸͋Δ w ͔͠͠ɺ w &5-ج൫ͩͱͦͦ͜͜ॏ͍ॲཧͳͷͰຖ೔࢖Θͳ͍σʔλΛ࡞Δͷ͸අ༻໘Ͱແବ w ࢖Θͳ͍ҙຯͷͳ͍ϫʔΫϑϩʔຖ೔ಈ͔͢΂͖͔ 

    w ͨͩɺ։ൃϑϩʔͷதͰ4UFQ'VODUJPOTͷಈ࡞֬ೝͱͯ͠ຊ൪ద༻લͷεςʔδϯά͸ඞਢ w ݁ՌɺεέδϡʔϧΛແޮԽͯ͠։ൃ࣌ʹखಈͰ4UFQ'VODUJPOTΛ࣮ߦͯ͠ಈ࡞֬ೝ͢Δܗʹ w &5-΍σʔλج൫ϫʔΫϑϩʔͷεςʔδϯάΛͲ͏؅ཧ͢Δ΂͖ͳͷ͔ ͸·ͩਖ਼ղΛݟ͚ͭΒΕ͍ͯͳ͍ͷ͕ਖ਼௚ͳͱ͜ΖͳͷͰ΍Γͳ͕Βਖ਼ղΛ ݟ͚͍͖͍ͭͯͨ
  18. © 2021, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AWS SAMΛ࢖ͬͨ ϫʔΫϑϩʔͷίʔυ؅ཧ
  19. "844".ͱ͸Կ͔ w "844FSWFSMFTT"QQMJDBUJPO.PEFMͷུশ w αʔόϨεΞϓϦέʔγϣϯߏங༻ͷϑϨʔϜϫʔΫ w "84$MPVE'PSNBUJPO $'O ΑΓ؆୯ͳهड़͕ग़དྷΔͷ͕ಛ௃ w

    σϓϩΠ࣌ʹ͸ɺ$'Oʹม׵͞Ε࣮ͯߦ͞ΕΔ w 4".$-*ʹΑΔσϓϩΠ΍ϩʔΧϧ࣮ߦ͕ग़དྷΔ w ྫ͑͹ɺ-BNCEBΛϩʔΧϧ࣮ߦ͢ΔΑ͏ͳ͜ͱ͕Մೳ
  20. "844".Ͱѻ͑Δ"84Ϧιʔε w 5ZQFʹࢦఆՄೳͳϦιʔεୡ w "844FSWFSMFTT4UBUF.BDIJOF˞4UFQ'VODUJPOT w "844FSWFSMFTT'VODUJPO˞MBNCEB w "844FSWFSMFTT"QJ˞"1*(BUFXBZ w

    "844FSWFSMFTT4JNQMF5BCMF˞%ZOBNP%# w ʜ IUUQTEPDTBXTBNB[PODPNKB@KQTFSWFSMFTTBQQMJDBUJPONPEFMMBUFTUEFWFMPQFSHVJEFTBNTQFDJGJDBUJPOSFTPVSDFTBOEQSPQFSUJFTIUNM
  21. &5-ج൫Λ؅ཧ͢ΔϦϙδτϦͷߏ੒ FNCVML FNCVMLͷఆٛ  cDPOG FNCVMLͷఆٛ ˒ TBN "844".ͷఆٛ 

    cTUBUFNBDIJOF 4UFQ'VODUJPOTͷ"4- ˒ cGVODUJPOT -BNCEBͷίʔυ  cFOW cEFW EFW؀ڥ༻ͷఆٛ  cTBNDPOGJHUPNM 4".σϓϩΠ࣌ͷઃఆ  cUFNQMBUFZBNM 4".ͷఆٛ˞$MPVE'PSNBUJPOʹδΣωϨʔτ͞ΕΔ  cQSE QSE؀ڥ༻ͷఆٛ  cTBNDPOGJHUPNM 4".σϓϩΠ࣌ͷઃఆ  cUFNQMBUFZBNM 4".ͷఆٛ˞$MPVE'PSNBUJPOʹδΣωϨʔτ͞ΕΔ 
  22. &NCVMLҎ֎ͷϫʔΫϑϩʔؔ࿈͸શͯ"844".Ͱ؅ཧ FNCVML FNCVMLͷఆٛ  cDPOG FNCVMLͷఆٛ ˒ TBN "844".ͷఆٛ 

    cTUBUFNBDIJOF 4UFQ'VODUJPOTͷ"4- ˒ cGVODUJPOT -BNCEBͷίʔυ  cFOW cEFW EFW؀ڥ༻ͷఆٛ  cTBNDPOGJHUPNM 4".σϓϩΠ࣌ͷઃఆ  cUFNQMBUFZBNM 4".ͷఆٛ˞$MPVE'PSNBUJPOʹδΣωϨʔτ͞ΕΔ  cQSE QSE؀ڥ༻ͷఆٛ  cTBNDPOGJHUPNM 4".σϓϩΠ࣌ͷઃఆ  cUFNQMBUFZBNM 4".ͷఆٛ˞$MPVE'PSNBUJPOʹδΣωϨʔτ͞ΕΔ 
  23. UFNQMBUFZBNMͷϨΠΞ΢τ w 5SBOTGPSNηΫγϣϯͰ$'Oʹม׵͢Δ ϚΫϩΛએݴ w $'OʹΑͬͯϗετ͞ΕΔϚΫϩͰɺ"84 4".ߏจͰهड़͞ΕͨςϯϓϨʔτશମΛ ड͚औΓɺͦΕΛ४ڌͨ͠$'OςϯϓϨʔ τʹม׵͓Αͼ֦ு w

    5ZQFΛ"844FSWFMFTTʹ͠ͳ͍৔߹͸ $'Oͷจ๏΋௚઀࢖͑Δ IUUQTEPDTBXTBNB[PODPNKB@KQ"84$MPVE'PSNBUJPO MBUFTU6TFS(VJEFUSBOTGPSNBXTTFSWFSMFTTIUNM
  24. &5-ॲཧΛ௥Ճ͢Δͱ͖͸Օॴͷमਖ਼ͰࡁΉΑ͏ʹ͍ͯ͠Δ FNCVML FNCVMLͷఆٛ  cDPOG FNCVMLͷఆٛ ˒ TBN "844".ͷఆٛ 

    cTUBUFNBDIJOF 4UFQ'VODUJPOTͷ"4- ˒ cGVODUJPOT -BNCEBͷίʔυ  cFOW cEFW EFW؀ڥ༻ͷఆٛ  cTBNDPOGJHUPNM 4".σϓϩΠ࣌ͷઃఆ  cUFNQMBUFZBNM 4".ͷఆٛ˞$MPVE'PSNBUJPOʹδΣωϨʔτ͞ΕΔ  cQSE QSE؀ڥ༻ͷఆٛ  cTBNDPOGJHUPNM 4".σϓϩΠ࣌ͷઃఆ  cUFNQMBUFZBNM 4".ͷఆٛ˞$MPVE'PSNBUJPOʹδΣωϨʔτ͞ΕΔ 
  25. ·ͱΊ w ग़དྷΔ͚ͩӡ༻ίετΛ௿͘͢Δͱ͍͏؍఺Ͱ૊Μͩ&5-ج൫ʹ͍ͭͯ঺ հ͠·ͨ͠ w &5-ج൫ͱฉ͘ͱӡ༻΋ؚΊͯେมͱ͍͏Πϝʔδ͕͋Δ͕ɺ"84Λ׆༻ ͢Δ͜ͱͰൺֱత௿ӡ༻ίετͰ૊Ή͜ͱ΋ग़དྷΔ w &NCVMLΛ'BSHBUFͰαʔόϨε ૄ

    ʹಈ͔͠ w 4UFQ'VODUJPOTΛ࢖͏͜ͱͰৗறϓϩηεͳ͠ʹϫʔΫϑϩʔΛ࡞Γ w "844".Λ࢖͏͜ͱͰγϯϓϧͳهड़ͰαʔόϨεϦιʔεΛ؅ཧ ࠓճͷߏ੒͕ਖ਼ղͱ͍͏Θ͚Ͱ͸ܾͯ͠ͳ͍ɻ AWSͷਐԽʹ௥͍͖ͭͳ͕Β෢ثΛ૿΍͠ ՝୊ʹԠͯ͡ద੾ͳબ୒Λͯ͠ɺ ࣮૷͍ͯ͘͠ྗ͕͜Ε͔ΒΑΓେࣄʹͳͬͯ͘Δͱࢥ͏ɻ
  26. Thank you! © 2021, Amazon Web Services, Inc. or its

    affiliates. All rights reserved. ίωώτגࣜձࣾ ӬҪউҰ࿠(shnagai)