Upgrade to Pro — share decks privately, control downloads, hide ads and more …

未経験からデータ基盤を1年で作ってみた

 未経験からデータ基盤を1年で作ってみた

2023/05/11(木)に行われた
【AWS構築・開発の技術勉強会】
にて発表した
「未経験からデータ基盤を1年で作ってみた」
の資料です。

https://beyondjapan.connpass.com/event/280427/

Takafumi Omuro

May 11, 2023
Tweet

More Decks by Takafumi Omuro

Other Decks in Technology

Transcript

  1. !.PSJTBXB*OD
    ະܦݧ͔Βσʔλج൫Λ೥Ͱ࡞ͬͯΈͨ
    גࣜձࣾϞϦαϫখࣨو࢙
    !.PSJTBXB*OD

    View full-size slide

  2. !.PSJTBXB*OD
    w ࣗݾ঺հ
    w גࣜձࣾϞϦαϫͷ঺հ
    w .PSJTBXB'POUTͷΞʔΩςΫνϟ
    w σʔλج൫ͷΞʔΩςΫνϟ
    w ·ͱΊ
    ΞδΣϯμ

    View full-size slide

  3. !.PSJTBXB*OD
    ໊લɿখࣨو࢙ʢ͓ΉΖ͔ͨ;Έʣ
    ॴଐɿגࣜձࣾϞϦαϫ

    ɹɹɹγεςϜ։ൃ෦໳γχΞϚωʔδϟʔ
    ࢓ࣄɿςοΫϦʔυ

    ɹɹɹ"84ΠϯϑϥΤϯδχΞ
    झຯɿόΠΫ
    ࣗݾ঺հ

    View full-size slide

  4. !.PSJTBXB*OD

    View full-size slide

  5. !.PSJTBXB*OD
    ૑ۀͷϧʔπͰ͋Δࣸਅ২ࣈػൃ໌͔Β
    ΋͏͙͢प೥Λܴ͑Δ࿝ฮاۀ
    גࣜձࣾϞϦαϫ
    ೥ࣸਅͷݪཧͰจࣈΛݱͯ͠૊Ήํ๏Λ
    ੈքʹઌۦ͚ͯൃ໌͠ʮ๜จࣸਅ২ࣈػʯΛಛڐਃ੥

    View full-size slide

  6. !.PSJTBXB*OD
    ෺ཧഔମ͔ΒΫϥ΢υ΁
    ϑΥϯτൢചํ๏΋มֵ

    0QFO5ZQFϑΥϯτ
    ύοέʔδ

    ചΓ੾Γ

    .03*4"8"1"441035
    ύοέʔδ
    αϒεΫϦϓγϣϯ

    .PSJTBXB'POUT
    Ϋϥ΢υ
    αϒεΫϦϓγϣϯ

    View full-size slide

  7. !.PSJTBXB*OD
    ։ൃ෦໳ͷ.77
    w .*44*0/ଘࡏҙٛ
    <&WFSZXIFSFY8PSME8JEF>
    จࣈΛ௨ͯ͡ੈքதͷਓʑͷੜ׆ͷதʹଘࡏ͠ଓ͚Δ
    w 7*40/໨ࢦ࢟͢
    <&YQFSUY4VTUBJOBCJMJUZ>
    Ϟϊͮ͘Γͱٕज़Λָ͠ΊΔ<࣋ଓՄೳ>ͳ<ΤΩεύʔτूஂ>΁ਐԽ
    w 7"-6&૊৫ͷՁ஋
    <ڞ૑ͷਫ਼ਆ>૬ޓཧղͱϦεϖΫτɺΫϩεϘʔμʔ
    <σʔλυϦϒϯ>σʔλͱ߹ཧతͳߦಈ൑அʹՁ஋ج४Λ͓͘
    <ָ׆>ָͪΜY҆Β͔Yָ͍͠Λ௥ٻ͢Δ։ൃ׆ಈʹՁ஋Λ͓͘
    .*44*0/7*40/7"-6&ͷ੍ఆ

    View full-size slide

  8. !.PSJTBXB*OD
    ։ൃ෦໳ͷ.77
    w .*44*0/ଘࡏҙٛ
    <&WFSZXIFSFY8PSME8JEF>
    จࣈΛ௨ͯ͡ੈքதͷਓʑͷੜ׆ͷதʹଘࡏ͠ଓ͚Δ
    w 7*40/໨ࢦ࢟͢
    <&YQFSUY4VTUBJOBCJMJUZ>
    Ϟϊͮ͘Γͱٕज़Λָ͠ΊΔ<࣋ଓՄೳ>ͳ<ΤΩεύʔτूஂ>΁ਐԽ
    w 7"-6&૊৫ͷՁ஋
    <ڞ૑ͷਫ਼ਆ>૬ޓཧղͱϦεϖΫτɺΫϩεϘʔμʔ
    <σʔλυϦϒϯ>σʔλͱ߹ཧతͳߦಈ൑அʹՁ஋ج४Λ͓͘
    <ָ׆>ָͪΜY҆Β͔Yָ͍͠Λ௥ٻ͢Δ։ൃ׆ಈʹՁ஋Λ͓͘
    .*44*0/7*40/7"-6&ͷ੍ఆ

    View full-size slide

  9. !.PSJTBXB*OD
    σʔλج൫࡞ͬͯΈͨ

    View full-size slide

  10. !.PSJTBXB*OD
    αʔόʔϨεߏ੒Ͱͷ࣮૷
    .PSJTBXB'POUTͷ#BDLFOEΞʔΩςΫνϟ

    View full-size slide

  11. !.PSJTBXB*OD
    αʔόʔϨεߏ੒Ͱͷ࣮૷
    .PSJTBXB'POUTͷ#BDLFOEΞʔΩςΫνϟ
    τϥϯβΫγϣϯσʔλ͸

    %ZOBNP%#ʹอଘ

    View full-size slide

  12. !.PSJTBXB*OD
    σʔλج൫ઐ༻ͷ"84Λ४උ
    σʔλج൫ͷΞʔΩςΫνϟ

    View full-size slide

  13. !.PSJTBXB*OD
    σʔλج൫ઐ༻ͷ"84Λ४උ
    σʔλج൫ͷΞʔΩςΫνϟ
    ෼ੳج൫͸ผ"84
    4ͷΫϩεΞΧ΢ϯτઃఆ

    View full-size slide

  14. !.PSJTBXB*OD
    %ZOBNP%#৘ใΛ4΁ఆظతʹग़ྗ
    σʔλج൫ͷΞʔΩςΫνϟ
    %ZOBNP%#ͷσʔλ

    1J53ઃఆͰ4΁ΤΫεϙʔτ

    View full-size slide

  15. !.PSJTBXB*OD
    %ZOBNP%#৘ใΛ4΁ఆظతʹग़ྗ
    σʔλج൫ͷΞʔΩςΫνϟ
    "QBDIF)JWFܗࣜͰύεΛ੾Δ
    -BNCEBΛ࢖࣮ͬͯ૷

    View full-size slide

  16. !.PSJTBXB*OD
    $MPVE8BUDI-PHTΛ4΁ετϦʔϜग़ྗ
    σʔλج൫ͷΞʔΩςΫνϟ
    $MPVE8BUDI-PHTͷσʔλ

    ,JOFTJT'JSFIPTFͰग़ྗ

    View full-size slide

  17. !.PSJTBXB*OD
    σʔλϨΠΫ׬੒
    σʔλج൫ͷΞʔΩςΫνϟ

    View full-size slide

  18. !.PSJTBXB*OD
    ఆظతʹσʔλϚʔτ࡞੒
    σʔλج൫ͷΞʔΩςΫνϟ
    4UFQ'VODUJPOT͔Β-BNCEBܦ༝Ͱ
    (MVF8PSL
    fl
    PX࣮ߦ

    View full-size slide

  19. !.PSJTBXB*OD
    ఆظతʹσʔλϚʔτ࡞੒
    σʔλج൫ͷΞʔΩςΫνϟ
    -BNCEB͸(MVF8PSL
    fl
    PXͷ
    ύϥϝʔλ࡞͍ͬͯΔ͚ͩ

    View full-size slide

  20. !.PSJTBXB*OD
    ఆظతʹσʔλϚʔτ࡞੒
    σʔλج൫ͷΞʔΩςΫνϟ
    (MVF+PCTͰσʔλϚʔτ࡞੒

    1Z4QBSLͰ࣮૷

    View full-size slide

  21. !.PSJTBXB*OD
    ఆظతʹσʔλϚʔτ࡞੒
    σʔλج൫ͷΞʔΩςΫνϟ
    "UIFOBͰॊೈͳݕࡧ͕Մೳʹ

    View full-size slide

  22. !.PSJTBXB*OD
    ఆظతʹσʔλϚʔτ࡞੒
    σʔλج൫ͷΞʔΩςΫνϟ
    "UIFOB͔Β2VJDL4JHIUͷ

    41*$&ʹఆظతʹσʔλऔಘ

    View full-size slide

  23. !.PSJTBXB*OD
    γϯάϧαΠϯΦϯͰͷϩάΠϯରԠ
    σʔλج൫ͷΞʔΩςΫνϟ
    ࣾ಺ެ։͸440ରԠ

    .4ͱ࿈ܞ͠4".-ೝূ

    View full-size slide

  24. !.PSJTBXB*OD
    2VJDL4JHIUͷμογϡϘʔυ࡞੒
    σʔλج൫ͷΞʔΩςΫνϟ
    ˞ςετ؀ڥͷΩϟϓνϟʹগ͠ϞβΠΫ͔͚ͯ·͢

    View full-size slide

  25. !.PSJTBXB*OD
    ࢀߟॻ੶
    σʔλج൫ͷΞʔΩςΫνϟ

    View full-size slide

  26. !.PSJTBXB*OD
    σʔλαΠΤϯςΟετ΋
    ܴ͑ͯ͞ΒͳΔվળΛଓ͚ͨ

    View full-size slide

  27. !.PSJTBXB*OD
    σʔλϨΠΫʹ৽ͨͳσʔλιʔεͷ௥Ճ
    σʔλج൫ͷΞʔΩςΫνϟ

    View full-size slide

  28. !.PSJTBXB*OD
    (PPHMF"OBMZUJDTͷσʔλऔΓࠐΈ
    σʔλج൫ͷΞʔΩςΫνϟ
    "QQ'MPXͷόά࢓༷ͷ໰୊Ͱ

    -BNCEB͔Β"1*Λୟ͍ͯऔಘ

    View full-size slide

  29. !.PSJTBXB*OD
    σʔλϨΠΫʹ͞ΒͳΔσʔλιʔεͷ௥Ճ
    σʔλج൫ͷΞʔΩςΫνϟ
    5(8௥Ճ
    ձࣾͱ71/ΛషͬͯΦϯϓϨϛεͷ
    %#͔ΒσʔλΛऔಘ

    View full-size slide

  30. !.PSJTBXB*OD
    ΞυϗοΫͳΫΤϦΛ͔͚ΔΑ͏ʹରԠ
    σʔλج൫ͷΞʔΩςΫνϟ
    440ͰϩάΠϯͯ͠

    "UIFOBʹΫΤϦൃߦΛՄೳʹ

    View full-size slide

  31. !.PSJTBXB*OD
    ΞυϗοΫͳΫΤϦΛ͔͚ΔΑ͏ʹରԠ
    σʔλج൫ͷΞʔΩςΫνϟ
    %ZOBNP%#ͷσʔλʹରͯ͠
    ॊೈͳݕࡧ͕Մೳʹ

    View full-size slide

  32. !.PSJTBXB*OD
    41*$&ͷ༰ྔʹ͸஫ҙ
    σʔλج൫ͷΞʔΩςΫνϟ
    ؅ཧऀͷਓ਺ʹ४ͯ͡

    ແྉͰ41*$&༰ྔ͕૿͑Δ

    View full-size slide

  33. !.PSJTBXB*OD
    ·ͱΊ
    ෼ੳج൫ߏஙͷ·ͱΊ
    w ෼ੳج൫͸ผ"84ͱͯ͠ߏங
    w σʔλϨΠΫͷ࡞੒
    w %ZOBNP%#ͷσʔλ1J53Ͱ4΁ग़ྗ
    w $MPVE8BUDI-PHT͔Β,JOFTJT'JSFIPTFͰ4΁࿈ܞ
    w σʔλϚʔτͷ࡞੒
    w (MVF8PSL
    fl
    PXΛ༻͍ͯ(MVF%BUB$BUBMPHΛ࡞੒
    w 440ϩάΠϯରԠ
    w 2VJDL4JHIUɺ"UIFOB΁؆୯ʹΞΫηε
    w ࠓޙͷ՝୊
    w σʔλྔͷ૿େʹΑΔॲཧ࣌ؒͷ૿Ճ
    w 41*$&ͷར༻ྔͷ૿Ճ

    View full-size slide

  34. !.PSJTBXB*OD
    ະܦݧ͔ΒͰ΋σʔλج൫ߏங͕Ͱ͖·ͨ͠
    σʔλαΠΤϯςΟετ͕ՃΘΓՃ଎͍͖ͯ͠·ͨ͠

    View full-size slide

  35. !.PSJTBXB*OD
    ͋Γ͕ͱ͏͍͟͝·ͨ͠

    ஥ؒืूதͰ͢ʂ
    IUUQTXXXNPSJTBXBDPKQBCPVUSFDSVJU

    View full-size slide