Upgrade to Pro — share decks privately, control downloads, hide ads and more …

実践的データ基盤への処方箋_2-9_2-12

Avatar for Tomoya Koike Tomoya Koike
February 03, 2022

 実践的データ基盤への処方箋_2-9_2-12

Avatar for Tomoya Koike

Tomoya Koike

February 03, 2022
Tweet

More Decks by Tomoya Koike

Other Decks in Programming

Transcript

  1. ϩάͱ͸ 3 Α͘෼ੳ͞ΕΔϩά 1. WebαʔόͷΞΫηεϩά 2. ΞϓϦέʔγϣϯͷϩά WebαʔόͷΞΫηεϩάͷྫ • ΞΫηεͨ࣌ؒ͠ɺURL

    • ΞΫηεݩͷIPΞυϨε • ΞΫηεʹ༻͍ͨ୺຤৘ใ • Web App͕ઃఆͨ͠ɺϢʔβࣝผ৘ใ
  2. ϩάऩू͕Ͱ͖Δ੡඼ 5 ੡඼໊ ఏڙํ๏ ఏڙɾαϙʔτ ͍ͯ͠Δձࣾ fluentd, fluent-bit OSS Treasure

    Data Logstash OSS Elastic CloudWatch Cloud AWS Cloud Logging Agent Cloud GCP
  3. ETL੡඼ͱ͸ 14 ఏڙܗଶͷҧ͍ ETL…Extract Transform LoadͷུͰɺσʔλͷநग़ɺՃ޻ɺϩʔυΛҙຯ͢Δɻ ෳࡶͳՃ޻͕Ͱ͖Δ͔Ͳ͏͔ ҟͳΔσʔλιʔε͔ΒͷσʔλΛՃ޻ɺ݁߹ͯ͠ϩʔυ͢Δ౳ɻ Apache Nifi,

    DataSpider, Glue, Cloud Data Fusion OSSɺ༗ঈ੡඼ɺΫϥ΢υͷ3छྨɻ OSS…embulk΍fluentd, Apache Sqoop, ༗ঈ੡඼ͱͯ͠DataSpider, ASTERIAͳͲɻ Ϋϥ΢υͰ͸AWSͷGlue΍GCPͷCloud Data Fusion, ྆αʔϏεͷDMS, embulkͷϚωʔδυαʔϏεͰ͋ΔtroccoͳͲɻ ※DMS…Database Migration Service