Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サーバレス時代におけるヘテロジニアス時系列データベースアーキテクチャ / HeteroTSDB

サーバレス時代におけるヘテロジニアス時系列データベースアーキテクチャ / HeteroTSDB

第2回ウェブシステムアーキテクチャ研究会

Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Research

Transcript

  1. αʔόϨε࣌୅ʹ͓͚Δ
    ϔςϩδχΞε࣌ܥྻσʔλϕʔε
    ΞʔΩςΫνϟ
    ୈ2ճ Web System Architecture(WSA) ݚڀձ@෱Ԭ
    ͸ͯͳ id:y_uuki / @y_uuk1

    View full-size slide

  2. id:y_uuki / @y_uuk1
    Yuuki TSUBOUCHI
    https://yuuk.io/
    ͸ͯͳͷ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ
    WSAݚ ෭ࠪ

    View full-size slide

  3. ΋͘͡
    1. എܠͱ໨త
    2. ΞʔΩςΫνϟ
    3. ࣮૷
    4. ߟ࡯
    5. ·ͱΊͱࠓޙͷ՝୊

    View full-size slide

  4. 1. എܠͱ໨త

    View full-size slide

  5. ߴ͍ੑೳΛ΋ͭ࣌ܥྻDBͷඞཁੑ
    • ΢ΣϒγεςϜ͕ෳࡶԽͨ݁͠ՌɺΑΓΑ͍ϞχλϦϯ
    άٕज़͕ඞཁͱ͞Ε͍ͯΔ
    • ΑΓΑ͍ϞχλϦϯάͷͨΊʹɺ࣌ܥྻσʔλΛߴස౓
    ͔ͭେྔʹऩू͠ɺ௕ظؒอଘ͍ͨ͠

    View full-size slide

  6. ࣌ܥྻDBͷ࣮૷ํࣜ - ಺෦DBMSํࣜ
    • ࣌ܥྻσʔλϕʔεʹ࠷దԽ͞ΕͨΤϯδϯΛ࣮૷
    • ੑೳͱσʔλอଘޮ཰͸Α͍܏޲͕͋Δ
    • ҰํͰɺ෼ࢄγεςϜͱͯ͠ͷ৴པੑʹ͍ͭͯɺ࣮ݧஈ
    ֊Ͱ͋Δ͜ͱ΋ଟ͍
    • GraphiteɺInfluxDBɺPrometheusͳͲ
    [4]: Jensen, Søren Kejser, et al. "Time Series Management Systems: A Survey." IEEE Transactions on Knowledge and Data Engineering, vol.29, no.11,
    2017, pp. 2581-2600.
    [5]: B. Mitschang et al. "Survey and Comparison of Open Source Time Series Databases.", In Proceedings of Database Systems for Business,
    Technology, and Web, 2017

    View full-size slide

  7. ࣌ܥྻDBͷ࣮૷ํࣜ - ֎෦DBMSํࣜ
    • ൚༻DBMS্ͷΞϓϦέʔγϣϯͱ࣮ͯ͠૷
    • ෼ࢄγεςϜͱͯ͠ͷ৴པੑʹؔΘΔ࣮૷͕੒ख़͍ͯ͠
    Δ܏޲͕͋Δ
    • ੑೳͱޮ཰͸಺෦DBMSํࣜʹྼΔ܏޲͕͋Δ
    • OpenTSDB(HBase)ɺKairosDB(Cassandra)ͳͲ
    [4]: Jensen, Søren Kejser, et al. "Time Series Management Systems: A Survey." IEEE Transactions on Knowledge and Data Engineering, vol.29, no.11,
    2017, pp. 2581-2600.
    [5]: B. Mitschang et al. "Survey and Comparison of Open Source Time Series Databases.", In Proceedings of Database Systems for Business,
    Technology, and Web, 2017

    View full-size slide

  8. ࣌ܥྻDBͷ՝୊
    • ಺෦DBMSํࣜͷߴՄ༻ੑɺεέʔϥϏϦςΟͷ੒ख़౓
    • ಺෦DBMSํࣜͷߴՄ༻ੑ࣮૷ͷ೉͠͞
    • ֎෦DBMSํࣜͷੑೳͱޮ཰ͷ௿͞
    ߴ৴པੑΛٻΊΒΕΔϢʔεέʔεΛ૝ఆ͠
    ֎෦DBMSํࣜΛ࠾༻ͭͭ͠ɺੑೳͱޮ཰ͷΑ͍Ξʔ
    ΩςΫνϟΛఏҊ͢Δ

    View full-size slide

  9. ϔςϩδχΞε࣌ܥྻDBͷఏҊ
    • ಛੑͷҟͳΔෳ਺ͷ֎෦DBMSΛ૊Έ߹Θͤɺಁաతʹ
    ΞΫηεՄೳͳ࣌ܥྻDBΛఏҊ͢Δ
    • DiamondΞʔΩςΫνϟͱݺͿ͜ͱͱ͢Δ

    View full-size slide

  10. 2. ΞʔΩςΫνϟ

    View full-size slide

  11. ΞʔΩςΫνϟͷཁ݅
    • (a) ࣌ܥྻσʔλʹର͢Δߴ͍ॻ͖ࠐΈεέʔϧΞ΢τੑ
    • (b) ߴՄ༻ੑ
    • (c) ॻ͖ࠐΈI/Oͱσʔλอ࣋ͷͦΕͧΕʹ͍ͭͯɺߴ͍ίϯ
    ϐϡʔςΟϯάϦιʔεޮ཰
    • (d) ϦιʔεͷϓϩϏδϣχϯά͓Αͼނো࣌ͷ෮چͷ༰қ͞
    • (e) ݕূ͓Αͼ։ൃͷ༰қ͞

    View full-size slide

  12. ಈ࡞ϑϩʔ
    ϝοηʔδ
    ϒϩʔΧʔ
    write path
    ΠϯϝϞϦ
    ,74
    Φϯ44%
    ,74
    Φϯ)%%
    ,74
    SFBEFS
    OPEF
    read path
    XSJUFS
    OPEF
    flush
    flush

    View full-size slide

  13. σʔλߏ଄
    • KVSؒͰͷσʔλ੔߹ੑΛ͍͔ʹ୲อ͢Δ͔
    • KVS্ͰҎԼͷ2఺Λୡ੒͠ɺղܾ͢Δ
    • (1) ࣌ܥྻσʔλͷ΂͖౳ॻ͖ࠐΈ
    • (2) Valueʹؚ·ΕΔσʔλϙΠϯτ਺ͷ࠷େ஋Λ੍ޚ
    • (1) ͸ϋογϡϚοϓʹΑΓ࣮ݱ
    • (2) ͸ղ૾౓ΞϥΠϯϝϯτͱλΠϜ΢Οϯυ΢ʹΑΓ࣮ݱ

    View full-size slide

  14. σʔλߏ଄: ϋογϡϚοϓ
    • timestampΛΩʔɺvalueΛ஋ͱͨ͠ϋογϡϚοϓ
    • ϋογϡϚοϓʹର͢Δ্ॻ͖ʹΑΔ΂͖౳ॻ͖ࠐΈ͕
    Մೳ
    • ௥هͷͨͼʹɺValueαΠζ͕૿େ͠ɺιϑτϦϛοτ
    ·ͨ͸ϋʔυϦϛοτΛ௒͑ΔՄೳੑ͕͋Δ
    (name) —> {timestamp:value, timestamp:value, ...}

    View full-size slide

  15. σʔλߏ଄: λΠϜ΢Οϯυ΢
    • ݻఆ෯ͷλΠϜ΢Οϯυ΢ͰϨίʔυΛ۠੾Δ
    • wtimestamp͸λΠϜ΢Οϯυ΢ͷ։࢝࣌ࠁ
    • ͔͠͠΢Οϯυ΢಺ͷσʔλϙΠϯτ਺ͷ࠷େ஋͸ෆఆ
    (name, wtimestamp) —> {timestamp:value,...}

    View full-size slide

  16. σʔλߏ଄: ղ૾౓ΞϥΠϯϝϯτ
    • λΠϜελϯϓΛΞϥΠϯϝϯτͨ͠ޙʹॻ͖ࠐΉ͜ͱ
    ͰɺϨίʔυ಺σʔλϙΠϯτ਺ͷ࠷େ஋Λ੍ݶ͢Δ
    • ࣌ܥྻDB͕αϙʔτ͢Δղ૾౓ͷഒ਺ʹἧ͑Δ

    View full-size slide

  17. ߏ੒ਤ
    ,JOFTJT
    4USFBNT
    write path
    3FEJT
    $MVTUFS
    %ZOBNP%#
    4
    SFBEFS
    read path
    -BNCEB
    flush
    flush
    [2]: ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
    http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud, 2017, AWS Summit Tokyo

    View full-size slide

  18. KVSؒͷσʔλҠಈ: Redis => DynamoDB
    • όονॲཧͰ͸ͳ͘ɺσʔλϙΠϯτॻ͖ࠐΈ࣌ʹඞཁ
    ʹԠͯ͡Ҡಈͤ͞Δ
    • Redis্ͷ֘౰ϝτϦοΫܥྻʹؚ·ΕΔσʔλϙΠϯ
    τ਺͕NݸҎ্͔Ͳ͏͔ͰҠಈ൑ఆ͢Δ

    View full-size slide

  19. KVSؒͷσʔλҠಈ: DynamoDB => S3

    View full-size slide

  20. DiamondΞʔΩςΫνϟͷར఺
    • (1) ࠷దԽ͞Εͨ಺෦DBMSํࣜͱൺ΂ɺॲཧޮ཰ͷ؍
    ఺Ͱແବ͕ଟ͍
    • (2) ֤KVSʹ࣮ͯ૷͞Ε͍ͯΔσʔλߏ଄͕ҟͳΔͨ
    Ίɺ͋ΔಛఆͷKVSʹ࣮૷͞Ε͍ͯΔಛघͳσʔλߏ଄
    Λ࢖͍ͮΒ͍
    • (3) ߏ੒͕ҰݟෳࡶʹΈ͑Δ

    View full-size slide

  21. 5. ·ͱΊͱࠓޙͷ՝୊

    View full-size slide

  22. ·ͱΊ
    • ॻ͖ࠐΈεέʔϧΞ΢τੑɺߴՄ༻ੑɺඅ༻ޮ཰ɺ௿͍
    ӡ༻ੑɺ։ൃ༰қੑΛ࣮ݱ
    • ෳ਺ͷKVSΛ૊Έ߹ΘͤͨϔςϩδχΞεߏ੒ɺϝο
    ηʔδϒϩʔΧʔʹΑΔWALϦϓϨΠɺαʔόϨεΞʔ
    ΩςΫνϟɺ΂͖౳ੑɺσʔλ෯

    View full-size slide