サーバレス時代におけるヘテロジニアス時系列データベースアーキテクチャ / HeteroTSDB

Transcript

1. 1.

   αʔόϨε࣌୅ʹ͓͚Δ ϔςϩδχΞε࣌ܥྻσʔλϕʔε ΞʔΩςΫνϟ ୈ2ճ Web System Architecture(WSA) ݚڀձ@෱Ԭ ͸ͯͳ id:y_uuki

   / @y_uuk1
2. 2.

   id:y_uuki / @y_uuk1 Yuuki TSUBOUCHI https://yuuk.io/ ͸ͯͳͷ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ WSAݚ ෭ࠪ

3. 3.

   ΋͘͡ 1. എܠͱ໨త 2. ΞʔΩςΫνϟ 3. ࣮૷ 4. ߟ࡯ 5.

   ·ͱΊͱࠓޙͷ՝୊
4. 4.

   1. എܠͱ໨త

5. 5.

   ߴ͍ੑೳΛ΋ͭ࣌ܥྻDBͷඞཁੑ • ΢ΣϒγεςϜ͕ෳࡶԽͨ݁͠ՌɺΑΓΑ͍ϞχλϦϯ άٕज़͕ඞཁͱ͞Ε͍ͯΔ • ΑΓΑ͍ϞχλϦϯάͷͨΊʹɺ࣌ܥྻσʔλΛߴස౓ ͔ͭେྔʹऩू͠ɺ௕ظؒอଘ͍ͨ͠

6. 6.

   ࣌ܥྻDBͷ࣮૷ํࣜ - ಺෦DBMSํࣜ • ࣌ܥྻσʔλϕʔεʹ࠷దԽ͞ΕͨΤϯδϯΛ࣮૷ • ੑೳͱσʔλอଘޮ཰͸Α͍܏޲͕͋Δ • ҰํͰɺ෼ࢄγεςϜͱͯ͠ͷ৴པੑʹ͍ͭͯɺ࣮ݧஈ ֊Ͱ͋Δ͜ͱ΋ଟ͍

   • GraphiteɺInfluxDBɺPrometheusͳͲ [4]: Jensen, Søren Kejser, et al. "Time Series Management Systems: A Survey." IEEE Transactions on Knowledge and Data Engineering, vol.29, no.11, 2017, pp. 2581-2600. [5]: B. Mitschang et al. "Survey and Comparison of Open Source Time Series Databases.", In Proceedings of Database Systems for Business, Technology, and Web, 2017
7. 7.

   ࣌ܥྻDBͷ࣮૷ํࣜ - ֎෦DBMSํࣜ • ൚༻DBMS্ͷΞϓϦέʔγϣϯͱ࣮ͯ͠૷ • ෼ࢄγεςϜͱͯ͠ͷ৴པੑʹؔΘΔ࣮૷͕੒ख़͍ͯ͠ Δ܏޲͕͋Δ • ੑೳͱޮ཰͸಺෦DBMSํࣜʹྼΔ܏޲͕͋Δ

   • OpenTSDB(HBase)ɺKairosDB(Cassandra)ͳͲ [4]: Jensen, Søren Kejser, et al. "Time Series Management Systems: A Survey." IEEE Transactions on Knowledge and Data Engineering, vol.29, no.11, 2017, pp. 2581-2600. [5]: B. Mitschang et al. "Survey and Comparison of Open Source Time Series Databases.", In Proceedings of Database Systems for Business, Technology, and Web, 2017
8. 8.

   ࣌ܥྻDBͷ՝୊ • ಺෦DBMSํࣜͷߴՄ༻ੑɺεέʔϥϏϦςΟͷ੒ख़౓ • ಺෦DBMSํࣜͷߴՄ༻ੑ࣮૷ͷ೉͠͞ • ֎෦DBMSํࣜͷੑೳͱޮ཰ͷ௿͞ ߴ৴པੑΛٻΊΒΕΔϢʔεέʔεΛ૝ఆ͠ ֎෦DBMSํࣜΛ࠾༻ͭͭ͠ɺੑೳͱޮ཰ͷΑ͍Ξʔ ΩςΫνϟΛఏҊ͢Δ

9. 9.

   ϔςϩδχΞε࣌ܥྻDBͷఏҊ • ಛੑͷҟͳΔෳ਺ͷ֎෦DBMSΛ૊Έ߹Θͤɺಁաతʹ ΞΫηεՄೳͳ࣌ܥྻDBΛఏҊ͢Δ • DiamondΞʔΩςΫνϟͱݺͿ͜ͱͱ͢Δ

10. 10.

    2. ΞʔΩςΫνϟ

11. 11.

    ΞʔΩςΫνϟͷཁ݅ • (a) ࣌ܥྻσʔλʹର͢Δߴ͍ॻ͖ࠐΈεέʔϧΞ΢τੑ • (b) ߴՄ༻ੑ • (c) ॻ͖ࠐΈI/Oͱσʔλอ࣋ͷͦΕͧΕʹ͍ͭͯɺߴ͍ίϯ

    ϐϡʔςΟϯάϦιʔεޮ཰ • (d) ϦιʔεͷϓϩϏδϣχϯά͓Αͼނো࣌ͷ෮چͷ༰қ͞ • (e) ݕূ͓Αͼ։ൃͷ༰қ͞
12. 12.

    ಈ࡞ϑϩʔ ϝοηʔδ
    ϒϩʔΧʔ write path ΠϯϝϞϦ
    ,74 Φϯ44%
    ,74 Φϯ)%%
    

    ,74 SFBEFS
    OPEF read path XSJUFS
    OPEF flush flush
13. 13.

    σʔλߏ଄ • KVSؒͰͷσʔλ੔߹ੑΛ͍͔ʹ୲อ͢Δ͔ • KVS্ͰҎԼͷ2఺Λୡ੒͠ɺղܾ͢Δ • (1) ࣌ܥྻσʔλͷ΂͖౳ॻ͖ࠐΈ • (2)

    Valueʹؚ·ΕΔσʔλϙΠϯτ਺ͷ࠷େ஋Λ੍ޚ • (1) ͸ϋογϡϚοϓʹΑΓ࣮ݱ • (2) ͸ղ૾౓ΞϥΠϯϝϯτͱλΠϜ΢Οϯυ΢ʹΑΓ࣮ݱ
14. 14.

    σʔλߏ଄: ϋογϡϚοϓ • timestampΛΩʔɺvalueΛ஋ͱͨ͠ϋογϡϚοϓ • ϋογϡϚοϓʹର͢Δ্ॻ͖ʹΑΔ΂͖౳ॻ͖ࠐΈ͕ Մೳ • ௥هͷͨͼʹɺValueαΠζ͕૿େ͠ɺιϑτϦϛοτ ·ͨ͸ϋʔυϦϛοτΛ௒͑ΔՄೳੑ͕͋Δ

    (name) —> {timestamp:value, timestamp:value, ...}
15. 15.

    σʔλߏ଄: λΠϜ΢Οϯυ΢ • ݻఆ෯ͷλΠϜ΢Οϯυ΢ͰϨίʔυΛ۠੾Δ • wtimestamp͸λΠϜ΢Οϯυ΢ͷ։࢝࣌ࠁ • ͔͠͠΢Οϯυ΢಺ͷσʔλϙΠϯτ਺ͷ࠷େ஋͸ෆఆ (name, wtimestamp)

    —> {timestamp:value,...}
16. 16.

    σʔλߏ଄: ղ૾౓ΞϥΠϯϝϯτ • λΠϜελϯϓΛΞϥΠϯϝϯτͨ͠ޙʹॻ͖ࠐΉ͜ͱ ͰɺϨίʔυ಺σʔλϙΠϯτ਺ͷ࠷େ஋Λ੍ݶ͢Δ • ࣌ܥྻDB͕αϙʔτ͢Δղ૾౓ͷഒ਺ʹἧ͑Δ

17. 17.

    3.࣮૷

18. 18.

    ߏ੒ਤ ,JOFTJT
    4USFBNT write path 3FEJT
    $MVTUFS %ZOBNP%# 4 SFBEFS

    read path -BNCEB flush flush [2]: ࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ http://blog.yuuk.io/entry/the-rebuild-of-tsdb-on-cloud, 2017, AWS Summit Tokyo
19. 19.

    KVSؒͷσʔλҠಈ: Redis => DynamoDB • όονॲཧͰ͸ͳ͘ɺσʔλϙΠϯτॻ͖ࠐΈ࣌ʹඞཁ ʹԠͯ͡Ҡಈͤ͞Δ • Redis্ͷ֘౰ϝτϦοΫܥྻʹؚ·ΕΔσʔλϙΠϯ τ਺͕NݸҎ্͔Ͳ͏͔ͰҠಈ൑ఆ͢Δ

20. 20.

    KVSؒͷσʔλҠಈ: DynamoDB => S3

21. 21.

    4. ߟ࡯

22. 22.

    DiamondΞʔΩςΫνϟͷར఺ • (1) ࠷దԽ͞Εͨ಺෦DBMSํࣜͱൺ΂ɺॲཧޮ཰ͷ؍ ఺Ͱແବ͕ଟ͍ • (2) ֤KVSʹ࣮ͯ૷͞Ε͍ͯΔσʔλߏ଄͕ҟͳΔͨ Ίɺ͋ΔಛఆͷKVSʹ࣮૷͞Ε͍ͯΔಛघͳσʔλߏ଄ Λ࢖͍ͮΒ͍

    • (3) ߏ੒͕ҰݟෳࡶʹΈ͑Δ
23. 23.

    5. ·ͱΊͱࠓޙͷ՝୊

24. 24.

    ·ͱΊ • ॻ͖ࠐΈεέʔϧΞ΢τੑɺߴՄ༻ੑɺඅ༻ޮ཰ɺ௿͍ ӡ༻ੑɺ։ൃ༰қੑΛ࣮ݱ • ෳ਺ͷKVSΛ૊Έ߹ΘͤͨϔςϩδχΞεߏ੒ɺϝο ηʔδϒϩʔΧʔʹΑΔWALϦϓϨΠɺαʔόϨεΞʔ ΩςΫνϟɺ΂͖౳ੑɺσʔλ෯