時系列データベースという概念をクラウドの技で再構築する / The rebuild of time-series database on AWS

時系列データベースという概念をクラウドの技で再構築する / The rebuild of time-series database on AWS

サーバ監視サービス Mackerel を提供するはてなが開発中の、高解像度・長期間のサーバメトリック収集を実現する、新しい時系列データベースを紹介します。具体的には、Amazon ElastiCache、Amazon DynamoDB、Amazon S3 を組み合わせ、Amazon Kinesis Streams と AWS Lambda によりコンポーネント接続した、階層構造のデータストアアーキテクチャの設計と実装を解説します。

A658ec7f1badf73819dfa501165016c1?s=128

Yuuki Tsubouchi (yuuk1)

June 01, 2017
Tweet

Transcript

  1. 3.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read αʔό؂ࢹαʔϏεMackerelʹ͓͚Δ ࣍ੈ୅ͷ࣌ܥྻσʔλϕʔεΞʔΩςΫνϟ ͷઃܭͱ࣮૷
  2. 18.

    Graphiteͷ໰୊఺ᶃ • ෛՙ෼ࢄͷͨΊͷӡ༻ίετ͕ߴ͍ • γϟʔυ૿ݮ࣌ʹσʔλ࠶഑ஔ͕ඞཁ • ෼ࢄ͕Ή͔͍ͣͨ͠ΊεέʔϧΞοϓΛબ୒ • σʔλอ࣋ظؒΛ૿΍͢ͱۚમίετ͕ܹ૿ •

    NANDϑϥογϡϝϞϦͷͨΊ༰ྔ୯Ձ͕େ͖͍ • ࢀরස౓ͷ௿͍σʔλ͚ͩ௿଎ͳσΟεΫʹஔ͖͍͕ͨ ϥ΢ϯυϩϏϯσʔλϕʔεͷੑ্࣭೉͍͠
  3. 22.

    Andreas Bader, Oliver Kopp, Michael Falkenthal. “Survey and Comparison of

    Open Source time Series Databases”. In proceedings of BTW 2017.
  4. 27.

    ࣌ܥྻDBͷ෼ྨ TSDB on DBMS TSDB standalone OpenTSDB InfluxDB KairosDB Blueflood

    Graphite DalmatinerDB Beringei HBase Cassandra Elasticsearch Riak Core …
  5. 34.

    ͳͥAWS͔ • Amazon DynamoDBɺAmazon S3ɺAmazon Kinesis ͱ͍ͬͨϑϧϚωʔδυͳσʔλετΞ͕͋Δ • ੑೳΛ͓ۚʹ׵ࢉͯ͠ܭࢉ͠΍͍͢ •

    GCPɺAzureͳͲ͸ࣾ಺࣮੷͕ͳ͍ • Mackerelͷ੒௕଎౓΍ࠓޙ΍Γ͍ͨ͜ͱΛؑΈͯɺ͋ Δఔ౓ͷϩοΫΠϯΛड͚ೖΕΔ͜ͱʹܾఆ
  6. 49.

    ৽࣌ܥྻDBΞʔΩςΫνϟ֓ཁ • ۚમίετ࠷దԽͷͨΊʹࢀরہॴੑΛར༻ • ϗοτσʔλΛDynamoDBɺίʔϧυσʔλΛS3ʹ഑ஔ • writeόοϑΝͱͯ͠Amazon ElastiCache(Redis)Λར༻ • DynamoDBͷTTLػೳʹΑΓγʔϜϨεʹσʔλҠಈ

    • ॻ͖ࠐΈΛAmazon Kinesis StreamsͰड৴͠ɺσʔλϩ ετ଱ੑͱ଱ো֐ੑͷ޲্ • Graphiteޓ׵ΠϯλϑΣʔεͰ֤ετϨʔδ͔Βσʔλऔ ಘ͢ΔWebΞϓϦέʔγϣϯΛ࣮૷
  7. 52.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read
  8. 53.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read
  9. 54.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read ֊૚ܕσʔλετΞ Buffer Hot Data Cold Data
  10. 55.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read ίϯϙʔωϯτؒ ͷ઀ଓ
  11. 56.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read 3छྨͷσʔλϕʔε͔Β σʔλΛಡΈग़͢ WebΞϓϦέʔγϣϯ
  12. 57.

    write datapoints Kinesis Streams Lambda read datapoints ElastiCache DynamoDB Lambda

    TTL expired flush write Web App (Golang) S3 read read read DynamoDB TTLʹΑΔ γʔϜϨεͳσʔλҠಈ
  13. 60.

    σʔλߏ଄ ϝτϦοΫ໊ [ ɹ… ] : {timestamp, value} Key Value

    ※࣮ࡍʹ͸ɺ֤ετϨʔδ͝ͱʹ޻෉͕͋Δ
  14. 67.

    ·ͱΊ • Write-Intensive ApplicationͷAWSʹ͓͚ΔઃܭࣄྫΛ঺հ • Amazon ElastiCacheɺAmazon DynamoDBɺAmazon S3 Λ૊Έ߹Θͤͨ֊૚ܕσʔλετΞΞʔΩςΫνϟͷఏҊ

    • Amazon Kinesis StreamsɺAWS LambdaΛކͱͯ͠ίϯ ϙʔωϯτؒΛ઀ଓ • Amazon DynamoDB TTLʹΑΔγʔϜϨεͳσʔλ഑ஔ
  15. 68.

    ҰൠԽͷͨΊͷٞ࿦ • ඞͣ3֊૚ʹ͢Δඞཁ͸ͳ͍ • σΟεΫ࢖༻͕গͳ͚Ε͹ ElastiCache + DynamoDB • I/O͕গͳ͚Ε͹

    DynamoDB + S3 • readϨΠςϯγΛؾʹ͠ͳ͚Ε͹ɺRedis + S3 • 3֊૚͋Ε͹ଟ͘ͷঢ়گͰίετΛ࠷దԽͰ͖Δ