サーバ監視サービス Mackerel を提供するはてなが開発中の、高解像度・長期間のサーバメトリック収集を実現する、新しい時系列データベースを紹介します。具体的には、Amazon ElastiCache、Amazon DynamoDB、Amazon S3 を組み合わせ、Amazon Kinesis Streams と AWS Lambda によりコンポーネント接続した、階層構造のデータストアアーキテクチャの設計と実装を解説します。
࣌ܥྻσʔλϕʔεͱ͍͏֓೦ΛΫϥυͷٕͰ࠶ߏங͢Δͯͳ id:y_uukiAWS Summit Tokyo 2017
View Slide
id:y_uuki / @y_uuk1TSUBOUCHI Yuukihttps://yuuk.io/גࣜձࣾͯͳWebΦϖϨʔγϣϯΤϯδχΞ / γχΞΤϯδχΞMackerelΞʔΩςΫνϟ৽ϓϩδΣΫτϦʔμʔ
writedatapointsKinesis StreamsLambdareaddatapointsElastiCacheDynamoDBLambdaTTL expiredflushwriteWeb App(Golang)S3readreadreadαʔόࢹαʔϏεMackerelʹ͓͚Δ࣍ੈͷ࣌ܥྻσʔλϕʔεΞʔΩςΫνϟͷઃܭͱ࣮
ΰʔϧ•Write-Intensive ApplicationΛAWS্ͰΞʔΩςΫνϟઃܭ͢ΔࣄྫΛհ• ൺֱత୯७ͳσʔλΛอ࣋͢Δ࣌ܥྻσʔλϕʔεΑ͍ϞσϧέʔεͱͳΔͣ• ίετ࠷దԽͷͨΊʹɺࢀরہॴੑΛར༻ͨ͠֊ܕσʔλετΞΞʔΩςΫνϟͷఏҊ
ΰʔϧઃఆͷഎܠ• AWS SummitͳͷͰɺAWSΛ༻͍ͯΑΓྑ͍γεςϜΛߏங͢ΔͨΊͷώϯτΛ͍͑ͨ• ͦͷͨΊʹɺ࣌ܥྻσʔλϕʔεͦͷͷΑΓɺ൚༻తͳΞʔΩςΫνϟઃܭͷͨΊͷٞͷͱͳΔͷΛఏҊ͍ͨ͠
Mackerelhttps://mackerel.io/
αʔόͷϝτϦοΫՄࢹԽ
https://speakerdeck.com/sugiyama88/mackerel-meetup-number-8
ߴͳઍݸͷܥྻͷදࣔ
1ղ૾σʔλΛظؒอ࣋https://mackerel.io/ja/docs/entry/overviewྫ͑1
1ҎԼͷղ૾ͷରԠhttps://mackerel.io/ja/docs/entry/overviewྫ͑15ඵ
͚ͭͮ͠ΔαʔϏεͷεέʔϥϏϦςΟͷ֬อσʔλྔͱI/O͕ܹ૿͢Δػೳͷ࣮ݱ
MackerelΞʔΩςΫνϟ࣌ܥྻσʔλϕʔε.BDLFSFM"QQ 4DBMB"HFOUϝτϦοΫߘread/writedatapoints6TFSάϥϑදࣔ
࣌ܥྻσʔλϕʔεͱ• ࣌ܥྻσʔλΛѻ͏͜ͱʹಛԽͨ͠σʔλϕʔε• αʔόϞχλϦϯάɺIoTͳͲͷ༻్ʹΘΕΔ• ༗໊ͳ࣮Graphite/InfluxDB/OpenTSDBͳͲ• ຯʹΈ͑ͯҙ֎ͱଟ͘ͷ࣮͕ଘࡏ͢Δʢৄࡉޙड़)• ֶज़ݚڀจͷରʹͳΔ
http://blog.yuuk.io/entry/high-performance-graphite
http://blog.yuuk.io/entry/high-performance-graphiteGraphitehttp://graphite.readthedocs.io/en/latest/
ݱঢ়ͷ
Graphiteͷᶃ• ෛՙࢄͷͨΊͷӡ༻ίετ͕ߴ͍• γϟʔυ૿ݮ࣌ʹσʔλ࠶ஔ͕ඞཁ• ࢄ͕Ή͔͍ͣͨ͠ΊεέʔϧΞοϓΛબ• σʔλอ࣋ظؒΛ૿͢ͱۚમίετ͕ܹ૿• NANDϑϥογϡϝϞϦͷͨΊ༰ྔ୯Ձ͕େ͖͍• ࢀরසͷ͍σʔλ͚ͩͳσΟεΫʹஔ͖͍͕ͨϥϯυϩϏϯσʔλϕʔεͷੑ্࣭͍͠
Graphiteͷᶄ• σʔλϩετੑ͕͍• ΞϓϦέʔγϣϯ͔ΒඇಉظͰϝϞϦ্ͷΩϡʔʹॻ͖ࠐΉͨΊɺαʔόμϯ͢Δͱσʔλϩετ• ະདྷͷྖҬΛϑΝΠϧ࡞࣌ʹ֬อ͢ΔͷͰɺσΟεΫ༻ޮ͕͍• ಛʹίϯςφͷΑ͏ͳ͍ࣺͯͷϗετ͕େྔʹ࡞ΒΕΔͱແବ͕૿͑Δ
Graphiteͷᶄ• σʔλϩετੑ͕͍ (durability)• ΞϓϦέʔγϣϯ͔ΒඇಉظͰϝϞϦ্ͷΩϡʔʹॻ͖ࠐΉͨΊɺαʔόμϯ͢Δͱσʔλϩετ• ະདྷʹॻ͖ࠐΉͰ͋Ζ͏ྖҬΛϑΝΠϧ࡞࣌ʹ֬อ͢ΔͷͰɺσΟεΫ༻ޮ͕ѱ͍• ಛʹίϯςφͷΑ͏ͳ͍ࣺͯͷϗετ͕େྔʹ࡞ΒΕΔͱແବ͕ଟ͍ΞʔΩςΫνϟ৽͕ඞཁ
࣌ܥྻσʔλϕʔεͷطଘOSS࣮ͷௐࠪ
Andreas Bader, Oliver Kopp, Michael Falkenthal. “Survey and Comparison of Open Sourcetime Series Databases”. In proceedings of BTW 2017.
https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/pubhtmlOpen Source Time Series DB Comparison
࣌ܥྻσʔλϕʔεͷఆٛ• ҎԼͷੑ࣭Λຬͨ͢DBMSͷ͜ͱ• λΠϜελϯϓɺɺଐੑʢϝτϦοΫ໊ͳͲʣͰߏ͞ΕΔσʔλͷߦΛ֨ೲͰ͖Δ• ࣌ܥྻͱͯ͠άϧʔϓԽ͞ΕͨෳͷߦΛ֨ೲͰ͖Δ• σʔλߦʹରͯ͠ΫΤϦΛൃߦͰ͖Δ• λΠϜελϯϓ·ͨ࣌ؒൣғΛΫΤϦʹؚΊΒΕΔ
࣌ܥྻσʔλϕʔεͷ֓೦• ࣌ܥྻσʔλϕʔεͷఆٛΛຬͨ͢ͷ• ࣌ܥྻσʔλϕʔεͷΑ͋͘Δػೳཁٻ• ࢄ/ΫϥελϦϯά• Function• Rollup Aggregation / λά• σʔλղ૾• ΠϯλϑΣʔε
࣌ܥྻDBͷྨTSDB on DBMS TSDB standaloneOpenTSDB InfluxDBKairosDBBluefloodGraphiteDalmatinerDBBeringei
࣌ܥྻDBͷྨTSDB on DBMS TSDB standaloneOpenTSDB InfluxDBKairosDBBluefloodGraphiteDalmatinerDBBeringeiHBaseCassandraElasticsearchRiak Core…
طଘͷ࣌ܥྻDBͷσϝϦοτTSDB on DBMS TSDB standaloneࢄγεςϜͱͯ͠ͷӡ༻ίετ͕ߴ͍ࢄγεςϜͱͯ͠ͷ৴པੑΛଌΔͨΊͷ࣮͕গͳ͍
طଘͷ࣌ܥྻDBͷσϝϦοτTSDB on DBMS TSDB standaloneRollup AggregationʹରԠ͍ͯ͠ͳ͍ΠϯλϑΣʔε͕େ͖͘มΘΔͷͰมߋίετMackerelͷϫʔΫϩʔυͰΠϯϑϥίετΛ͑ΒΕΔ͔Θ͔Βͳ͍
http://developer.hatenastaff.com/entry/2015/12/25/140233
ٕज़બͷϙΠϯτ• ಛʹσʔλϕʔε҆ఆ͢Δ͔Ͳ͏͔͕ॏཁ• ͋·ΓΘΕ͍ͯͳ͍OSSͷ࠾༻ආ͚͍ͨ• σʔλϕʔεܥͷOSSݕূɾӡ༻ίετ͕ඇৗʹେ͖͍• طଘͷ࣌ܥྻσʔλϕʔεಋೖ·Ͱָ͕ͳΘΓʹɺॊೈੑ͕͍• Πϯϑϥίετ͕ଟগ૿͑ͨͱͯ͠ɺࠓޙͷػೳ֦ॆڝ߹༏ҐੑΛ֬อ͍ͨ͠
AWSϚωʔδυαʔϏεͷ࠾༻
AWSϚωʔδυαʔϏε্ʹಠࣗͷ࣌ܥྻDBΞϓϦέʔγϣϯΛ࣮
ͳͥAWS͔• Amazon DynamoDBɺAmazon S3ɺAmazon Kinesisͱ͍ͬͨϑϧϚωʔδυͳσʔλετΞ͕͋Δ• ੑೳΛ͓ۚʹࢉͯ͠ܭࢉ͍͢͠• GCPɺAzureͳͲ࣮͕ࣾͳ͍• MackerelͷࠓޙΓ͍ͨ͜ͱΛؑΈͯɺ͋ΔఔͷϩοΫΠϯΛड͚ೖΕΔ͜ͱʹܾఆ
ͳͥಠ࣮͔ࣗ• MackerelͷཁٻΛຬ࣮͕ͨ͢ͳ͍• ͘͠ཁٻΛຬ͔ͨ͢Ͳ͏͔Λݕূ͕ऴΘΔ·Ͱʹ͕͔͔࣌ؒΔ• AWSϚωʔδυαʔϏε࠾༻ʹΑΓσʔλϕʔεͷݕূ/ӡ༻ίετΛݮͰ͖ΔͷͰɺͦͷ͔ΘΓͷ։ൃΛ֬อͰ͖Δ
ݱঢ়ͷ՝ (࠶ܝ)• ෛՙࢄͷӡ༻ίετͷݮ• σʔλอ࣋ظؒͷԆ• σʔλϩετੑͷ্• σΟεΫ༻ޮͷ্
ෛՙࢄͷӡ༻ίετᶃ• Amazon DynamoDB/Amazon S3ͳͲͷϑϧϚωʔδυαʔϏεͷར༻ʹΑΓɺӡ༻ίετΛݮ• γϟʔσΟϯάͷͨΊͷΩʔͷઃܭΛؒҧ͑ͳ͚Εɺ͓ۚΛ͚ͬͨͩI/Oεϧʔϓοτ͕εέʔϧ• αʔόՃɾݮ࡞ۀ͕ෆཁ
ෛՙࢄͷӡ༻ίετᶄ• ͱ͜Ζ͕DynamoDBͷσΟεΫI/Oίετ͕ߴ͍• ࠓͷMackerelͷI/OΛͳ͠ͰDynamoDBʹ͚ΔͱΠϯϑϥίετ͕ܹ૿͢Δ• ݸʑͷσʔλϙΠϯτΛผʑʹॻ͖ࠐΉͷͰͳ͘·ͱΊͯॻ͖ࠐΈɺI/OίετΛԼ͛Δඞཁ͕͋Δ• ࠷ऴతʹɺલஈʹAmazon ElastiCache(Redis)Λஔ͠ɺόοϑΝͱΈͨͯͯར༻͢Δ͜ͱʹͳͬͨ
σʔλอ࣋ظؒͷԆᶃ• DynamoDBͷσΟεΫ༰ྔ୯Ձײ֮తʹ͍҆• ͔͠͠ɺS3ͷίετελϯμʔυετϨʔδͰDynamoDBͷ1/10ఔ• ͜͜ͰίετΛු͔͓͚ͤͯɺଞͷػೳ֦ॆʹίετΛ͋ͯΒΕΔͨΊɺͳΔ͘S3Λ͍͍ͨ• S3ʹஔ͍͓͚ͯɺσʔλղੳܥͷAWSαʔϏεͱ࿈ܞ͍͢͠ͱ͍͏ͷ͋Δ
σʔλอ࣋ظؒͷԆᶄ• Mackerelͷ߹ɺݹ͍ߴղ૾σʔλ΄ͱΜͲࢀর͞Εͳ͍ (ࢀরہॴੑ)• ࢀরճ͕গͳ͍σʔλͷදࣔɺଟগϨεϙϯε͕ͯ͘Α͍• ͜ͷੑ࣭Λར༻͠ɺϗοτσʔλΛDynamoDBɺίʔϧυσʔλΛS3ʹஔ͢Δ͜ͱͰίετ࠷దԽ͢Δ
͜ͷ࣌Ͱ֊ܕσʔλετΞΞʔΩςΫνϟʹͳΔ͜ͱ͕ܾఆ
σʔλϩετੑͷ্• σΟεΫॻ͖ࠐΈ͢ΔϝοηʔδΩϡʔΛ࠾༻• OSSͰ͋ΕKafkaRabbitMQͳͲ• AWSͰɺAmazon Kinesis Streams• ۙͷ24࣌ؒͷϨίʔυΛσΟεΫʹอଘ• AWS LambdaʹϨίʔυΛ͢͜ͱ͕؆୯
σΟεΫ༻ޮͷ্• σʔλߏ(ޙड़)͕શ͘ҟͳΔͨΊɺະདྷͷྖҬΛ֬อ͢Δ͜ͱ͠ͳ͍ͷͰɺແବ͕ͳ͍• ଞͷOSSͷ࣌ܥྻDB͕࣮͍ͯ͠ΔΑ͏ͳѹॖอଘΒͳ͍• S3Λར༻͢Δ͜ͱͰσΟεΫ༻ίετΛԼ͍͛ͯΔͷͰѹॖอଘΒͳͯ͘Α͍ͱஅ
ݱঢ়ͷ՝ͱղܾ ·ͱΊ✓ ෛՙࢄͷӡ༻ίετͷݮ➡ ElastiCacheͱDynamoDBʹΑΓεέʔϥϏϦςΟͱӡ༻ޮΛ֬อ✓ σʔλอ࣋ظؒͷԆ➡ S3ʹΑΓ༰ྔ୯Ձͷ͍ετϨʔδ
ݱঢ়ͷ՝ͱղܾ ·ͱΊ✓ σʔλϩετੑͷ্➡ Kinesis StreamsΛલஈʹஔ✓ σΟεΫ༻ޮͷ্➡ σΟεΫ༻ޮΛҙࣝͨ͠σʔλߏͱS3ͷར༻
࣌ܥྻσʔλΛಡΈग़͢෦ΛMicroservicesͱͯ͠։ൃ
Microservicesͷ։ൃ• GraphiteޓͷΠϯλϑΣʔεΛ࣮͠ɺΠϯλϑΣʔεมߋίετΛ࠷খԽ• MackerelͰຊମΞϓϦέʔγϣϯScalaɺαςϥΠτͷΞϓϦέʔγϣϯGoͳͷͰɺࠓճGoΛ࠾༻• ੑೳ্ͷͨΊʹσʔλετΞฒߦͯ͠ΫΤϦൃߦ͢Δ͜ͱ͕༧Ͱ͖ͨͷͰɺܰྔεϨουΛͭGoΑ͍બ
৽࣌ܥྻσʔλϕʔεΞʔΩςΫνϟͷઃܭ
৽࣌ܥྻDBΞʔΩςΫνϟ֓ཁ• ۚમίετ࠷దԽͷͨΊʹࢀরہॴੑΛར༻• ϗοτσʔλΛDynamoDBɺίʔϧυσʔλΛS3ʹஔ• writeόοϑΝͱͯ͠Amazon ElastiCache(Redis)Λར༻• DynamoDBͷTTLػೳʹΑΓγʔϜϨεʹσʔλҠಈ• ॻ͖ࠐΈΛAmazon Kinesis StreamsͰड৴͠ɺσʔλϩετੑͱোੑͷ্• GraphiteޓΠϯλϑΣʔεͰ֤ετϨʔδ͔Βσʔλऔಘ͢ΔWebΞϓϦέʔγϣϯΛ࣮
σʔλετΞؒͷγʔϜϨεͳσʔλҠಈ• ෳͷσʔλετΞΛΈ߹ΘͤΔ߹ɺσʔλΛγϯϓϧͳख๏ͰҠಈͤ͞Δ͜ͱ͕͍͠• 20172݄ʹDynamoDB͕TTLαϙʔτ• ΞΠςϜ୯ҐͰTTLΛઃఆՄೳ• DynamoDB TriggersʹΑΓɺTTLΠϕϯτΛܖػʹLambdaΛىಈ• DynamoDB͔ΒS3ͷݹ͍σʔλͷҠಈ͕༰қʹ
Mackerel ࣌ܥྻσʔλ֓؍࣌ܥྻσʔλϕʔε.BDLFSFM"QQ 4DBMB"HFOUϝτϦοΫߘread/writedatapoints6TFSάϥϑදࣔ
writedatapointsKinesis StreamsLambdareaddatapointsElastiCacheDynamoDBLambdaTTL expiredflushwriteWeb App(Golang)S3readreadread
writedatapointsKinesis StreamsLambdareaddatapointsElastiCacheDynamoDBLambdaTTL expiredflushwriteWeb App(Golang)S3readreadread֊ܕσʔλετΞBufferHot DataCold Data
writedatapointsKinesis StreamsLambdareaddatapointsElastiCacheDynamoDBLambdaTTL expiredflushwriteWeb App(Golang)S3readreadreadίϯϙʔωϯτؒͷଓ
writedatapointsKinesis StreamsLambdareaddatapointsElastiCacheDynamoDBLambdaTTL expiredflushwriteWeb App(Golang)S3readreadread3छྨͷσʔλϕʔε͔ΒσʔλΛಡΈग़͢WebΞϓϦέʔγϣϯ
writedatapointsKinesis StreamsLambdareaddatapointsElastiCacheDynamoDBLambdaTTL expiredflushwriteWeb App(Golang)S3readreadreadDynamoDB TTLʹΑΔγʔϜϨεͳσʔλҠಈ
৽࣌ܥྻσʔλϕʔεΞʔΩςΫνϟͷ࣮
ΞʔΩςΫνϟͷ࣮• σʔλߏ• RedisͰͷόοϑΝϦϯά• GraphiteޓΞϓϦέʔγϣϯͷI/OଟॏԽ※࣮࣌ܥྻσʔλϕʔεݻ༗ͷʹͳΔ
σʔλߏϝτϦοΫ໊ [ ɹ… ]: {timestamp, value}Key Value※࣮ࡍʹɺ֤ετϨʔδ͝ͱʹ͕͋Δ
σʔλߏϝτϦοΫ໊ [ ɹ… ]Key ValueάϥϑදࣔͰ࣌ܥྻʹฒΜͩσʔλϙΠϯτΛҰ੪ʹಡΈऔΔ
RedisͰͷόοϑΝϦϯά• cronͷΑ͏ͳόονॲཧʹΑΔϑϥογϡͨ͘͠ͳ͍• Redis্ͷಉҰϝτϦοΫͷσʔλϙΠϯτ͕ҰఆҎ্ʹͳΔͱDynamoDBͷରԠ͢ΔϨίʔυʹϑϥογϡॻ͖ࠐΈ• ʮ࣌ܥྻʯσʔλͳͷͰɺಉ࣌ʹಉ͡ܥྻͷσʔλϙΠϯτ౸ண͠ͳ͍ͨΊɺόοϑΝͯͭ͠ඞཁ͕͋Δ• Lambda Function্ͰσʔλϙΠϯτ͕౸ண͢ΔͨͼʹɺϙΠϯτ͕ҰఆҎ্͔Ͳ͏͔ఆ͠ɺDynamoDBϑϥογϡ
RedisͰͷόοϑΝϦϯάϝτϦοΫ໊ [ ɹ… ]RedisDynamoDBϝτϦοΫ໊ [ ɹ… ]Flush>= Nݸ
GraphiteޓΞϓϦέʔγϣϯͷI/OଟॏԽ• I/OଟॏԽʹΑΔಡΈग़͠࠷దԽੑೳ؍Ͱඞਢ• 1ͭͷϝτϦοΫॲཧʹରͯ͠goroutineΛׂΓͯ• ElastiCache/DynamoDB/S3ͷ֤σʔλετΞͷॲཧʹରͯ͠goroutineΛׂΓͯ• ֤σʔλετΞͷ1Ϩίʔυʹରͯ͠goroutineׂΓͯ• ͨͩ͠DynamoDBʹ͍ͭͯBatchGetItem APIΛར༻
GraphiteޓΞϓϦέʔγϣϯͷI/OଟॏԽElastiCacheDynamoDBS3............ϝτϦοΫgoroutineϨίʔυ
ίʔυωʔϜDiamondݱࡏ։ൃத
·ͱΊ• Write-Intensive ApplicationͷAWSʹ͓͚ΔઃܭࣄྫΛհ• Amazon ElastiCacheɺAmazon DynamoDBɺAmazon S3ΛΈ߹Θͤͨ֊ܕσʔλετΞΞʔΩςΫνϟͷఏҊ• Amazon Kinesis StreamsɺAWS LambdaΛކͱͯ͠ίϯϙʔωϯτؒΛଓ• Amazon DynamoDB TTLʹΑΔγʔϜϨεͳσʔλஔ
ҰൠԽͷͨΊͷٞ• ඞͣ3֊ʹ͢Δඞཁͳ͍• σΟεΫ༻͕গͳ͚Ε ElastiCache + DynamoDB• I/O͕গͳ͚Ε DynamoDB + S3• readϨΠςϯγΛؾʹ͠ͳ͚ΕɺRedis + S3• 3֊͋Εଟ͘ͷঢ়گͰίετΛ࠷దԽͰ͖Δ
DiamonDBhttps://github.com/yuuki/diamondb※ݸਓ։ൃͷOSSϓϩμΫτͰ͋ΓMackerelͰ։ൃதͷͷͱίʔυϕʔε͕ҟͳΓ·͢
Mackerelలࣔϒʔεͥͻ͓ӽ͠Լ͍͞
yuukit
shumpei0111
loic425
mh4gf
yatasunshine
star_zero
taehwandev
daipresents
skanehira
pospome
akarin
amixedcolor
lnit
jmmastey
smashingmag
mthomps
mongodb
addyosmani
vanstee
3n
jasonvnalue
thoeni
destraynor
roundedbygravity
mza