Upgrade to Pro — share decks privately, control downloads, hide ads and more …

時系列データベースという概念をクラウドの技で再構築する / The rebuild of time-series database on AWS

時系列データベースという概念をクラウドの技で再構築する / The rebuild of time-series database on AWS

サーバ監視サービス Mackerel を提供するはてなが開発中の、高解像度・長期間のサーバメトリック収集を実現する、新しい時系列データベースを紹介します。具体的には、Amazon ElastiCache、Amazon DynamoDB、Amazon S3 を組み合わせ、Amazon Kinesis Streams と AWS Lambda によりコンポーネント接続した、階層構造のデータストアアーキテクチャの設計と実装を解説します。

Yuuki Tsubouchi (yuuk1)

June 01, 2017
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Programming

Transcript

  1. ࣌ܥྻσʔλϕʔεͱ͍͏֓೦
    ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
    ͸ͯͳ id:y_uuki
    AWS Summit Tokyo 2017

    View full-size slide

  2. id:y_uuki / @y_uuk1
    TSUBOUCHI Yuuki
    https://yuuk.io/
    גࣜձࣾ͸ͯͳ
    WebΦϖϨʔγϣϯΤϯδχΞ / γχΞΤϯδχΞ
    MackerelΞʔΩςΫνϟ࡮৽ϓϩδΣΫτϦʔμʔ

    View full-size slide

  3. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    αʔό؂ࢹαʔϏεMackerelʹ͓͚Δ
    ࣍ੈ୅ͷ࣌ܥྻσʔλϕʔεΞʔΩςΫνϟ
    ͷઃܭͱ࣮૷

    View full-size slide

  4. ΰʔϧ
    •Write-Intensive ApplicationΛAWS্ͰΞʔΩ
    ςΫνϟઃܭ͢ΔࣄྫΛ঺հ
    • ൺֱత୯७ͳσʔλΛอ࣋͢Δ࣌ܥྻσʔλϕʔε
    ͸Α͍ϞσϧέʔεͱͳΔ͸ͣ
    • ίετ࠷దԽͷͨΊʹɺࢀরہॴੑΛར༻ͨ͠֊૚ܕ
    σʔλετΞΞʔΩςΫνϟͷఏҊ

    View full-size slide

  5. ΰʔϧઃఆͷഎܠ
    • AWS SummitͳͷͰɺAWSΛ༻͍ͯΑΓྑ͍γεςϜ
    Λߏங͢ΔͨΊͷώϯτΛ఻͍͑ͨ
    • ͦͷͨΊʹɺ࣌ܥྻσʔλϕʔεͦͷ΋ͷΑΓ΋ɺ൚
    ༻తͳΞʔΩςΫνϟઃܭͷͨΊͷٞ࿦ͷ౔୆ͱͳΔ
    ΋ͷΛఏҊ͍ͨ͠

    View full-size slide

  6. Mackerel
    https://mackerel.io/

    View full-size slide

  7. αʔόͷϝτϦοΫՄ
    ࢹԽ

    View full-size slide

  8. https://speakerdeck.com/sugiyama88/mackerel-meetup-number-8

    View full-size slide

  9. ߴ଎ͳ਺ઍݸͷܥྻͷදࣔ

    View full-size slide

  10. 1෼ղ૾౓σʔλΛ௕ظؒอ࣋
    https://mackerel.io/ja/docs/entry/overview
    ྫ͑͹1೥

    View full-size slide

  11. 1෼ҎԼͷղ૾౓ͷରԠ
    https://mackerel.io/ja/docs/entry/overview
    ྫ͑͹15ඵ

    View full-size slide

  12. ੒௕͚ͭͮ͠ΔαʔϏεͷεέʔ
    ϥϏϦςΟͷ֬อ
    σʔλྔͱI/O͕ܹ૿͢Δػೳͷ
    ࣮ݱ

    View full-size slide

  13. MackerelΞʔΩςΫνϟ
    ࣌ܥྻ
    σʔλϕʔε
    .BDLFSFM
    "QQ 4DBMB

    "HFOU
    ϝτϦοΫ
    ౤ߘ
    read/write
    datapoints
    6TFS
    άϥϑදࣔ

    View full-size slide

  14. ࣌ܥྻσʔλϕʔεͱ͸
    • ࣌ܥྻσʔλΛѻ͏͜ͱʹಛԽͨ͠σʔλϕʔε
    • αʔόϞχλϦϯάɺIoTͳͲͷ༻్ʹ࢖ΘΕΔ
    • ༗໊ͳ࣮૷͸Graphite/InfluxDB/OpenTSDBͳͲ
    • ஍ຯʹΈ͑ͯҙ֎ͱଟ͘ͷ࣮૷͕ଘࡏ͢Δʢৄࡉޙड़)
    • ֶज़ݚڀ࿦จͷର৅ʹ΋ͳΔ෼໺

    View full-size slide

  15. http://blog.yuuk.io/entry/high-performance-graphite

    View full-size slide

  16. http://blog.yuuk.io/entry/high-performance-graphite
    Graphite
    http://graphite.readthedocs.io/en/latest/

    View full-size slide

  17. ݱঢ়ͷ໰୊఺

    View full-size slide

  18. Graphiteͷ໰୊఺ᶃ
    • ෛՙ෼ࢄͷͨΊͷӡ༻ίετ͕ߴ͍
    • γϟʔυ૿ݮ࣌ʹσʔλ࠶഑ஔ͕ඞཁ
    • ෼ࢄ͕Ή͔͍ͣͨ͠ΊεέʔϧΞοϓΛબ୒
    • σʔλอ࣋ظؒΛ૿΍͢ͱۚમίετ͕ܹ૿
    • NANDϑϥογϡϝϞϦͷͨΊ༰ྔ୯Ձ͕େ͖͍
    • ࢀরස౓ͷ௿͍σʔλ͚ͩ௿଎ͳσΟεΫʹஔ͖͍͕ͨ
    ϥ΢ϯυϩϏϯσʔλϕʔεͷੑ্࣭೉͍͠

    View full-size slide

  19. Graphiteͷ໰୊఺ᶄ
    • σʔλϩετ଱ੑ͕௿͍
    • ΞϓϦέʔγϣϯ͔Β͸ඇಉظͰϝϞϦ্ͷΩϡʔʹ
    ॻ͖ࠐΉͨΊɺαʔόμ΢ϯ͢Δͱσʔλϩετ
    • ະདྷͷྖҬΛϑΝΠϧ࡞੒࣌ʹ֬อ͢ΔͷͰɺσΟεΫ
    ࢖༻ޮ཰͕௿͍
    • ಛʹίϯςφͷΑ͏ͳ࢖͍ࣺͯͷϗετ͕େྔʹ࡞Β
    ΕΔͱແବ͕૿͑Δ

    View full-size slide

  20. Graphiteͷ໰୊఺ᶄ
    • σʔλϩετ଱ੑ͕௿͍ (௿durability)
    • ΞϓϦέʔγϣϯ͔Β͸ඇಉظͰϝϞϦ্ͷΩϡʔʹ
    ॻ͖ࠐΉͨΊɺαʔόμ΢ϯ͢Δͱσʔλϩετ
    • ະདྷʹॻ͖ࠐΉͰ͋Ζ͏ྖҬΛϑΝΠϧ࡞੒࣌ʹ֬อ͢
    ΔͷͰɺσΟεΫ࢖༻ޮ཰͕ѱ͍
    • ಛʹίϯςφͷΑ͏ͳ࢖͍ࣺͯͷϗετ͕େྔʹ࡞Β
    ΕΔͱແବ͕ଟ͍
    ΞʔΩςΫνϟ
    ࡮৽͕ඞཁ

    View full-size slide

  21. ࣌ܥྻσʔλϕʔεͷ
    طଘOSS࣮૷ͷௐࠪ

    View full-size slide

  22. Andreas Bader, Oliver Kopp, Michael Falkenthal. “Survey and Comparison of Open Source
    time Series Databases”. In proceedings of BTW 2017.

    View full-size slide

  23. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/pubhtml
    Open Source Time Series DB Comparison

    View full-size slide

  24. ࣌ܥྻσʔλϕʔεͷఆٛ
    • ҎԼͷੑ࣭Λຬͨ͢DBMSͷ͜ͱ
    • λΠϜελϯϓɺ஋ɺଐੑʢϝτϦοΫ໊ͳͲʣͰߏ
    ੒͞ΕΔσʔλͷߦΛ֨ೲͰ͖Δ
    • ࣌ܥྻͱͯ͠άϧʔϓԽ͞Εͨෳ਺ͷߦΛ֨ೲͰ͖Δ
    • σʔλߦʹରͯ͠ΫΤϦΛൃߦͰ͖Δ
    • λΠϜελϯϓ·ͨ͸࣌ؒൣғΛΫΤϦʹؚΊΒΕΔ

    View full-size slide

  25. ࣌ܥྻσʔλϕʔεͷ֓೦
    • ࣌ܥྻσʔλϕʔεͷఆٛΛຬͨ͢΋ͷ
    • ࣌ܥྻσʔλϕʔε΁ͷΑ͋͘Δػೳཁٻ
    • ෼ࢄ/ΫϥελϦϯά
    • Function
    • Rollup Aggregation / λά
    • σʔλղ૾౓
    • ΠϯλϑΣʔε

    View full-size slide

  26. ࣌ܥྻDBͷ෼ྨ
    TSDB on DBMS TSDB standalone
    OpenTSDB InfluxDB
    KairosDB
    Blueflood
    Graphite
    DalmatinerDB
    Beringei

    View full-size slide

  27. ࣌ܥྻDBͷ෼ྨ
    TSDB on DBMS TSDB standalone
    OpenTSDB InfluxDB
    KairosDB
    Blueflood
    Graphite
    DalmatinerDB
    Beringei
    HBase
    Cassandra
    Elasticsearch
    Riak Core

    View full-size slide

  28. طଘͷ࣌ܥྻDBͷσϝϦοτ
    TSDB on DBMS TSDB standalone
    ෼ࢄγεςϜͱͯ͠ͷ
    ӡ༻ίετ͕ߴ͍
    ෼ࢄγεςϜͱͯ͠ͷ
    ৴པੑΛଌΔͨΊͷ
    ࣮੷͕গͳ͍

    View full-size slide

  29. طଘͷ࣌ܥྻDBͷσϝϦοτ
    TSDB on DBMS TSDB standalone
    Rollup AggregationʹରԠ͍ͯ͠ͳ͍
    ΠϯλϑΣʔε͕େ͖͘มΘΔͷͰมߋίετ
    MackerelͷϫʔΫϩʔυͰΠϯϑϥίετ
    Λ཈͑ΒΕΔ͔Θ͔Βͳ͍

    View full-size slide

  30. http://developer.hatenastaff.com/entry/2015/12/25/140233

    View full-size slide

  31. ٕज़બ୒ͷϙΠϯτ
    • ಛʹσʔλϕʔε͸҆ఆ͢Δ͔Ͳ͏͔͕ॏཁ
    • ͋·Γ࢖ΘΕ͍ͯͳ͍OSSͷ࠾༻͸ආ͚͍ͨ
    • σʔλϕʔεܥͷOSS͸ݕূɾӡ༻ίετ͕ඇৗʹେ͖͍
    • طଘͷ࣌ܥྻσʔλϕʔε͸ಋೖ·Ͱָ͕ͳ୅ΘΓʹɺॊ
    ೈੑ͕௿͍
    • Πϯϑϥίετ͕ଟগ૿͑ͨͱͯ͠΋ɺࠓޙͷػೳ֦ॆ
    ΍ڝ߹༏ҐੑΛ֬อ͍ͨ͠

    View full-size slide

  32. AWSϚωʔδυαʔϏε
    ͷ࠾༻

    View full-size slide

  33. AWSϚωʔδυαʔϏε্ʹ
    ಠࣗͷ࣌ܥྻDBΞϓϦέʔγϣϯ
    Λ࣮૷

    View full-size slide

  34. ͳͥAWS͔
    • Amazon DynamoDBɺAmazon S3ɺAmazon Kinesis
    ͱ͍ͬͨϑϧϚωʔδυͳσʔλετΞ͕͋Δ
    • ੑೳΛ͓ۚʹ׵ࢉͯ͠ܭࢉ͠΍͍͢
    • GCPɺAzureͳͲ͸ࣾ಺࣮੷͕ͳ͍
    • Mackerelͷ੒௕଎౓΍ࠓޙ΍Γ͍ͨ͜ͱΛؑΈͯɺ͋
    Δఔ౓ͷϩοΫΠϯΛड͚ೖΕΔ͜ͱʹܾఆ

    View full-size slide

  35. ͳͥಠ࣮ࣗ૷͔
    • MackerelͷཁٻΛຬ࣮ͨ͢૷͕ͳ͍
    • ΋͘͠͸ཁٻΛຬ͔ͨ͢Ͳ͏͔Λݕূ͕ऴΘΔ·Ͱ
    ʹ͕͔͔࣌ؒΔ
    • AWSϚωʔδυαʔϏε࠾༻ʹΑΓσʔλϕʔεͷ
    ݕূ/ӡ༻ίετΛ࡟ݮͰ͖ΔͷͰɺͦͷ͔ΘΓͷ։
    ൃ޻਺Λ֬อͰ͖Δ

    View full-size slide

  36. ݱঢ়ͷ՝୊ (࠶ܝ)
    • ෛՙ෼ࢄͷӡ༻ίετͷ௿ݮ
    • σʔλอ࣋ظؒͷԆ௕
    • σʔλϩετ଱ੑͷ޲্
    • σΟεΫ࢖༻ޮ཰ͷ޲্

    View full-size slide

  37. ෛՙ෼ࢄͷӡ༻ίετᶃ
    • Amazon DynamoDB/Amazon S3ͳͲͷϑϧϚωʔ
    δυαʔϏεͷར༻ʹΑΓɺӡ༻ίετΛ࡟ݮ
    • γϟʔσΟϯάͷͨΊͷΩʔͷઃܭΛؒҧ͑ͳ͚Ε
    ͹ɺ͓ۚΛ෷ͬͨ෼͚ͩI/Oεϧʔϓοτ͕εέʔϧ
    • αʔό௥Ճɾ࡟ݮ࡞ۀ͕ෆཁ

    View full-size slide

  38. ෛՙ෼ࢄͷӡ༻ίετᶄ
    • ͱ͜Ζ͕DynamoDBͷσΟεΫI/Oίετ͕ߴ͍
    • ࠓͷMackerelͷI/OΛ޻෉ͳ͠ͰDynamoDBʹ޲͚ΔͱΠ
    ϯϑϥίετ͕ܹ૿͢Δ
    • ݸʑͷσʔλϙΠϯτΛผʑʹॻ͖ࠐΉͷͰ͸ͳ͘·ͱΊ
    ͯॻ͖ࠐΈɺI/OίετΛԼ͛Δඞཁ͕͋Δ
    • ࠷ऴతʹ͸ɺલஈʹAmazon ElastiCache(Redis)Λ഑ஔ
    ͠ɺόοϑΝ૚ͱΈͨͯͯར༻͢Δ͜ͱʹͳͬͨ

    View full-size slide

  39. σʔλอ࣋ظؒͷԆ௕ᶃ
    • DynamoDBͷσΟεΫ༰ྔ୯Ձ͸ײ֮తʹ͸͍҆
    • ͔͠͠ɺS3ͷίετ͸ελϯμʔυετϨʔδͰ΋
    DynamoDBͷ1/10ఔ౓
    • ͜͜ͰίετΛු͔͓͚ͤͯ͹ɺଞͷػೳ֦ॆʹίετΛ
    ͋ͯΒΕΔͨΊɺͳΔ΂͘S3Λ࢖͍͍ͨ
    • S3ʹஔ͍͓͚ͯ͹ɺσʔλղੳܥͷAWSαʔϏεͱ࿈ܞ͠
    ΍͍͢ͱ͍͏ͷ΋͋Δ

    View full-size slide

  40. σʔλอ࣋ظؒͷԆ௕ᶄ
    • Mackerelͷ৔߹ɺݹ͍ߴղ૾౓σʔλ͸΄ͱΜͲࢀর
    ͞Εͳ͍ (ࢀরہॴੑ)
    • ࢀরճ਺͕গͳ͍σʔλͷදࣔ͸ɺଟগϨεϙϯε͕
    ஗ͯ͘΋Α͍
    • ͜ͷੑ࣭Λར༻͠ɺϗοτσʔλΛDynamoDBɺίʔ
    ϧυσʔλΛS3ʹ഑ஔ͢Δ͜ͱͰίετ࠷దԽ͢Δ

    View full-size slide

  41. ͜ͷ࣌఺Ͱ
    ֊૚ܕσʔλετΞΞʔΩςΫνϟ
    ʹͳΔ͜ͱ͕ܾఆ

    View full-size slide

  42. σʔλϩετ଱ੑͷ޲্
    • σΟεΫॻ͖ࠐΈ͢ΔϝοηʔδΩϡʔΛ࠾༻
    • OSSͰ͋Ε͹Kafka΍RabbitMQͳͲ
    • AWSͰ͸ɺAmazon Kinesis Streams
    • ௚ۙͷ24࣌ؒ෼ͷϨίʔυΛσΟεΫʹอଘ
    • AWS LambdaʹϨίʔυΛ౉͢͜ͱ͕؆୯

    View full-size slide

  43. σΟεΫ࢖༻ޮ཰ͷ޲্
    • σʔλߏ଄(ޙड़)͕શ͘ҟͳΔͨΊɺະདྷͷྖҬΛ֬
    อ͢Δ͜ͱ͸͠ͳ͍ͷͰɺແବ͕ͳ͍
    • ଞͷOSSͷ࣌ܥྻDB͕࣮૷͍ͯ͠ΔΑ͏ͳѹॖอଘ
    ͸΍Βͳ͍
    • S3Λར༻͢Δ͜ͱͰσΟεΫ࢖༻ίετΛԼ͍͛ͯ
    ΔͷͰѹॖอଘ͸΍Βͳͯ͘Α͍ͱ൑அ

    View full-size slide

  44. ݱঢ়ͷ՝୊ͱղܾ ·ͱΊ
    ✓ ෛՙ෼ࢄͷӡ༻ίετͷ௿ݮ
    ➡ ElastiCacheͱDynamoDBʹΑΓεέʔϥ
    ϏϦςΟͱӡ༻ޮ཰Λ֬อ
    ✓ σʔλอ࣋ظؒͷԆ௕
    ➡ S3ʹΑΓ༰ྔ୯Ձͷ௿͍ετϨʔδ

    View full-size slide

  45. ݱঢ়ͷ՝୊ͱղܾ ·ͱΊ
    ✓ σʔλϩετ଱ੑͷ޲্
    ➡ Kinesis StreamsΛલஈʹ഑ஔ
    ✓ σΟεΫ࢖༻ޮ཰ͷ޲্
    ➡ σΟεΫ࢖༻ޮ཰Λҙࣝͨ͠σʔλߏ଄ͱ
    S3ͷར༻

    View full-size slide

  46. ࣌ܥྻσʔλΛಡΈग़͢෦෼Λ
    Microservicesͱͯ͠։ൃ

    View full-size slide

  47. Microservicesͷ։ൃ
    • Graphiteޓ׵ͷΠϯλϑΣʔεΛ࣮૷͠ɺΠϯλϑΣʔ
    εมߋίετΛ࠷খԽ
    • MackerelͰ͸ຊମΞϓϦέʔγϣϯ͸ScalaɺαςϥΠτ
    ͷΞϓϦέʔγϣϯ͸GoͳͷͰɺࠓճ΋GoΛ࠾༻
    • ੑೳ޲্ͷͨΊʹσʔλετΞ΁ฒߦͯ͠ΫΤϦൃߦ͢
    Δ͜ͱ͕༧૝Ͱ͖ͨͷͰɺܰྔεϨουΛ΋ͭGo͸Α͍
    બ୒

    View full-size slide

  48. ৽࣌ܥྻσʔλϕʔε
    ΞʔΩςΫνϟͷઃܭ

    View full-size slide

  49. ৽࣌ܥྻDBΞʔΩςΫνϟ֓ཁ
    • ۚમίετ࠷దԽͷͨΊʹࢀরہॴੑΛར༻
    • ϗοτσʔλΛDynamoDBɺίʔϧυσʔλΛS3ʹ഑ஔ
    • writeόοϑΝͱͯ͠Amazon ElastiCache(Redis)Λར༻
    • DynamoDBͷTTLػೳʹΑΓγʔϜϨεʹσʔλҠಈ
    • ॻ͖ࠐΈΛAmazon Kinesis StreamsͰड৴͠ɺσʔλϩ
    ετ଱ੑͱ଱ো֐ੑͷ޲্
    • Graphiteޓ׵ΠϯλϑΣʔεͰ֤ετϨʔδ͔Βσʔλऔ
    ಘ͢ΔWebΞϓϦέʔγϣϯΛ࣮૷

    View full-size slide

  50. σʔλετΞؒͷγʔϜϨεͳσʔλҠಈ
    • ෳ਺ͷσʔλετΞΛ૊Έ߹ΘͤΔ৔߹ɺσʔλΛγ
    ϯϓϧͳख๏ͰҠಈͤ͞Δ͜ͱ͕೉͍͠
    • 2017೥2݄ʹDynamoDB͕TTLαϙʔτ
    • ΞΠςϜ୯ҐͰTTLΛઃఆՄೳ
    • DynamoDB TriggersʹΑΓɺTTLΠϕϯτΛܖػʹ
    LambdaΛىಈ
    • DynamoDB͔ΒS3΁ͷݹ͍σʔλͷҠಈ͕༰қʹ

    View full-size slide

  51. Mackerel ࣌ܥྻσʔλ֓؍
    ࣌ܥྻ
    σʔλϕʔε
    .BDLFSFM
    "QQ 4DBMB

    "HFOU
    ϝτϦοΫ
    ౤ߘ
    read/write
    datapoints
    6TFS
    άϥϑදࣔ

    View full-size slide

  52. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read

    View full-size slide

  53. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read

    View full-size slide

  54. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    ֊૚ܕσʔλετΞ
    Buffer
    Hot Data
    Cold Data

    View full-size slide

  55. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    ίϯϙʔωϯτؒ
    ͷ઀ଓ

    View full-size slide

  56. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    3छྨͷσʔλϕʔε͔Β
    σʔλΛಡΈग़͢
    WebΞϓϦέʔγϣϯ

    View full-size slide

  57. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    DynamoDB TTLʹΑΔ
    γʔϜϨεͳσʔλҠಈ

    View full-size slide

  58. ৽࣌ܥྻσʔλϕʔε
    ΞʔΩςΫνϟͷ࣮૷

    View full-size slide

  59. ΞʔΩςΫνϟͷ࣮૷
    • σʔλߏ଄
    • RedisͰͷόοϑΝϦϯά
    • Graphiteޓ׵ΞϓϦέʔγϣϯͷI/OଟॏԽ
    ※࣮૷͸࣌ܥྻσʔλϕʔεݻ༗ͷ࿩ʹͳΔ

    View full-size slide

  60. σʔλߏ଄
    ϝτϦοΫ໊ [ ɹ… ]
    : {timestamp, value}
    Key Value
    ※࣮ࡍʹ͸ɺ֤ετϨʔδ͝ͱʹ޻෉͕͋Δ

    View full-size slide

  61. σʔλߏ଄
    ϝτϦοΫ໊ [ ɹ… ]
    Key Value
    άϥϑදࣔͰ͸࣌ܥྻʹฒΜͩσʔλϙΠϯ
    τΛҰ੪ʹಡΈऔΔ

    View full-size slide

  62. RedisͰͷόοϑΝϦϯά
    • cronͷΑ͏ͳόονॲཧʹΑΔϑϥογϡ͸ͨ͘͠ͳ͍
    • Redis্ͷಉҰϝτϦοΫͷσʔλϙΠϯτ਺͕ҰఆҎ্ʹͳ
    ΔͱDynamoDBͷରԠ͢ΔϨίʔυʹϑϥογϡॻ͖ࠐΈ
    • ʮ࣌ܥྻʯσʔλͳͷͰɺಉ࣌ʹಉ͡ܥྻͷσʔλϙΠϯτ͸
    ౸ண͠ͳ͍ͨΊɺόοϑΝͯ͠଴ͭඞཁ͕͋Δ
    • Lambda Function্ͰσʔλϙΠϯτ͕౸ண͢ΔͨͼʹɺϙΠ
    ϯτ਺͕ҰఆҎ্͔Ͳ͏͔൑ఆ͠ɺDynamoDB΁ϑϥογϡ

    View full-size slide

  63. RedisͰͷόοϑΝϦϯά
    ϝτϦοΫ໊ [ ɹ… ]
    Redis
    DynamoDB
    ϝτϦοΫ໊ [ ɹ… ]
    Flush
    >= Nݸ

    View full-size slide

  64. Graphiteޓ׵ΞϓϦέʔγϣϯͷI/OଟॏԽ
    • I/OଟॏԽʹΑΔಡΈग़͠࠷దԽ͸ੑೳ؍఺Ͱඞਢ
    • 1ͭͷϝτϦοΫॲཧʹରͯ͠goroutineΛׂΓ౰ͯ
    • ElastiCache/DynamoDB/S3ͷ֤σʔλετΞͷॲཧʹର
    ͯ͠goroutineΛׂΓ౰ͯ
    • ֤σʔλετΞ಺ͷ1Ϩίʔυʹରͯ͠goroutineׂΓ౰ͯ
    • ͨͩ͠DynamoDBʹ͍ͭͯ͸BatchGetItem APIΛར༻

    View full-size slide

  65. Graphiteޓ׵ΞϓϦέʔγϣϯͷI/OଟॏԽ
    ElastiCache
    DynamoDB
    S3
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    ϝτϦοΫ
    goroutine
    Ϩίʔυ

    View full-size slide

  66. ίʔυωʔϜ
    Diamond
    ݱࡏ։ൃத

    View full-size slide

  67. ·ͱΊ
    • Write-Intensive ApplicationͷAWSʹ͓͚ΔઃܭࣄྫΛ঺հ
    • Amazon ElastiCacheɺAmazon DynamoDBɺAmazon S3
    Λ૊Έ߹Θͤͨ֊૚ܕσʔλετΞΞʔΩςΫνϟͷఏҊ
    • Amazon Kinesis StreamsɺAWS LambdaΛކͱͯ͠ίϯ
    ϙʔωϯτؒΛ઀ଓ
    • Amazon DynamoDB TTLʹΑΔγʔϜϨεͳσʔλ഑ஔ

    View full-size slide

  68. ҰൠԽͷͨΊͷٞ࿦
    • ඞͣ3֊૚ʹ͢Δඞཁ͸ͳ͍
    • σΟεΫ࢖༻͕গͳ͚Ε͹ ElastiCache + DynamoDB
    • I/O͕গͳ͚Ε͹ DynamoDB + S3
    • readϨΠςϯγΛؾʹ͠ͳ͚Ε͹ɺRedis + S3
    • 3֊૚͋Ε͹ଟ͘ͷঢ়گͰίετΛ࠷దԽͰ͖Δ

    View full-size slide

  69. DiamonDB
    https://github.com/yuuki/diamondb
    ※ݸਓ։ൃͷOSSϓϩμΫτͰ͋Γ
    MackerelͰ։ൃதͷ΋ͷͱ͸ίʔυϕʔε͕ҟͳΓ·͢

    View full-size slide

  70. ࣌ܥྻσʔλϕʔεͱ͍͏֓೦
    ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
    ͸ͯͳ id:y_uuki
    AWS Summit Tokyo 2017

    View full-size slide

  71. Mackerelలࣔϒʔε΁
    ͥͻ͓ӽ͠Լ͍͞

    View full-size slide