Upgrade to Pro — share decks privately, control downloads, hide ads and more …

時系列データベースという概念をクラウドの技で再構築する / The rebuild of time-series database on AWS

時系列データベースという概念をクラウドの技で再構築する / The rebuild of time-series database on AWS

サーバ監視サービス Mackerel を提供するはてなが開発中の、高解像度・長期間のサーバメトリック収集を実現する、新しい時系列データベースを紹介します。具体的には、Amazon ElastiCache、Amazon DynamoDB、Amazon S3 を組み合わせ、Amazon Kinesis Streams と AWS Lambda によりコンポーネント接続した、階層構造のデータストアアーキテクチャの設計と実装を解説します。

Yuuki Tsubouchi (yuuk1)

June 01, 2017
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Programming

Transcript

  1. ࣌ܥྻσʔλϕʔεͱ͍͏֓೦
    ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
    ͸ͯͳ id:y_uuki
    AWS Summit Tokyo 2017

    View Slide

  2. id:y_uuki / @y_uuk1
    TSUBOUCHI Yuuki
    https://yuuk.io/
    גࣜձࣾ͸ͯͳ
    WebΦϖϨʔγϣϯΤϯδχΞ / γχΞΤϯδχΞ
    MackerelΞʔΩςΫνϟ࡮৽ϓϩδΣΫτϦʔμʔ

    View Slide

  3. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    αʔό؂ࢹαʔϏεMackerelʹ͓͚Δ
    ࣍ੈ୅ͷ࣌ܥྻσʔλϕʔεΞʔΩςΫνϟ
    ͷઃܭͱ࣮૷

    View Slide

  4. ΰʔϧ
    •Write-Intensive ApplicationΛAWS্ͰΞʔΩ
    ςΫνϟઃܭ͢ΔࣄྫΛ঺հ
    • ൺֱత୯७ͳσʔλΛอ࣋͢Δ࣌ܥྻσʔλϕʔε
    ͸Α͍ϞσϧέʔεͱͳΔ͸ͣ
    • ίετ࠷దԽͷͨΊʹɺࢀরہॴੑΛར༻ͨ͠֊૚ܕ
    σʔλετΞΞʔΩςΫνϟͷఏҊ

    View Slide

  5. ΰʔϧઃఆͷഎܠ
    • AWS SummitͳͷͰɺAWSΛ༻͍ͯΑΓྑ͍γεςϜ
    Λߏங͢ΔͨΊͷώϯτΛ఻͍͑ͨ
    • ͦͷͨΊʹɺ࣌ܥྻσʔλϕʔεͦͷ΋ͷΑΓ΋ɺ൚
    ༻తͳΞʔΩςΫνϟઃܭͷͨΊͷٞ࿦ͷ౔୆ͱͳΔ
    ΋ͷΛఏҊ͍ͨ͠

    View Slide

  6. Mackerel
    https://mackerel.io/

    View Slide

  7. αʔόͷϝτϦοΫՄ
    ࢹԽ

    View Slide

  8. https://speakerdeck.com/sugiyama88/mackerel-meetup-number-8

    View Slide

  9. ߴ଎ͳ਺ઍݸͷܥྻͷදࣔ

    View Slide

  10. 1෼ղ૾౓σʔλΛ௕ظؒอ࣋
    https://mackerel.io/ja/docs/entry/overview
    ྫ͑͹1೥

    View Slide

  11. 1෼ҎԼͷղ૾౓ͷରԠ
    https://mackerel.io/ja/docs/entry/overview
    ྫ͑͹15ඵ

    View Slide

  12. ੒௕͚ͭͮ͠ΔαʔϏεͷεέʔ
    ϥϏϦςΟͷ֬อ
    σʔλྔͱI/O͕ܹ૿͢Δػೳͷ
    ࣮ݱ

    View Slide

  13. MackerelΞʔΩςΫνϟ
    ࣌ܥྻ
    σʔλϕʔε
    .BDLFSFM
    "QQ 4DBMB

    "HFOU
    ϝτϦοΫ
    ౤ߘ
    read/write
    datapoints
    6TFS
    άϥϑදࣔ

    View Slide

  14. ࣌ܥྻσʔλϕʔεͱ͸
    • ࣌ܥྻσʔλΛѻ͏͜ͱʹಛԽͨ͠σʔλϕʔε
    • αʔόϞχλϦϯάɺIoTͳͲͷ༻్ʹ࢖ΘΕΔ
    • ༗໊ͳ࣮૷͸Graphite/InfluxDB/OpenTSDBͳͲ
    • ஍ຯʹΈ͑ͯҙ֎ͱଟ͘ͷ࣮૷͕ଘࡏ͢Δʢৄࡉޙड़)
    • ֶज़ݚڀ࿦จͷର৅ʹ΋ͳΔ෼໺

    View Slide

  15. http://blog.yuuk.io/entry/high-performance-graphite

    View Slide

  16. http://blog.yuuk.io/entry/high-performance-graphite
    Graphite
    http://graphite.readthedocs.io/en/latest/

    View Slide

  17. ݱঢ়ͷ໰୊఺

    View Slide

  18. Graphiteͷ໰୊఺ᶃ
    • ෛՙ෼ࢄͷͨΊͷӡ༻ίετ͕ߴ͍
    • γϟʔυ૿ݮ࣌ʹσʔλ࠶഑ஔ͕ඞཁ
    • ෼ࢄ͕Ή͔͍ͣͨ͠ΊεέʔϧΞοϓΛબ୒
    • σʔλอ࣋ظؒΛ૿΍͢ͱۚમίετ͕ܹ૿
    • NANDϑϥογϡϝϞϦͷͨΊ༰ྔ୯Ձ͕େ͖͍
    • ࢀরස౓ͷ௿͍σʔλ͚ͩ௿଎ͳσΟεΫʹஔ͖͍͕ͨ
    ϥ΢ϯυϩϏϯσʔλϕʔεͷੑ্࣭೉͍͠

    View Slide

  19. Graphiteͷ໰୊఺ᶄ
    • σʔλϩετ଱ੑ͕௿͍
    • ΞϓϦέʔγϣϯ͔Β͸ඇಉظͰϝϞϦ্ͷΩϡʔʹ
    ॻ͖ࠐΉͨΊɺαʔόμ΢ϯ͢Δͱσʔλϩετ
    • ະདྷͷྖҬΛϑΝΠϧ࡞੒࣌ʹ֬อ͢ΔͷͰɺσΟεΫ
    ࢖༻ޮ཰͕௿͍
    • ಛʹίϯςφͷΑ͏ͳ࢖͍ࣺͯͷϗετ͕େྔʹ࡞Β
    ΕΔͱແବ͕૿͑Δ

    View Slide

  20. Graphiteͷ໰୊఺ᶄ
    • σʔλϩετ଱ੑ͕௿͍ (௿durability)
    • ΞϓϦέʔγϣϯ͔Β͸ඇಉظͰϝϞϦ্ͷΩϡʔʹ
    ॻ͖ࠐΉͨΊɺαʔόμ΢ϯ͢Δͱσʔλϩετ
    • ະདྷʹॻ͖ࠐΉͰ͋Ζ͏ྖҬΛϑΝΠϧ࡞੒࣌ʹ֬อ͢
    ΔͷͰɺσΟεΫ࢖༻ޮ཰͕ѱ͍
    • ಛʹίϯςφͷΑ͏ͳ࢖͍ࣺͯͷϗετ͕େྔʹ࡞Β
    ΕΔͱແବ͕ଟ͍
    ΞʔΩςΫνϟ
    ࡮৽͕ඞཁ

    View Slide

  21. ࣌ܥྻσʔλϕʔεͷ
    طଘOSS࣮૷ͷௐࠪ

    View Slide

  22. Andreas Bader, Oliver Kopp, Michael Falkenthal. “Survey and Comparison of Open Source
    time Series Databases”. In proceedings of BTW 2017.

    View Slide

  23. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/pubhtml
    Open Source Time Series DB Comparison

    View Slide

  24. ࣌ܥྻσʔλϕʔεͷఆٛ
    • ҎԼͷੑ࣭Λຬͨ͢DBMSͷ͜ͱ
    • λΠϜελϯϓɺ஋ɺଐੑʢϝτϦοΫ໊ͳͲʣͰߏ
    ੒͞ΕΔσʔλͷߦΛ֨ೲͰ͖Δ
    • ࣌ܥྻͱͯ͠άϧʔϓԽ͞Εͨෳ਺ͷߦΛ֨ೲͰ͖Δ
    • σʔλߦʹରͯ͠ΫΤϦΛൃߦͰ͖Δ
    • λΠϜελϯϓ·ͨ͸࣌ؒൣғΛΫΤϦʹؚΊΒΕΔ

    View Slide

  25. ࣌ܥྻσʔλϕʔεͷ֓೦
    • ࣌ܥྻσʔλϕʔεͷఆٛΛຬͨ͢΋ͷ
    • ࣌ܥྻσʔλϕʔε΁ͷΑ͋͘Δػೳཁٻ
    • ෼ࢄ/ΫϥελϦϯά
    • Function
    • Rollup Aggregation / λά
    • σʔλղ૾౓
    • ΠϯλϑΣʔε

    View Slide

  26. ࣌ܥྻDBͷ෼ྨ
    TSDB on DBMS TSDB standalone
    OpenTSDB InfluxDB
    KairosDB
    Blueflood
    Graphite
    DalmatinerDB
    Beringei

    View Slide

  27. ࣌ܥྻDBͷ෼ྨ
    TSDB on DBMS TSDB standalone
    OpenTSDB InfluxDB
    KairosDB
    Blueflood
    Graphite
    DalmatinerDB
    Beringei
    HBase
    Cassandra
    Elasticsearch
    Riak Core

    View Slide

  28. طଘͷ࣌ܥྻDBͷσϝϦοτ
    TSDB on DBMS TSDB standalone
    ෼ࢄγεςϜͱͯ͠ͷ
    ӡ༻ίετ͕ߴ͍
    ෼ࢄγεςϜͱͯ͠ͷ
    ৴པੑΛଌΔͨΊͷ
    ࣮੷͕গͳ͍

    View Slide

  29. طଘͷ࣌ܥྻDBͷσϝϦοτ
    TSDB on DBMS TSDB standalone
    Rollup AggregationʹରԠ͍ͯ͠ͳ͍
    ΠϯλϑΣʔε͕େ͖͘มΘΔͷͰมߋίετ
    MackerelͷϫʔΫϩʔυͰΠϯϑϥίετ
    Λ཈͑ΒΕΔ͔Θ͔Βͳ͍

    View Slide

  30. http://developer.hatenastaff.com/entry/2015/12/25/140233

    View Slide

  31. ٕज़બ୒ͷϙΠϯτ
    • ಛʹσʔλϕʔε͸҆ఆ͢Δ͔Ͳ͏͔͕ॏཁ
    • ͋·Γ࢖ΘΕ͍ͯͳ͍OSSͷ࠾༻͸ආ͚͍ͨ
    • σʔλϕʔεܥͷOSS͸ݕূɾӡ༻ίετ͕ඇৗʹେ͖͍
    • طଘͷ࣌ܥྻσʔλϕʔε͸ಋೖ·Ͱָ͕ͳ୅ΘΓʹɺॊ
    ೈੑ͕௿͍
    • Πϯϑϥίετ͕ଟগ૿͑ͨͱͯ͠΋ɺࠓޙͷػೳ֦ॆ
    ΍ڝ߹༏ҐੑΛ֬อ͍ͨ͠

    View Slide

  32. AWSϚωʔδυαʔϏε
    ͷ࠾༻

    View Slide

  33. AWSϚωʔδυαʔϏε্ʹ
    ಠࣗͷ࣌ܥྻDBΞϓϦέʔγϣϯ
    Λ࣮૷

    View Slide

  34. ͳͥAWS͔
    • Amazon DynamoDBɺAmazon S3ɺAmazon Kinesis
    ͱ͍ͬͨϑϧϚωʔδυͳσʔλετΞ͕͋Δ
    • ੑೳΛ͓ۚʹ׵ࢉͯ͠ܭࢉ͠΍͍͢
    • GCPɺAzureͳͲ͸ࣾ಺࣮੷͕ͳ͍
    • Mackerelͷ੒௕଎౓΍ࠓޙ΍Γ͍ͨ͜ͱΛؑΈͯɺ͋
    Δఔ౓ͷϩοΫΠϯΛड͚ೖΕΔ͜ͱʹܾఆ

    View Slide

  35. ͳͥಠ࣮ࣗ૷͔
    • MackerelͷཁٻΛຬ࣮ͨ͢૷͕ͳ͍
    • ΋͘͠͸ཁٻΛຬ͔ͨ͢Ͳ͏͔Λݕূ͕ऴΘΔ·Ͱ
    ʹ͕͔͔࣌ؒΔ
    • AWSϚωʔδυαʔϏε࠾༻ʹΑΓσʔλϕʔεͷ
    ݕূ/ӡ༻ίετΛ࡟ݮͰ͖ΔͷͰɺͦͷ͔ΘΓͷ։
    ൃ޻਺Λ֬อͰ͖Δ

    View Slide

  36. ݱঢ়ͷ՝୊ (࠶ܝ)
    • ෛՙ෼ࢄͷӡ༻ίετͷ௿ݮ
    • σʔλอ࣋ظؒͷԆ௕
    • σʔλϩετ଱ੑͷ޲্
    • σΟεΫ࢖༻ޮ཰ͷ޲্

    View Slide

  37. ෛՙ෼ࢄͷӡ༻ίετᶃ
    • Amazon DynamoDB/Amazon S3ͳͲͷϑϧϚωʔ
    δυαʔϏεͷར༻ʹΑΓɺӡ༻ίετΛ࡟ݮ
    • γϟʔσΟϯάͷͨΊͷΩʔͷઃܭΛؒҧ͑ͳ͚Ε
    ͹ɺ͓ۚΛ෷ͬͨ෼͚ͩI/Oεϧʔϓοτ͕εέʔϧ
    • αʔό௥Ճɾ࡟ݮ࡞ۀ͕ෆཁ

    View Slide

  38. ෛՙ෼ࢄͷӡ༻ίετᶄ
    • ͱ͜Ζ͕DynamoDBͷσΟεΫI/Oίετ͕ߴ͍
    • ࠓͷMackerelͷI/OΛ޻෉ͳ͠ͰDynamoDBʹ޲͚ΔͱΠ
    ϯϑϥίετ͕ܹ૿͢Δ
    • ݸʑͷσʔλϙΠϯτΛผʑʹॻ͖ࠐΉͷͰ͸ͳ͘·ͱΊ
    ͯॻ͖ࠐΈɺI/OίετΛԼ͛Δඞཁ͕͋Δ
    • ࠷ऴతʹ͸ɺલஈʹAmazon ElastiCache(Redis)Λ഑ஔ
    ͠ɺόοϑΝ૚ͱΈͨͯͯར༻͢Δ͜ͱʹͳͬͨ

    View Slide

  39. σʔλอ࣋ظؒͷԆ௕ᶃ
    • DynamoDBͷσΟεΫ༰ྔ୯Ձ͸ײ֮తʹ͸͍҆
    • ͔͠͠ɺS3ͷίετ͸ελϯμʔυετϨʔδͰ΋
    DynamoDBͷ1/10ఔ౓
    • ͜͜ͰίετΛු͔͓͚ͤͯ͹ɺଞͷػೳ֦ॆʹίετΛ
    ͋ͯΒΕΔͨΊɺͳΔ΂͘S3Λ࢖͍͍ͨ
    • S3ʹஔ͍͓͚ͯ͹ɺσʔλղੳܥͷAWSαʔϏεͱ࿈ܞ͠
    ΍͍͢ͱ͍͏ͷ΋͋Δ

    View Slide

  40. σʔλอ࣋ظؒͷԆ௕ᶄ
    • Mackerelͷ৔߹ɺݹ͍ߴղ૾౓σʔλ͸΄ͱΜͲࢀর
    ͞Εͳ͍ (ࢀরہॴੑ)
    • ࢀরճ਺͕গͳ͍σʔλͷදࣔ͸ɺଟগϨεϙϯε͕
    ஗ͯ͘΋Α͍
    • ͜ͷੑ࣭Λར༻͠ɺϗοτσʔλΛDynamoDBɺίʔ
    ϧυσʔλΛS3ʹ഑ஔ͢Δ͜ͱͰίετ࠷దԽ͢Δ

    View Slide

  41. ͜ͷ࣌఺Ͱ
    ֊૚ܕσʔλετΞΞʔΩςΫνϟ
    ʹͳΔ͜ͱ͕ܾఆ

    View Slide

  42. σʔλϩετ଱ੑͷ޲্
    • σΟεΫॻ͖ࠐΈ͢ΔϝοηʔδΩϡʔΛ࠾༻
    • OSSͰ͋Ε͹Kafka΍RabbitMQͳͲ
    • AWSͰ͸ɺAmazon Kinesis Streams
    • ௚ۙͷ24࣌ؒ෼ͷϨίʔυΛσΟεΫʹอଘ
    • AWS LambdaʹϨίʔυΛ౉͢͜ͱ͕؆୯

    View Slide

  43. σΟεΫ࢖༻ޮ཰ͷ޲্
    • σʔλߏ଄(ޙड़)͕શ͘ҟͳΔͨΊɺະདྷͷྖҬΛ֬
    อ͢Δ͜ͱ͸͠ͳ͍ͷͰɺແବ͕ͳ͍
    • ଞͷOSSͷ࣌ܥྻDB͕࣮૷͍ͯ͠ΔΑ͏ͳѹॖอଘ
    ͸΍Βͳ͍
    • S3Λར༻͢Δ͜ͱͰσΟεΫ࢖༻ίετΛԼ͍͛ͯ
    ΔͷͰѹॖอଘ͸΍Βͳͯ͘Α͍ͱ൑அ

    View Slide

  44. ݱঢ়ͷ՝୊ͱղܾ ·ͱΊ
    ✓ ෛՙ෼ࢄͷӡ༻ίετͷ௿ݮ
    ➡ ElastiCacheͱDynamoDBʹΑΓεέʔϥ
    ϏϦςΟͱӡ༻ޮ཰Λ֬อ
    ✓ σʔλอ࣋ظؒͷԆ௕
    ➡ S3ʹΑΓ༰ྔ୯Ձͷ௿͍ετϨʔδ

    View Slide

  45. ݱঢ়ͷ՝୊ͱղܾ ·ͱΊ
    ✓ σʔλϩετ଱ੑͷ޲্
    ➡ Kinesis StreamsΛલஈʹ഑ஔ
    ✓ σΟεΫ࢖༻ޮ཰ͷ޲্
    ➡ σΟεΫ࢖༻ޮ཰Λҙࣝͨ͠σʔλߏ଄ͱ
    S3ͷར༻

    View Slide

  46. ࣌ܥྻσʔλΛಡΈग़͢෦෼Λ
    Microservicesͱͯ͠։ൃ

    View Slide

  47. Microservicesͷ։ൃ
    • Graphiteޓ׵ͷΠϯλϑΣʔεΛ࣮૷͠ɺΠϯλϑΣʔ
    εมߋίετΛ࠷খԽ
    • MackerelͰ͸ຊମΞϓϦέʔγϣϯ͸ScalaɺαςϥΠτ
    ͷΞϓϦέʔγϣϯ͸GoͳͷͰɺࠓճ΋GoΛ࠾༻
    • ੑೳ޲্ͷͨΊʹσʔλετΞ΁ฒߦͯ͠ΫΤϦൃߦ͢
    Δ͜ͱ͕༧૝Ͱ͖ͨͷͰɺܰྔεϨουΛ΋ͭGo͸Α͍
    બ୒

    View Slide

  48. ৽࣌ܥྻσʔλϕʔε
    ΞʔΩςΫνϟͷઃܭ

    View Slide

  49. ৽࣌ܥྻDBΞʔΩςΫνϟ֓ཁ
    • ۚમίετ࠷దԽͷͨΊʹࢀরہॴੑΛར༻
    • ϗοτσʔλΛDynamoDBɺίʔϧυσʔλΛS3ʹ഑ஔ
    • writeόοϑΝͱͯ͠Amazon ElastiCache(Redis)Λར༻
    • DynamoDBͷTTLػೳʹΑΓγʔϜϨεʹσʔλҠಈ
    • ॻ͖ࠐΈΛAmazon Kinesis StreamsͰड৴͠ɺσʔλϩ
    ετ଱ੑͱ଱ো֐ੑͷ޲্
    • Graphiteޓ׵ΠϯλϑΣʔεͰ֤ετϨʔδ͔Βσʔλऔ
    ಘ͢ΔWebΞϓϦέʔγϣϯΛ࣮૷

    View Slide

  50. σʔλετΞؒͷγʔϜϨεͳσʔλҠಈ
    • ෳ਺ͷσʔλετΞΛ૊Έ߹ΘͤΔ৔߹ɺσʔλΛγ
    ϯϓϧͳख๏ͰҠಈͤ͞Δ͜ͱ͕೉͍͠
    • 2017೥2݄ʹDynamoDB͕TTLαϙʔτ
    • ΞΠςϜ୯ҐͰTTLΛઃఆՄೳ
    • DynamoDB TriggersʹΑΓɺTTLΠϕϯτΛܖػʹ
    LambdaΛىಈ
    • DynamoDB͔ΒS3΁ͷݹ͍σʔλͷҠಈ͕༰қʹ

    View Slide

  51. Mackerel ࣌ܥྻσʔλ֓؍
    ࣌ܥྻ
    σʔλϕʔε
    .BDLFSFM
    "QQ 4DBMB

    "HFOU
    ϝτϦοΫ
    ౤ߘ
    read/write
    datapoints
    6TFS
    άϥϑදࣔ

    View Slide

  52. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read

    View Slide

  53. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read

    View Slide

  54. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    ֊૚ܕσʔλετΞ
    Buffer
    Hot Data
    Cold Data

    View Slide

  55. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    ίϯϙʔωϯτؒ
    ͷ઀ଓ

    View Slide

  56. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    3छྨͷσʔλϕʔε͔Β
    σʔλΛಡΈग़͢
    WebΞϓϦέʔγϣϯ

    View Slide

  57. write
    datapoints
    Kinesis Streams
    Lambda
    read
    datapoints
    ElastiCache
    DynamoDB
    Lambda
    TTL expired
    flush
    write
    Web App
    (Golang)
    S3
    read
    read
    read
    DynamoDB TTLʹΑΔ
    γʔϜϨεͳσʔλҠಈ

    View Slide

  58. ৽࣌ܥྻσʔλϕʔε
    ΞʔΩςΫνϟͷ࣮૷

    View Slide

  59. ΞʔΩςΫνϟͷ࣮૷
    • σʔλߏ଄
    • RedisͰͷόοϑΝϦϯά
    • Graphiteޓ׵ΞϓϦέʔγϣϯͷI/OଟॏԽ
    ※࣮૷͸࣌ܥྻσʔλϕʔεݻ༗ͷ࿩ʹͳΔ

    View Slide

  60. σʔλߏ଄
    ϝτϦοΫ໊ [ ɹ… ]
    : {timestamp, value}
    Key Value
    ※࣮ࡍʹ͸ɺ֤ετϨʔδ͝ͱʹ޻෉͕͋Δ

    View Slide

  61. σʔλߏ଄
    ϝτϦοΫ໊ [ ɹ… ]
    Key Value
    άϥϑදࣔͰ͸࣌ܥྻʹฒΜͩσʔλϙΠϯ
    τΛҰ੪ʹಡΈऔΔ

    View Slide

  62. RedisͰͷόοϑΝϦϯά
    • cronͷΑ͏ͳόονॲཧʹΑΔϑϥογϡ͸ͨ͘͠ͳ͍
    • Redis্ͷಉҰϝτϦοΫͷσʔλϙΠϯτ਺͕ҰఆҎ্ʹͳ
    ΔͱDynamoDBͷରԠ͢ΔϨίʔυʹϑϥογϡॻ͖ࠐΈ
    • ʮ࣌ܥྻʯσʔλͳͷͰɺಉ࣌ʹಉ͡ܥྻͷσʔλϙΠϯτ͸
    ౸ண͠ͳ͍ͨΊɺόοϑΝͯ͠଴ͭඞཁ͕͋Δ
    • Lambda Function্ͰσʔλϙΠϯτ͕౸ண͢ΔͨͼʹɺϙΠ
    ϯτ਺͕ҰఆҎ্͔Ͳ͏͔൑ఆ͠ɺDynamoDB΁ϑϥογϡ

    View Slide

  63. RedisͰͷόοϑΝϦϯά
    ϝτϦοΫ໊ [ ɹ… ]
    Redis
    DynamoDB
    ϝτϦοΫ໊ [ ɹ… ]
    Flush
    >= Nݸ

    View Slide

  64. Graphiteޓ׵ΞϓϦέʔγϣϯͷI/OଟॏԽ
    • I/OଟॏԽʹΑΔಡΈग़͠࠷దԽ͸ੑೳ؍఺Ͱඞਢ
    • 1ͭͷϝτϦοΫॲཧʹରͯ͠goroutineΛׂΓ౰ͯ
    • ElastiCache/DynamoDB/S3ͷ֤σʔλετΞͷॲཧʹର
    ͯ͠goroutineΛׂΓ౰ͯ
    • ֤σʔλετΞ಺ͷ1Ϩίʔυʹରͯ͠goroutineׂΓ౰ͯ
    • ͨͩ͠DynamoDBʹ͍ͭͯ͸BatchGetItem APIΛར༻

    View Slide

  65. Graphiteޓ׵ΞϓϦέʔγϣϯͷI/OଟॏԽ
    ElastiCache
    DynamoDB
    S3
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    ϝτϦοΫ
    goroutine
    Ϩίʔυ

    View Slide

  66. ίʔυωʔϜ
    Diamond
    ݱࡏ։ൃத

    View Slide

  67. ·ͱΊ
    • Write-Intensive ApplicationͷAWSʹ͓͚ΔઃܭࣄྫΛ঺հ
    • Amazon ElastiCacheɺAmazon DynamoDBɺAmazon S3
    Λ૊Έ߹Θͤͨ֊૚ܕσʔλετΞΞʔΩςΫνϟͷఏҊ
    • Amazon Kinesis StreamsɺAWS LambdaΛކͱͯ͠ίϯ
    ϙʔωϯτؒΛ઀ଓ
    • Amazon DynamoDB TTLʹΑΔγʔϜϨεͳσʔλ഑ஔ

    View Slide

  68. ҰൠԽͷͨΊͷٞ࿦
    • ඞͣ3֊૚ʹ͢Δඞཁ͸ͳ͍
    • σΟεΫ࢖༻͕গͳ͚Ε͹ ElastiCache + DynamoDB
    • I/O͕গͳ͚Ε͹ DynamoDB + S3
    • readϨΠςϯγΛؾʹ͠ͳ͚Ε͹ɺRedis + S3
    • 3֊૚͋Ε͹ଟ͘ͷঢ়گͰίετΛ࠷దԽͰ͖Δ

    View Slide

  69. DiamonDB
    https://github.com/yuuki/diamondb
    ※ݸਓ։ൃͷOSSϓϩμΫτͰ͋Γ
    MackerelͰ։ൃதͷ΋ͷͱ͸ίʔυϕʔε͕ҟͳΓ·͢

    View Slide

  70. ࣌ܥྻσʔλϕʔεͱ͍͏֓೦
    ΛΫϥ΢υͷٕͰ࠶ߏங͢Δ
    ͸ͯͳ id:y_uuki
    AWS Summit Tokyo 2017

    View Slide

  71. Mackerelలࣔϒʔε΁
    ͥͻ͓ӽ͠Լ͍͞

    View Slide