Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017

Ken Hamada
November 03, 2017

サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017

サーバレスアーキテクチャによる時系列データベースの構築と監視

Ken Hamada

November 03, 2017
Tweet

More Decks by Ken Hamada

Other Decks in Technology

Transcript

  1. αʔόϨεΞʔΩςΫνϟʹΑΔ
    ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹ
    גࣜձࣾ͸ͯͳɹ⅓ా ݈
    2017/11/03 Serverlessconf Tokyo

    View Slide

  2. ࣗݾ঺հ
    גࣜձࣾ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ
    MackerelνʔϜ ςοΫϦʔυ
    ޷͖: Rust, Haskell, Go, Scala, Vim
    ϓϩάϥϜϞάϞά itchyny.hatenablog.com
    lightline.vim, calendar.vim, etc.
    Thank you!
    ⅓ా ݈ Ken Hamada

    View Slide

  3. αʔόʔϨεͷίϯϙʔωϯτΛ࢖ͬͯ
    αʔϏεΛӡ༻͍ͯ͠Δਓ ✋

    View Slide

  4. αʔόʔϨεͷίϯϙʔωϯτΛ
    ͔ͬ͠Γ؂ࢹ͍ͯ͠Δਓ ✋

    View Slide

  5. ؂ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ
    ͲͷΑ͏ʹϚωʔδυαʔϏεΛ
    ؂ࢹɾ؅ཧ͍ͯ͠Δ͔Λ͓఻͑͠·͢

    View Slide

  6. ໨࣍
    Mackerelʹ͍ͭͯ
    ࣌ܥྻσʔλϕʔε
    Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ
    Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά
    ֤ίϯϙʔωϯτͷಛੑͱ؂ࢹ

    View Slide

  7. Mackerelʹ͍ͭͯ

    View Slide

  8. Mackerel
    mackerel.io

    View Slide

  9. Mackerel
    SaaSͷαʔόʔ؂ࢹɾ؅ཧπʔϧ
    ͸ͯͳࣾ಺ͷ؂ࢹγεςϜΛαʔϏεԽͯ͠ެ։
    mackerel-agent͕ϝτϦοΫΛ౤ߘ͢Δpushܕ
    νϟοτπʔϧ΍ߏ੒؅ཧπʔϧͱ૊Έ߹Θͤ
    Slack, HipChat, etc. Chef, Ansible, etc.

    View Slide

  10. Mackerelུ֓ਤ
    CPUɾMemory࢖༻཰
    OS৘ใ
    ϝτϦοΫσʔλ
    ϝλσʔλ
    ϝʔϧ
    ؅ཧը໘
    νϟοτπʔϧ
    Ξϥʔτൃੜ

    View Slide

  11. Mackerelͷάϥϑ
    ϝτϦοΫͷσʔλ͸
    ࣌ܥྻσʔλϕʔεʹ
    อଘ͞Ε͍ͯΔ
    ϗετ৘ใ΍ϝλσʔλ͸
    ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ

    View Slide

  12. ࣌ܥྻσʔλϕʔε

    View Slide

  13. ࣌ܥྻσʔλϕʔε
    • ࣌ܥྻͷσʔλ (యܕతʹ͸਺஋σʔλ) ͷอଘʹಛԽ
    • αʔόʔϞχλϦϯάɾIoT༻్
    • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ
    • ߴղ૾౓ɾ௕ظอଘɾେྔϝτϦοΫ 㱺 ෼ࢄDB
    • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB
    Mackerel͕࢖͍ͬͯ·ͨ͠

    View Slide

  14. GraphiteΛӡ༻͢Δ্Ͱͷ՝୊
    ϋʔυ΢ΣΞͷݶք (disk༰ྔ)
    • ετϨʔδͷௐୡίετ
    εέʔϥϏϦςΟʔͷ֬อ͕ࠔ೉
    • ϝτϦοΫͷ௕ظอଘ (1m:25h 㱺 1m:400d+)
    σʔλϩετ଱ੑͷ௿͞
    ϋοΫɾ࠷దԽ͠ʹ͘͞
    Ϋϥ΢υΛ׆༻ͯ͠ӡ༻ίετ࡟ݮ͍ͨ͠
    Application servers
    mackerel.io

    View Slide

  15. ৽࣌ܥྻDBͰୡ੒͍ͨ͜͠ͱ
    εέʔϥϏϦςΟʔ
    ӡ༻ίετɾϋʔυ΢ΣΞௐୡίετ཈͑Δ
    σʔλϩετ଱ੑ

    View Slide

  16. ৽࣌ܥྻDBͷઃܭ
    εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετ࡟ݮ
    • ϚωʔδυɾαʔϏεΛ࢖͏ 㱺 AWS DynamoDB
    DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ཈͍͑ͨ
    • RedisΛΩϟογϡͱͯ͠ར༻
    ݹ͍ϝτϦοΫσʔλ͸༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ
    • DynamoDB 㱺 S3 ΁Ҡಈ
    σʔλϩετ଱ੑ
    • Amazon Kinesis Streams

    View Slide

  17. ہॴࢀরੑͱετϨʔδͷબ୒
    සൟʹࢀর͞ΕΔ
    ৽͍͠ϝτϦοΫ
    ݹ͍
    ͋·Γࢀর͞Εͳ͍
    ಡΈॻ͖ίετ
    อ࣋ίετ
    DynamoDB
    S3 Redis
    ϨΠςϯγ
    εέʔϥϏϦςΟʔ
    ༰ྔ୯Ձ

    View Slide

  18. ہॴࢀরੑͱετϨʔδͷબ୒
    සൟʹࢀর͞ΕΔ
    ৽͍͠ϝτϦοΫ
    ݹ͍
    ͋·Γࢀর͞Εͳ͍
    ಡΈॻ͖ίετ
    อ࣋ίετ
    DynamoDB
    S3 Redis
    ϨΠςϯγ
    εέʔϥϏϦςΟʔ
    ༰ྔ୯Ձ
    ಛੑͷҟͳΔετϨʔδΛ૊Έ߹ΘͤΔ͜ͱͰ
    ͔͔ΔίετΛ࠷దԽ͢Δ

    View Slide

  19. Aug. 2017 ৽࣌ܥྻDBʹҠߦ

    View Slide

  20. diamondߏ੒ਤ
    Mackerelͷ৽࣌ܥྻσʔλϕʔε
    diamond: graphiteͷಉૉମ (ͱͯ΋ߗ͍)
    DynamoDB
    Kinesis Steams
    S3
    Lambda
    Redis cluster
    Lambda
    TTL expired
    Application servers
    diamond-reader
    mackerel.io
    diamond-writer

    View Slide

  21. Mackerelͷ৽࣌ܥྻσʔλϕʔε
    diamond-writerͷ։ൃ

    View Slide

  22. diamond-writer
    • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ
    • AWS Lambda (node.js)
    • RedisΛ࢖͍ॻ͖ࠐΈίετΛ࠷దԽ
    • ΞΠςϜʹTTLΛઃఆ
    • ݹ͍ϝτϦοΫ͸ࣗಈͰফ͑Δ DynamoDB
    Kinesis Steams
    Redis cluster
    Lambda

    View Slide

  23. Kinesis Streams→Lambda
    • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ
    • Lambda͕ॲཧͯ͠΋σʔλ͸Kinesis Streamsʹ࢒Δ
    • ো֐ൃੜ࣌఺͔Β࠶ॲཧͰ͖Δ
    • backup༻ʹผͷLambdaΛ࢖͏
    • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ
    • Lambdaͷॲཧͷႈ౳ੑ͕ॏཁ
    Kinesis Steams Lambda
    Lambda backup
    TRIM_HORIZON
    AT_TIMESTAMP
    ো֐࣌ͷόοΫΞοϓܥ

    View Slide

  24. Lambda→Redis cluster
    • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻
    • EC2্ͰclusterΛ૊ΜͰ͍Δ
    • ౰ॳ͸ElastiCacheΛ૝ఆ͍ͯͨ͠
    • εέʔϧ {Ξοϓ,Ξ΢τ} ͕ΦϯϥΠϯͰͰ͖ͳ͍
    • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢
    • ϝτϦοΫ͕Ұఆݸ਺Ҏ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ
    • cron͸εέʔϧͤ͞ʹ͍͘ͷͰجຊతʹ͸࢖Θͳ͍
    Lambda
    Redis cluster

    View Slide

  25. Lambda→DynamoDB
    • MetricName (partition key)ɾTimestamp (sort key)
    • ϝτϦοΫ஋͸Mapʹอଘ
    • BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ
    • Write capacity͸جຊతʹҰఆ
    • Read capacity͸ΞΫηε͕ूத͢Δ͜ͱ͕͋Δ
    DynamoDB
    Lambda

    View Slide

  26. Lambda ίʔυߏ੒ɾdeploy
    ϩδοΫ͸libσΟϨΫτϦʹ੾Γग़͢
    • function handlerΛ࠷খʹอͭ (client஫ೖɾcallback)
    • ֤ϥΠϒϥϦʔΛखް͘ςετ
    apexͰdeploy
    • build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production
    • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ
    • zipΛ࡞Δͱ͖ʹsymlink΋ḷͬͯ͘ΕΔ
    functions
    diamond-writer
    lib
    diamondwriter.js
    index.js
    metriccache.js
    metriccache.spec.js
    package.json
    diamondwriter.spec.js
    diamond-writer-backup
    index.js

    View Slide

  27. Lambda functionͷ
    ύϑΥʔϚϯεɾνϡʔχϯά

    View Slide

  28. τϥϒϧൃੜʂ
    ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ
    ૝ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ
    ࢥͬͨΑΓ͓஋ஈ͕ߴ͍

    View Slide

  29. ܭଌͤΑʂ
    LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ
    top΋netstat΋tcpdump΋ଧͯͳ͍ʂ

    View Slide

  30. ͓͍ͪͭͯ CPU bound ͔ I/O bound ͔
    Έ͖ΘΊ·͠ΐ͏

    View Slide

  31. CPU bound ͔ I/O bound
    • Kinesis Streams͕triggerͷ࣌
    • Batch sizeΛ૿΍ͨ͠ͱ͖ͷdurationͷԠ౴
    • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷ͸ͣ
    • ઢܗԠ౴ͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧ഑త (͔΋)
    • Ұൠʹ͸…ʁ
    • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ౴
    • I/O͕ࢧ഑తͳΒCPU্͕͕ͬͯ΋ͦ͜·ͰԼ͕Βͳ͍͸ͣ
    • యܕతͳϦΫΤετͷlatency΍duration͔Β௚ײతʹ͸Θ͔Γͦ͏

    View Slide

  32. ύϑΥʔϚϯεܭଌͱվળ
    CPU boundͷͱ͖
    • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ๿)
    • खݩͰܭଌ͢Δ
    • sjsp (Simple JavaScript Profiler)
    վળ
    • ϝϞϦʔΛ૿΍͢ (CPUεϖοΫ΋޲্)
    • ΞϧΰϦζϜɾॲཧ಺༰ͷվળ

    View Slide

  33. sjsp
    ੿࡞ “Simple JavaScript Profiler”
    • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ
    • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥ΢βʹґଘ͠ͳ͍)
    • Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ΋࢖͑Δ
    CPU boundͳॏ͍ॲཧ͕෼͔Δ
    • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ
    ʮ͍ͭ΋ͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ

    View Slide

  34. ύϑΥʔϚϯεܭଌͱվળ
    I/O boundͷͱ͖
    • AWS X-RayͰॏ͍ϦΫΤετΛௐ΂Δ
    վળ
    • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢
    • ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ
    • Batch sizeΛ૿΍͢

    View Slide

  35. AWS X-Ray
    ϚωʔδυαʔϏε΁ͷϦΫΤετͷ৘ใΛऩूɾ෼ੳ
    AWSͷSDKʹ૊ΈࠐΉ
    • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ
    • I/O boundͷ৔߹ʹ໰୊ΛൃݟͰ͖Δ
    const AWSXRay = require('aws-xray-sdk-core');
    const AWS = AWSXRay.captureAWS(require('aws-sdk'));

    View Slide

  36. ֤ίϯϙʔωϯτͷಛੑͱ؂ࢹ

    View Slide

  37. Mackerel plugins
    ͸ͯͳʹ͓͚ΔAWSαʔϏεͷ؂ࢹ
    Monitoring host
    Mackerelͷpluginͱ
    AWSΠϯςάϨʔγϣϯͰ
    ϝτϦοΫΛऩूɾ ՄࢹԽ
    CloudWatch
    Application servers
    DB, Proxies, etc.
    #mackerel
    #bookmark
    #blog
    E-mail
    AWS Integration
    CloudWatchͷϝτϦοΫ΋Mackerelʹू໿
    ϗετ؅ཧɾΞϥʔτɾ௨஌؅ཧΛҰݩԽ
    EC2
    ΦϯϓϨͷϗετ΋
    ͨ͘͞Μ͋Δ

    View Slide

  38. diamondߏ੒ਤ
    DynamoDB
    Kinesis Steams
    S3
    Lambda
    Redis cluster
    Lambda
    TTL expired
    Application servers
    diamond-reader
    mackerel.io
    diamond-writer

    View Slide

  39. Kinesis Streamsͷಛੑ
    Kinesis Steams
    Lambda
    #Shard
    Records
    PutRecords
    Partition Key
    γϟʔυΛݻఆ͍ͨ࣌͠͸
    Partition KeyΛࢦఆ
    ϝτϦοΫ໊Λݩʹੜ੒
    ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ
    GetShardIterator
    GetRecords
    γϟʔυ਺ * ར༻࣌ؒ
    PUTϖΠϩʔυαΠζ
    σʔλอ࣋ظؒ (1d ~ 7d)
    ྉۚମܥ

    View Slide

  40. Kinesis Streamsͷ؂ࢹ
    I/O bytes͕Լ͕͍ͬͯΔ
    • PutRecordsʹࣦഊ͍ͯ͠Δ
    • ຊମͷΞϓϦέʔγϣϯΛٙ͏
    • GetRecordsʹࣦഊ͍ͯ͠Δ
    • ޙஈͷLambdaΛٙ͏
    Read delay্͕͕͍ͬͯΔ
    • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍
    • LambdaͷΤϥʔɾλΠϜΞ΢τ
    • γϟʔυ਺͕଍Γͳ͍
    • γϟʔυ਺Λ૿΍ͯ͠

    throughputΛ্͛Δ
    Read delay
    I/O bytes

    View Slide

  41. Lambdaͷಛੑ
    FaaS (Function as a Service)
    ֤ίϯςφ͸جຊతʹ࠶ར༻͞ΕΔ
    • deploy΍ઃఆมߋͰ࡞Γ௚͞ΕΔ
    • client͸globalม਺ͰӬଓԽ
    ͦΕͧΕͷίϯςφ͸ಉ࣌ʹ1ͭॲཧ
    • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ
    Kinesis Steams Lambda
    GetShardIterator
    GetRecords
    IteratorType
    Batch Size
    DynamoDB
    Redis cluster
    ֤container͸handlerΛ
    ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍
    ϝϞϦʔαΠζ (ॲཧೳྗ)
    ߹ܭ࣮ߦ࣌ؒ
    ϦΫΤετճ਺
    ྉۚମܥ

    View Slide

  42. Lambdaͷ؂ࢹ
    Iterator Age (ms)
    ࣮ߦॲཧ࣌ؒ (ms)
    ࣮ߦճ਺ɾΤϥʔճ਺
    Τϥʔճ਺্͕͕͍ͬͯΔ
    • CloudWatch LogsͰௐࠪ
    • apex logs —follow | grep
    Duration avg (ms) ্͕͕͍ͬͯΔ
    • ຊମଆͷෛՙɾKinesisνΣοΫ
    • DynamoDBͷRequest latency
    • ϝϞϦʔΛ૿΍ͦ͏
    • CPUεϖοΫ΋ྉۚ΋͕͋Δ
    Iterator Age্͕͕͍ͬͯΔ
    • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔
    • ޙஈͷthroughput଍Γͳ͍
    • DynamoDBͷCapacity্͛Δ
    • Kinesis shard਺Λ૿΍͢

    View Slide

  43. Lambdaͷ؂ࢹ
    ฏۉಉ࣌ॲཧ਺ = 1෼ؒͷ࣮ߦճ਺ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)

    ≤ Kinesis shard਺
    time
    1 minute
    container
    processing
    idle
    Mackerel advanced graph
    Shard਺ͷ7ׂҎԼ(໨҆)ʹ཈͑Δ
    ShardΛ૿΍͢ͷ͸͕͔͔࣌ؒΔ
    ੵ෼஋͕՝ۚର৅
    ύϑΥʔϚϯενϡʔχϯάͷࢦඪ
    Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count

    View Slide

  44. DynamoDBͷಛੑ
    DynamoDB
    Lambda
    diamond-reader
    BatchWriteItem
    BatchGetItem
    ϚωʔδυͳNoSQLσʔλϕʔε
    ʮΫϥ΢υ൛ͷ෼ࢄσʔλϕʔεʯ
    ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ
    3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ
    {Read,Write} ΩϟύγςΟ
    σʔλసૹྔ
    σʔλอଘ༰ྔ
    ྉۚମܥ

    View Slide

  45. DynamoDBͷ؂ࢹ
    Write capacity Read capacity Throttled requests
    Write Capacity্͕͕͍ͬͯΔ
    • LatencyͷѱԽɾ400Λฦ͢
    • Throttled requests͕Ͱ͍ͯΕ͹

    Provisioned Write Capacity͋͛Δ
    Read Capacity্͕͕͍ͬͯΔ
    • Writeͱಉ༷ɾ400Λฦ͢
    • Provisioned Read Capacity͋͛Δ
    Throttle events্͕͕͍ͬͯΔ
    • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ
    • Provisioned Capacity͕଍Γͳ͍
    • ϦΫΤετΛΏͬ͘Γʹ͢Δ
    • ϦΫΤετΛ·ͱΊΔ

    View Slide

  46. Redisͷ؂ࢹ
    CPU % Redis used memory Redis processed commands
    CPU %্͕͕͍ͬͯΔ
    • Redis͸1 core͔͠࢖͑ͳ͍
    • readΫΤϦΛslaveʹ޲͚Δ
    • nodeΛ௥Ճͯ͠resharding
    memory࢖༻ྔ
    • ϋʔυ΢ΣΞͷݶք͕͋Δ
    • key͕ফ͑ͯ΋OSʹฦ͞ͳ͍
    • redis-cli info | grep used_memory
    • resharding
    ࣮ߦͨ͠ίϚϯυ਺
    • RedisͰॲཧͨ͠ྔͷࢦඪ
    ֤Πϯελϯεʹͯ
    RedisϓϥάΠϯͰՄࢹԽ

    View Slide

  47. diamond-readerͷಛੑ
    GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ
    ෳ਺ͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ
    ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ
    DynamoDB
    S3
    Redis cluster
    Application servers
    diamond-reader

    View Slide

  48. diamond-readerͷ؂ࢹ
    golang-stats-api-handlerͱ
    mackerel-plugin-gostatsͰ
    GoͷwebΞϓϦέʔγϣϯͷ
    ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ
    GC਺ ώʔϓ ϝϞϦʔ
    mallocɾfree goroutineɾCGO਺

    View Slide

  49. ࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ͸
    1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ
    2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ

    View Slide

  50. ॲཧ਺ ؂ࢹ
    Lambda͕ॲཧͨ͠ϝτϦοΫͷ਺ / 1m
    • RedisɾDynamoDBʹ௥Ճͨ͠ϝτϦοΫ਺
    • CloudWatchʹJSONͰ৘ใΛग़͢
    • ผͷLambda functionͰҰ෼ؒͷϩάΛूܭ (cron trigger)

    㱺 Mackerelʹ౤ߘ
    • ॲཧ਺͕Լ͕ͬͨΒΞϥʔτ
    • Redisʹॲཧ਺༻ͷΩʔΛҰ෼͝ͱʹ࡞ͬͯincrͯ͠΋Α͍

    View Slide

  51. End-to-end ؂ࢹ
    DynamoDB
    Kinesis Steams
    S3
    Lambda
    Redis cluster
    Lambda
    TTL expired
    Application servers
    diamond-reader
    mackerel.io
    ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ

    View Slide

  52. End-to-end ؂ࢹ
    DynamoDB
    Kinesis Steams
    S3
    Lambda
    Redis cluster
    Lambda
    TTL expired
    Application servers
    diamond-reader
    mackerel.io
    API
    ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
    Monitoring host

    View Slide

  53. End-to-end ؂ࢹ
    ؂ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛ౤ߘ
    sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ
    ͔͔ͬͨ࣌ؒΛMackerelʹ౤ߘɾ؂ࢹ
    ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2E͸ॏཁʂ

    View Slide

  54. ؂ࢹΛ࡞Δ
    ίϯϙʔωϯτΛ૊Έ߹Θͤͨ࣌͸

    શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱ͸Ͳ͏͍͏͜ͱ͔
    ܥશମͷՄ༻ੑͷ௚઀తͳূڌ
    ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾ؂ࢹ͠·͠ΐ͏

    View Slide

  55. ʮ؂ࢹΛ͢Δʯͱ͸
    ϝτϦοΫʹᮢ஋Λઃఆ͢Δ͜ͱ
    ͚ͩͰ͸ͳ͍

    View Slide

  56. ීஈ͔ΒோΊΔ
    Kinesis Iterator Age Lambda࣮ߦճ਺ Lambdaॲཧ࣌ؒ
    ߹ܭॲཧ਺ DynamoDB Write E2E
    MackerelͷάϥϑϘʔυ

    View Slide

  57. ؂ࢹ͢Δͱ͍͏͜ͱ͸
    ฏৗঢ়ଶΛ஌Γ
    ఆٛ͢Δͱ͍͏͜ͱ

    View Slide

  58. ·ͱΊ
    ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏
    ؂ࢹͷجૅ͸ฏৗঢ়ଶΛ஌Δ͜ͱ
    ܥશମͷՄ༻ੑΛ؂ࢹ͠Α͏
    ʮ؂ࢹΛ࡞Δʯ

    View Slide

  59. We are hiring!
    σΟϨΫλʔ
    CRE (Customer Reliability Engineer)
    WebΞϓϦέʔγϣϯΤϯδχΞ
    WebΦϖϨʔγϣϯΤϯδχΞ
    iPhoneɾAndroidΞϓϦΤϯδχΞ

    View Slide

  60. Thank you!

    View Slide