サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017

Transcript

1. αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹ גࣜձࣾ͸ͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo

2. ࣗݾ঺հ גࣜձࣾ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ޷͖: Rust, Haskell, Go, Scala,

   Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada

3. αʔόʔϨεͷίϯϙʔωϯτΛ࢖ͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋

4. αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γ؂ࢹ͍ͯ͠Δਓ ✋

5. ؂ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ؂ࢹɾ؅ཧ͍ͯ͠Δ͔Λ͓఻͑͠·͢

6. ໨࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱ؂ࢹ

7. Mackerelʹ͍ͭͯ

8. Mackerel mackerel.io

9. Mackerel SaaSͷαʔόʔ؂ࢹɾ؅ཧπʔϧ ͸ͯͳࣾ಺ͷ؂ࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛ౤ߘ͢Δpushܕ νϟοτπʔϧ΍ߏ੒؅ཧπʔϧͱ૊Έ߹Θͤ Slack, HipChat, etc. Chef, Ansible,

   etc.

10. Mackerelུ֓ਤ CPUɾMemory࢖༻཰ OS৘ใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ؅ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ

11. Mackerelͷάϥϑ ϝτϦοΫͷσʔλ͸ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετ৘ใ΍ϝλσʔλ͸ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ

12. ࣌ܥྻσʔλϕʔε

13. ࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹ͸਺஋σʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •

    ߴղ૾౓ɾ௕ظอଘɾେྔϝτϦοΫ 㱺 ෼ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕࢖͍ͬͯ·ͨ͠

14. GraphiteΛӡ༻͢Δ্Ͱͷ՝୊ ϋʔυ΢ΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ೉ • ϝτϦοΫͷ௕ظอଘ (1m:25h 㱺

    1m:400d+) σʔλϩετ଱ੑͷ௿͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ Ϋϥ΢υΛ׆༻ͯ͠ӡ༻ίετ࡟ݮ͍ͨ͠ Application servers mackerel.io

15. ৽࣌ܥྻDBͰୡ੒͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυ΢ΣΞௐୡίετ཈͑Δ σʔλϩετ଱ੑ

16. ৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετ࡟ݮ • ϚωʔδυɾαʔϏεΛ࢖͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ཈͍͑ͨ

    • RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ͸༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 ΁Ҡಈ σʔλϩετ଱ੑ • Amazon Kinesis Streams

17. ہॴࢀরੑͱετϨʔδͷબ୒ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis

    ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ

18. ہॴࢀরੑͱετϨʔδͷબ୒ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis

    ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛ૊Έ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ

19. Aug. 2017 ৽࣌ܥྻDBʹҠߦ

20. diamondߏ੒ਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯ΋ߗ͍) DynamoDB Kinesis Steams S3 Lambda

    Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer

21. Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ

22. diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ࢖͍ॻ͖ࠐΈίετΛ࠷దԽ •

    ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫ͸ࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda

23. Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠΋σʔλ͸Kinesis Streamsʹ࢒Δ • ো֐ൃੜ࣌఺͔Β࠶ॲཧͰ͖Δ

    • backup༻ʹผͷLambdaΛ࢖͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈ౳ੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো֐࣌ͷόοΫΞοϓܥ

24. Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛ૊ΜͰ͍Δ • ౰ॳ͸ElastiCacheΛ૝ఆ͍ͯͨ͠ •

    εέʔϧ {Ξοϓ,Ξ΢τ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕Ұఆݸ਺Ҏ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cron͸εέʔϧͤ͞ʹ͍͘ͷͰجຊతʹ͸࢖Θͳ͍ Lambda Redis cluster

25. Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫ஋͸Mapʹอଘ •

    BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacity͸جຊతʹҰఆ • Read capacity͸ΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda

26. Lambda ίʔυߏ੒ɾdeploy ϩδοΫ͸libσΟϨΫτϦʹ੾Γग़͢ • function handlerΛ࠷খʹอͭ (client஫ೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy

    • build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlink΋ḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js

27. Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά

28. τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ૝ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓஋ஈ͕ߴ͍

29. ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ top΋netstat΋tcpdump΋ଧͯͳ͍ʂ

30. ͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏

31. CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch

    sizeΛ૿΍ͨ͠ͱ͖ͷdurationͷԠ౴ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷ͸ͣ • ઢܗԠ౴ͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧ഑త (͔΋) • Ұൠʹ͸…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ౴ • I/O͕ࢧ഑తͳΒCPU্͕͕ͬͯ΋ͦ͜·ͰԼ͕Βͳ͍͸ͣ • యܕతͳϦΫΤετͷlatency΍duration͔Β௚ײతʹ͸Θ͔Γͦ͏

32. ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ๿) • खݩͰܭଌ͢Δ • sjsp

    (Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿΍͢ (CPUεϖοΫ΋޲্) • ΞϧΰϦζϜɾॲཧ಺༰ͷվળ

33. sjsp ੿࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥ΢βʹґଘ͠ͳ͍)

    • Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ΋࢖͑Δ CPU boundͳॏ͍ॲཧ͕෼͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭ΋ͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ

34. ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐ΂Δ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ •

    ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿΍͢

35. AWS X-Ray ϚωʔδυαʔϏε΁ͷϦΫΤετͷ৘ใΛऩूɾ෼ੳ AWSͷSDKʹ૊ΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ৔߹ʹ໰୊ΛൃݟͰ͖Δ const

    AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));

36. ֤ίϯϙʔωϯτͷಛੑͱ؂ࢹ

37. Mackerel plugins ͸ͯͳʹ͓͚ΔAWSαʔϏεͷ؂ࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch

    Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫ΋Mackerelʹू໿ ϗετ؅ཧɾΞϥʔτɾ௨஌؅ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ΋ ͨ͘͞Μ͋Δ

38. diamondߏ੒ਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL

    expired Application servers diamond-reader mackerel.io diamond-writer

39. Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key

    γϟʔυΛݻఆ͍ͨ࣌͠͸ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ੒ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ਺ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ

40. Kinesis Streamsͷ؂ࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ

    • ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞ΢τ • γϟʔυ਺͕଍Γͳ͍ • γϟʔυ਺Λ૿΍ͯ͠
 throughputΛ্͛Δ Read delay I/O bytes

41. Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφ͸جຊతʹ࠶ར༻͞ΕΔ • deploy΍ઃఆมߋͰ࡞Γ௚͞ΕΔ •

    client͸globalม਺ͰӬଓԽ ͦΕͧΕͷίϯςφ͸ಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤container͸handlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ਺ ྉۚମܥ

42. Lambdaͷ؂ࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճ਺ɾΤϥʔճ਺ Τϥʔճ਺্͕͕͍ͬͯΔ • CloudWatch

    LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿΍ͦ͏ • CPUεϖοΫ΋ྉۚ΋͕͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughput଍Γͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shard਺Λ૿΍͢

43. Lambdaͷ؂ࢹ ฏۉಉ࣌ॲཧ਺ = 1෼ؒͷ࣮ߦճ਺ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)


    ≤ Kinesis shard਺ time 1 minute container processing idle Mackerel advanced graph Shard਺ͷ7ׂҎԼ(໨҆)ʹ཈͑Δ ShardΛ૿΍͢ͷ͸͕͔͔࣌ؒΔ ੵ෼஋͕՝ۚର৅ ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count

44. DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥ΢υ൛ͷ෼ࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ

    {Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ

45. DynamoDBͷ؂ࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •

    LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ͹
 Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕଍Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ

46. Redisͷ؂ࢹ CPU % Redis used memory Redis processed commands CPU

    %্͕͕͍ͬͯΔ • Redis͸1 core͔͠࢖͑ͳ͍ • readΫΤϦΛslaveʹ޲͚Δ • nodeΛ௥Ճͯ͠resharding memory࢖༻ྔ • ϋʔυ΢ΣΞͷݶք͕͋Δ • key͕ফ͑ͯ΋OSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ਺ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ

47. diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳ਺ͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers

    diamond-reader

48. diamond-readerͷ؂ࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC਺ ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO਺

49. ࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ͸ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ

50. ॲཧ਺ ؂ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ਺ / 1m • RedisɾDynamoDBʹ௥Ճͨ͠ϝτϦοΫ਺ • CloudWatchʹJSONͰ৘ใΛग़͢ •

    ผͷLambda functionͰҰ෼ؒͷϩάΛूܭ (cron trigger)
 㱺 Mackerelʹ౤ߘ • ॲཧ਺͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ਺༻ͷΩʔΛҰ෼͝ͱʹ࡞ͬͯincrͯ͠΋Α͍

51. End-to-end ؂ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda

    TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ

52. End-to-end ؂ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda

    TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host

53. End-to-end ؂ࢹ ؂ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛ౤ߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹ౤ߘɾ؂ࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2E͸ॏཁʂ

54. ؂ࢹΛ࡞Δ ίϯϙʔωϯτΛ૊Έ߹Θͤͨ࣌͸
 શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱ͸Ͳ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷ௚઀తͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾ؂ࢹ͠·͠ΐ͏

55. ʮ؂ࢹΛ͢Δʯͱ͸ ϝτϦοΫʹᮢ஋Λઃఆ͢Δ͜ͱ ͚ͩͰ͸ͳ͍

56. ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ਺ Lambdaॲཧ࣌ؒ ߹ܭॲཧ਺ DynamoDB Write E2E

    MackerelͷάϥϑϘʔυ

57. ؂ࢹ͢Δͱ͍͏͜ͱ͸ ฏৗঢ়ଶΛ஌Γ ఆٛ͢Δͱ͍͏͜ͱ

58. ·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ؂ࢹͷجૅ͸ฏৗঢ়ଶΛ஌Δ͜ͱ ܥશମͷՄ༻ੑΛ؂ࢹ͠Α͏ ʮ؂ࢹΛ࡞Δʯ

59. We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ

    iPhoneɾAndroidΞϓϦΤϯδχΞ

60. Thank you!