サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017

サーバレスアーキテクチャによる時系列データベースの構築と監視

Transcript

1. αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱ؂ࢹ גࣜձࣾ͸ͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo
2. ࣗݾ঺հ גࣜձࣾ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ޷͖: Rust, Haskell, Go, Scala, Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
3. αʔόʔϨεͷίϯϙʔωϯτΛ࢖ͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋
4. αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γ؂ࢹ͍ͯ͠Δਓ ✋
5. ؂ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ؂ࢹɾ؅ཧ͍ͯ͠Δ͔Λ͓఻͑͠·͢
6. ໨࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱ؂ࢹ
7. Mackerelʹ͍ͭͯ
8. Mackerel mackerel.io
9. Mackerel SaaSͷαʔόʔ؂ࢹɾ؅ཧπʔϧ ͸ͯͳࣾ಺ͷ؂ࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛ౤ߘ͢Δpushܕ νϟοτπʔϧ΍ߏ੒؅ཧπʔϧͱ૊Έ߹Θͤ Slack, HipChat, etc. Chef, Ansible, etc.
10. Mackerelུ֓ਤ CPUɾMemory࢖༻཰ OS৘ใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ؅ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ
11. Mackerelͷάϥϑ ϝτϦοΫͷσʔλ͸ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετ৘ใ΍ϝλσʔλ͸ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
12. ࣌ܥྻσʔλϕʔε
13. ࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹ͸਺஋σʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ • ߴղ૾౓ɾ௕ظอଘɾେྔϝτϦοΫ 㱺 ෼ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕࢖͍ͬͯ·ͨ͠
14. GraphiteΛӡ༻͢Δ্Ͱͷ՝୊ ϋʔυ΢ΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ೉ • ϝτϦοΫͷ௕ظอଘ (1m:25h 㱺 1m:400d+) σʔλϩετ଱ੑͷ௿͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ Ϋϥ΢υΛ׆༻ͯ͠ӡ༻ίετ࡟ݮ͍ͨ͠ Application servers mackerel.io
15. ৽࣌ܥྻDBͰୡ੒͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυ΢ΣΞௐୡίετ཈͑Δ σʔλϩετ଱ੑ
16. ৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετ࡟ݮ • ϚωʔδυɾαʔϏεΛ࢖͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ཈͍͑ͨ • RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ͸༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 ΁Ҡಈ σʔλϩετ଱ੑ • Amazon Kinesis Streams
17. ہॴࢀরੑͱετϨʔδͷબ୒ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ
18. ہॴࢀরੑͱετϨʔδͷબ୒ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛ૊Έ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
19. Aug. 2017 ৽࣌ܥྻDBʹҠߦ
20. diamondߏ੒ਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯ΋ߗ͍) DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
21. Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ
22. diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ࢖͍ॻ͖ࠐΈίετΛ࠷దԽ • ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫ͸ࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
23. Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠΋σʔλ͸Kinesis Streamsʹ࢒Δ • ো֐ൃੜ࣌఺͔Β࠶ॲཧͰ͖Δ • backup༻ʹผͷLambdaΛ࢖͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈ౳ੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো֐࣌ͷόοΫΞοϓܥ
24. Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛ૊ΜͰ͍Δ • ౰ॳ͸ElastiCacheΛ૝ఆ͍ͯͨ͠ • εέʔϧ {Ξοϓ,Ξ΢τ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕Ұఆݸ਺Ҏ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cron͸εέʔϧͤ͞ʹ͍͘ͷͰجຊతʹ͸࢖Θͳ͍ Lambda Redis cluster
25. Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫ஋͸Mapʹอଘ • BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacity͸جຊతʹҰఆ • Read capacity͸ΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
26. Lambda ίʔυߏ੒ɾdeploy ϩδοΫ͸libσΟϨΫτϦʹ੾Γग़͢ • function handlerΛ࠷খʹอͭ (client஫ೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy • build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlink΋ḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
27. Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά
28. τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ૝ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓஋ஈ͕ߴ͍
29. ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ top΋netstat΋tcpdump΋ଧͯͳ͍ʂ
30. ͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏
31. CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch sizeΛ૿΍ͨ͠ͱ͖ͷdurationͷԠ౴ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷ͸ͣ • ઢܗԠ౴ͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧ഑త (͔΋) • Ұൠʹ͸…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ౴ • I/O͕ࢧ഑తͳΒCPU্͕͕ͬͯ΋ͦ͜·ͰԼ͕Βͳ͍͸ͣ • యܕతͳϦΫΤετͷlatency΍duration͔Β௚ײతʹ͸Θ͔Γͦ͏
32. ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ๿) • खݩͰܭଌ͢Δ • sjsp (Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿΍͢ (CPUεϖοΫ΋޲্) • ΞϧΰϦζϜɾॲཧ಺༰ͷվળ
33. sjsp ੿࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥ΢βʹґଘ͠ͳ͍) • Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ΋࢖͑Δ CPU boundͳॏ͍ॲཧ͕෼͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭ΋ͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
34. ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐ΂Δ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ • ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿΍͢
35. AWS X-Ray ϚωʔδυαʔϏε΁ͷϦΫΤετͷ৘ใΛऩूɾ෼ੳ AWSͷSDKʹ૊ΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ৔߹ʹ໰୊ΛൃݟͰ͖Δ const AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));
36. ֤ίϯϙʔωϯτͷಛੑͱ؂ࢹ
37. Mackerel plugins ͸ͯͳʹ͓͚ΔAWSαʔϏεͷ؂ࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫ΋Mackerelʹू໿ ϗετ؅ཧɾΞϥʔτɾ௨஌؅ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ΋ ͨ͘͞Μ͋Δ
38. diamondߏ੒ਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
39. Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key γϟʔυΛݻఆ͍ͨ࣌͠͸ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ੒ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ਺ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
40. Kinesis Streamsͷ؂ࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ • ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞ΢τ • γϟʔυ਺͕଍Γͳ͍ • γϟʔυ਺Λ૿΍ͯ͠
 throughputΛ্͛Δ Read delay I/O bytes
41. Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφ͸جຊతʹ࠶ར༻͞ΕΔ • deploy΍ઃఆมߋͰ࡞Γ௚͞ΕΔ • client͸globalม਺ͰӬଓԽ ͦΕͧΕͷίϯςφ͸ಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤container͸handlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ਺ ྉۚମܥ
42. Lambdaͷ؂ࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճ਺ɾΤϥʔճ਺ Τϥʔճ਺্͕͕͍ͬͯΔ • CloudWatch LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿΍ͦ͏ • CPUεϖοΫ΋ྉۚ΋͕͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughput଍Γͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shard਺Λ૿΍͢
43. Lambdaͷ؂ࢹ ฏۉಉ࣌ॲཧ਺ = 1෼ؒͷ࣮ߦճ਺ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)
 ≤ Kinesis shard਺ time 1 minute container processing idle Mackerel advanced graph Shard਺ͷ7ׂҎԼ(໨҆)ʹ཈͑Δ ShardΛ૿΍͢ͷ͸͕͔͔࣌ؒΔ ੵ෼஋͕՝ۚର৅ ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
44. DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥ΢υ൛ͷ෼ࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ {Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ
45. DynamoDBͷ؂ࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ • LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ͹
 Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕଍Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
46. Redisͷ؂ࢹ CPU % Redis used memory Redis processed commands CPU %্͕͕͍ͬͯΔ • Redis͸1 core͔͠࢖͑ͳ͍ • readΫΤϦΛslaveʹ޲͚Δ • nodeΛ௥Ճͯ͠resharding memory࢖༻ྔ • ϋʔυ΢ΣΞͷݶք͕͋Δ • key͕ফ͑ͯ΋OSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ਺ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
47. diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳ਺ͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers diamond-reader
48. diamond-readerͷ؂ࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC਺ ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO਺
49. ࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ͸ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
50. ॲཧ਺ ؂ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ਺ / 1m • RedisɾDynamoDBʹ௥Ճͨ͠ϝτϦοΫ਺ • CloudWatchʹJSONͰ৘ใΛग़͢ • ผͷLambda functionͰҰ෼ؒͷϩάΛूܭ (cron trigger)
 㱺 Mackerelʹ౤ߘ • ॲཧ਺͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ਺༻ͷΩʔΛҰ෼͝ͱʹ࡞ͬͯincrͯ͠΋Α͍
51. End-to-end ؂ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
52. End-to-end ؂ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host
53. End-to-end ؂ࢹ ؂ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛ౤ߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹ౤ߘɾ؂ࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2E͸ॏཁʂ
54. ؂ࢹΛ࡞Δ ίϯϙʔωϯτΛ૊Έ߹Θͤͨ࣌͸
 શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱ͸Ͳ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷ௚઀తͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾ؂ࢹ͠·͠ΐ͏
55. ʮ؂ࢹΛ͢Δʯͱ͸ ϝτϦοΫʹᮢ஋Λઃఆ͢Δ͜ͱ ͚ͩͰ͸ͳ͍
56. ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ਺ Lambdaॲཧ࣌ؒ ߹ܭॲཧ਺ DynamoDB Write E2E MackerelͷάϥϑϘʔυ
57. ؂ࢹ͢Δͱ͍͏͜ͱ͸ ฏৗঢ়ଶΛ஌Γ ఆٛ͢Δͱ͍͏͜ͱ
58. ·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ؂ࢹͷجૅ͸ฏৗঢ়ଶΛ஌Δ͜ͱ ܥશମͷՄ༻ੑΛ؂ࢹ͠Α͏ ʮ؂ࢹΛ࡞Δʯ
59. We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ iPhoneɾAndroidΞϓϦΤϯδχΞ
60. Thank you!