サーバレスアーキテクチャによる時系列データベースの構築と監視
αʔόϨεΞʔΩςΫνϟʹΑΔ࣌ܥྻσʔλϕʔεͷߏஙͱࢹגࣜձࣾͯͳɹ⅓ా ݈2017/11/03 Serverlessconf Tokyo
View Slide
ࣗݾհגࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞMackerelνʔϜ ςοΫϦʔυ͖: Rust, Haskell, Go, Scala, VimϓϩάϥϜϞάϞά itchyny.hatenablog.comlightline.vim, calendar.vim, etc.Thank you! ⅓ా ݈ Ken Hamada
αʔόʔϨεͷίϯϙʔωϯτΛͬͯαʔϏεΛӡ༻͍ͯ͠Δਓ ✋
αʔόʔϨεͷίϯϙʔωϯτΛ͔ͬ͠Γࢹ͍ͯ͠Δਓ ✋
ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪͲͷΑ͏ʹϚωʔδυαʔϏεΛࢹɾཧ͍ͯ͠Δ͔Λ͓͑͠·͢
࣍Mackerelʹ͍ͭͯ࣌ܥྻσʔλϕʔεMackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃLambda functionͷύϑΥʔϚϯεɾνϡʔχϯά֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerelʹ͍ͭͯ
Mackerelmackerel.io
MackerelSaaSͷαʔόʔࢹɾཧπʔϧͯͳࣾͷࢹγεςϜΛαʔϏεԽͯ͠ެ։mackerel-agent͕ϝτϦοΫΛߘ͢ΔpushܕνϟοτπʔϧߏཧπʔϧͱΈ߹ΘͤSlack, HipChat, etc. Chef, Ansible, etc.
Mackerelུ֓ਤCPUɾMemory༻OSใϝτϦοΫσʔλϝλσʔλϝʔϧཧը໘νϟοτπʔϧΞϥʔτൃੜ
MackerelͷάϥϑϝτϦοΫͷσʔλ࣌ܥྻσʔλϕʔεʹอଘ͞Ε͍ͯΔϗετใϝλσʔλϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
࣌ܥྻσʔλϕʔε
࣌ܥྻσʔλϕʔε• ࣌ܥྻͷσʔλ (యܕతʹσʔλ) ͷอଘʹಛԽ• αʔόʔϞχλϦϯάɾIoT༻్• ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ• ߴղ૾ɾظอଘɾେྔϝτϦοΫ 㱺 ࢄDB• ྫ: Graphite, InfluxDB, Prometheus, OpenTSDBMackerel͕͍ͬͯ·ͨ͠
GraphiteΛӡ༻͢Δ্Ͱͷ՝ϋʔυΣΞͷݶք (disk༰ྔ)• ετϨʔδͷௐୡίετεέʔϥϏϦςΟʔͷ֬อ͕ࠔ• ϝτϦοΫͷظอଘ (1m:25h 㱺 1m:400d+)σʔλϩετੑͷ͞ϋοΫɾ࠷దԽ͠ʹ͘͞ΫϥυΛ׆༻ͯ͠ӡ༻ίετݮ͍ͨ͠Application serversmackerel.io
৽࣌ܥྻDBͰୡ͍ͨ͜͠ͱεέʔϥϏϦςΟʔӡ༻ίετɾϋʔυΣΞௐୡίετ͑Δσʔλϩετੑ
৽࣌ܥྻDBͷઃܭεέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετݮ• ϚωʔδυɾαʔϏεΛ͏ 㱺 AWS DynamoDBDynamoDBͷॻ͖ࠐΈίετ (pricing) Λ͍͑ͨ• RedisΛΩϟογϡͱͯ͠ར༻ݹ͍ϝτϦοΫσʔλ༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ• DynamoDB 㱺 S3 Ҡಈσʔλϩετੑ• Amazon Kinesis Streams
ہॴࢀরੑͱετϨʔδͷબසൟʹࢀর͞ΕΔ৽͍͠ϝτϦοΫݹ͍͋·Γࢀর͞Εͳ͍ಡΈॻ͖ίετอ࣋ίετDynamoDBS3 RedisϨΠςϯγεέʔϥϏϦςΟʔ༰ྔ୯Ձ
ہॴࢀরੑͱετϨʔδͷબසൟʹࢀর͞ΕΔ৽͍͠ϝτϦοΫݹ͍͋·Γࢀর͞Εͳ͍ಡΈॻ͖ίετอ࣋ίετDynamoDBS3 RedisϨΠςϯγεέʔϥϏϦςΟʔ༰ྔ୯ՁಛੑͷҟͳΔετϨʔδΛΈ߹ΘͤΔ͜ͱͰ͔͔ΔίετΛ࠷దԽ͢Δ
Aug. 2017 ৽࣌ܥྻDBʹҠߦ
diamondߏਤMackerelͷ৽࣌ܥྻσʔλϕʔεdiamond: graphiteͷಉૉମ (ͱͯߗ͍)DynamoDBKinesis SteamsS3LambdaRedis clusterLambdaTTL expiredApplication serversdiamond-readermackerel.iodiamond-writer
Mackerelͷ৽࣌ܥྻσʔλϕʔεdiamond-writerͷ։ൃ
diamond-writer• ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ• AWS Lambda (node.js)• RedisΛ͍ॻ͖ࠐΈίετΛ࠷దԽ• ΞΠςϜʹTTLΛઃఆ• ݹ͍ϝτϦοΫࣗಈͰফ͑Δ DynamoDBKinesis SteamsRedis clusterLambda
Kinesis Streams→Lambda• Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ• Lambda͕ॲཧͯ͠σʔλKinesis StreamsʹΔ• োൃੜ͔࣌Β࠶ॲཧͰ͖Δ• backup༻ʹผͷLambdaΛ͏• ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ• Lambdaͷॲཧͷႈੑ͕ॏཁKinesis Steams LambdaLambda backupTRIM_HORIZONAT_TIMESTAMPো࣌ͷόοΫΞοϓܥ
Lambda→Redis cluster• DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻• EC2্ͰclusterΛΜͰ͍Δ• ॳElastiCacheΛఆ͍ͯͨ͠• εέʔϧ {Ξοϓ,Ξτ} ͕ΦϯϥΠϯͰͰ͖ͳ͍• Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢• ϝτϦοΫ͕ҰఆݸҎ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ• cronεέʔϧͤ͞ʹ͍͘ͷͰجຊతʹΘͳ͍LambdaRedis cluster
Lambda→DynamoDB• MetricName (partition key)ɾTimestamp (sort key)• ϝτϦοΫMapʹอଘ• BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ• Write capacityجຊతʹҰఆ• Read capacityΞΫηε͕ूத͢Δ͜ͱ͕͋ΔDynamoDBLambda
Lambda ίʔυߏɾdeployϩδοΫlibσΟϨΫτϦʹΓग़͢• function handlerΛ࠷খʹอͭ (clientೖɾcallback)• ֤ϥΠϒϥϦʔΛखް͘ςετapexͰdeploy• build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production• σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ• zipΛ࡞Δͱ͖ʹsymlinkḷͬͯ͘ΕΔfunctionsdiamond-writerlibdiamondwriter.jsindex.jsmetriccache.jsmetriccache.spec.jspackage.jsondiamondwriter.spec.jsdiamond-writer-backupindex.js
Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά
τϥϒϧൃੜʂϦΫΤετΛฒߦॲཧʹ͍ͯ͠ΔͷʹɺఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂࢥͬͨΑΓ͓ஈ͕ߴ͍
ܭଌͤΑʂLambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂtopnetstattcpdumpଧͯͳ͍ʂ
͓͍ͪͭͯ CPU bound ͔ I/O bound ͔Έ͖ΘΊ·͠ΐ͏
CPU bound ͔ I/O bound• Kinesis Streams͕triggerͷ࣌• Batch sizeΛ૿ͨ͠ͱ͖ͷdurationͷԠ• I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷͣ• ઢܗԠͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧత (͔)• Ұൠʹ…ʁ• ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ• I/O͕ࢧతͳΒCPU্͕͕ͬͯͦ͜·ͰԼ͕Βͳ͍ͣ• యܕతͳϦΫΤετͷlatencyduration͔ΒײతʹΘ͔Γͦ͏
ύϑΥʔϚϯεܭଌͱվળCPU boundͷͱ͖• ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ)• खݩͰܭଌ͢Δ• sjsp (Simple JavaScript Profiler)վળ• ϝϞϦʔΛ૿͢ (CPUεϖοΫ্)• ΞϧΰϦζϜɾॲཧ༰ͷվળ
sjsp࡞ “Simple JavaScript Profiler”• MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ• JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥβʹґଘ͠ͳ͍)• Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ͑ΔCPU boundͳॏ͍ॲཧ͕͔Δ• ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨʮ͍ͭͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
ύϑΥʔϚϯεܭଌͱվળI/O boundͷͱ͖• AWS X-RayͰॏ͍ϦΫΤετΛௐΔվળ• ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢• ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ• Batch sizeΛ૿͢
AWS X-RayϚωʔδυαʔϏεͷϦΫΤετͷใΛऩूɾੳAWSͷSDKʹΈࠐΉ• ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ• I/O boundͷ߹ʹΛൃݟͰ͖Δconst AWSXRay = require('aws-xray-sdk-core');const AWS = AWSXRay.captureAWS(require('aws-sdk'));
֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerel pluginsͯͳʹ͓͚ΔAWSαʔϏεͷࢹMonitoring hostMackerelͷpluginͱAWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऩूɾ ՄࢹԽCloudWatchApplication serversDB, Proxies, etc.#mackerel#bookmark#blogE-mailAWS IntegrationCloudWatchͷϝτϦοΫMackerelʹूϗετཧɾΞϥʔτɾ௨ཧΛҰݩԽEC2ΦϯϓϨͷϗετͨ͘͞Μ͋Δ
diamondߏਤDynamoDBKinesis SteamsS3LambdaRedis clusterLambdaTTL expiredApplication serversdiamond-readermackerel.iodiamond-writer
Kinesis StreamsͷಛੑKinesis SteamsLambda#ShardRecordsPutRecordsPartition KeyγϟʔυΛݻఆ͍ͨ࣌͠Partition KeyΛࢦఆϝτϦοΫ໊Λݩʹੜਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄGetShardIteratorGetRecordsγϟʔυ * ར༻࣌ؒPUTϖΠϩʔυαΠζσʔλอ࣋ظؒ (1d ~ 7d)ྉۚମܥ
Kinesis StreamsͷࢹI/O bytes͕Լ͕͍ͬͯΔ• PutRecordsʹࣦഊ͍ͯ͠Δ• ຊମͷΞϓϦέʔγϣϯΛٙ͏• GetRecordsʹࣦഊ͍ͯ͠Δ• ޙஈͷLambdaΛٙ͏Read delay্͕͕͍ͬͯΔ• Lambda͕ॲཧ͖͠Ε͍ͯͳ͍• LambdaͷΤϥʔɾλΠϜΞτ• γϟʔυ͕Γͳ͍• γϟʔυΛ૿ͯ͠ throughputΛ্͛ΔRead delayI/O bytes
LambdaͷಛੑFaaS (Function as a Service)֤ίϯςφجຊతʹ࠶ར༻͞ΕΔ• deployઃఆมߋͰ࡞Γ͞ΕΔ• clientglobalมͰӬଓԽͦΕͧΕͷίϯςφಉ࣌ʹ1ͭॲཧ• ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭKinesis Steams LambdaGetShardIteratorGetRecordsIteratorTypeBatch SizeDynamoDBRedis cluster֤containerhandlerΛಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ϝϞϦʔαΠζ (ॲཧೳྗ)߹ܭ࣮ߦ࣌ؒϦΫΤετճྉۚମܥ
LambdaͷࢹIterator Age (ms)࣮ߦॲཧ࣌ؒ (ms)࣮ߦճɾΤϥʔճΤϥʔճ্͕͕͍ͬͯΔ• CloudWatch LogsͰௐࠪ• apex logs —follow | grepDuration avg (ms) ্͕͕͍ͬͯΔ• ຊମଆͷෛՙɾKinesisνΣοΫ• DynamoDBͷRequest latency• ϝϞϦʔΛ૿ͦ͏• CPUεϖοΫྉ͕ۚ͋ΔIterator Age্͕͕͍ͬͯΔ• Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔• ޙஈͷthroughputΓͳ͍• DynamoDBͷCapacity্͛Δ• Kinesis shardΛ૿͢
Lambdaͷࢹฏۉಉ࣌ॲཧ = 1ؒͷ࣮ߦճ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms) ≤ Kinesis shardtime1 minutecontainerprocessingidleMackerel advanced graphShardͷ7ׂҎԼ(҆)ʹ͑ΔShardΛ૿͢ͷ͕͔͔࣌ؒΔੵ͕՝ۚରύϑΥʔϚϯενϡʔχϯάͷࢦඪInvocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
DynamoDBͷಛੑDynamoDBLambdadiamond-readerBatchWriteItemBatchGetItemϚωʔδυͳNoSQLσʔλϕʔεʮΫϥυ൛ͷࢄσʔλϕʔεʯΦϯϥΠϯͰΩϟύγςΟʔΛมߋ3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ{Read,Write} ΩϟύγςΟσʔλసૹྔσʔλอଘ༰ྔྉۚମܥ
DynamoDBͷࢹWrite capacity Read capacity Throttled requestsWrite Capacity্͕͕͍ͬͯΔ• LatencyͷѱԽɾ400Λฦ͢• Throttled requests͕Ͱ͍ͯΕ Provisioned Write Capacity͋͛ΔRead Capacity্͕͕͍ͬͯΔ• Writeͱಉ༷ɾ400Λฦ͢• Provisioned Read Capacity͋͛ΔThrottle events্͕͕͍ͬͯΔ• Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ• Provisioned Capacity͕Γͳ͍• ϦΫΤετΛΏͬ͘Γʹ͢Δ• ϦΫΤετΛ·ͱΊΔ
RedisͷࢹCPU % Redis used memory Redis processed commandsCPU %্͕͕͍ͬͯΔ• Redis1 core͔͑͠ͳ͍• readΫΤϦΛslaveʹ͚Δ• nodeΛՃͯ͠reshardingmemory༻ྔ• ϋʔυΣΞͷݶք͕͋Δ• key͕ফ͑ͯOSʹฦ͞ͳ͍• redis-cli info | grep used_memory• resharding࣮ߦͨ͠ίϚϯυ• RedisͰॲཧͨ͠ྔͷࢦඪ֤ΠϯελϯεʹͯRedisϓϥάΠϯͰՄࢹԽ
diamond-readerͷಛੑGoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯෳͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽDynamoDBS3Redis clusterApplication serversdiamond-reader
diamond-readerͷࢹgolang-stats-api-handlerͱmackerel-plugin-gostatsͰGoͷwebΞϓϦέʔγϣϯͷϞχλϦϯάΛ࢝ΊΒΕΔΑʂGC ώʔϓ ϝϞϦʔmallocɾfree goroutineɾCGO
࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
ॲཧ ࢹLambda͕ॲཧͨ͠ϝτϦοΫͷ / 1m• RedisɾDynamoDBʹՃͨ͠ϝτϦοΫ• CloudWatchʹJSONͰใΛग़͢• ผͷLambda functionͰҰؒͷϩάΛूܭ (cron trigger) 㱺 Mackerelʹߘ• ॲཧ͕Լ͕ͬͨΒΞϥʔτ• Redisʹॲཧ༻ͷΩʔΛҰ͝ͱʹ࡞ͬͯincrͯ͠Α͍
End-to-end ࢹDynamoDBKinesis SteamsS3LambdaRedis clusterLambdaTTL expiredApplication serversdiamond-readermackerel.ioϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
End-to-end ࢹDynamoDBKinesis SteamsS3LambdaRedis clusterLambdaTTL expiredApplication serversdiamond-readermackerel.ioAPIϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒMonitoring host
End-to-end ࢹࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛߘsleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ͔͔ͬͨ࣌ؒΛMackerelʹߘɾࢹܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2Eॏཁʂ
ࢹΛ࡞ΔίϯϙʔωϯτΛΈ߹Θͤͨ࣌ શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱͲ͏͍͏͜ͱ͔ܥશମͷՄ༻ੑͷతͳূڌܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾࢹ͠·͠ΐ͏
ʮࢹΛ͢ΔʯͱϝτϦοΫʹᮢΛઃఆ͢Δ͜ͱ͚ͩͰͳ͍
ීஈ͔ΒோΊΔKinesis Iterator Age Lambda࣮ߦճ Lambdaॲཧ࣌ؒ߹ܭॲཧ DynamoDB Write E2EMackerelͷάϥϑϘʔυ
ࢹ͢Δͱ͍͏͜ͱฏৗঢ়ଶΛΓఆٛ͢Δͱ͍͏͜ͱ
·ͱΊϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ࢹͷجૅฏৗঢ়ଶΛΔ͜ͱܥશମͷՄ༻ੑΛࢹ͠Α͏ʮࢹΛ࡞Δʯ
We are hiring!σΟϨΫλʔCRE (Customer Reliability Engineer)WebΞϓϦέʔγϣϯΤϯδχΞWebΦϖϨʔγϣϯΤϯδχΞiPhoneɾAndroidΞϓϦΤϯδχΞ
Thank you!
itchyny
oracle4engineer
k1low
toshi0383
poteboy
nishiuma
yokawasa
smasato
coconala_engineer
akiroom
naohachi89
is_ryo
s2terminal
jakevdp
brianwarren
soudai
mza
thekraken
davidbonilla
cherdarchuk
addyosmani
paulrobertlloyd
sferik
kastner
dougneiner