Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf...
Search
Ken Hamada
November 03, 2017
Technology
18
29k
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
サーバレスアーキテクチャによる時系列データベースの構築と監視
Ken Hamada
November 03, 2017
Tweet
Share
More Decks by Ken Hamada
See All by Ken Hamada
組織と技術の両輪で開発を加速させるkintoneチームの取り組み / JJUG CCC 2022 Fall Cybozu kintone
itchyny
1
2.2k
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
itchyny
0
5.5k
Other Decks in Technology
See All in Technology
より快適なエラーログ監視を目指して
leveragestech
3
610
セキュリティ監視の内製化 効率とリスク
mixi_engineers
PRO
7
880
突撃! 隣のAmazon Bedrockユーザー 〜YouはどうしてAWSで?〜
minorun365
PRO
3
250
すぐに始めるAWSコスト削減。短期でできる改善策と長期的な運用負荷軽減への取り組み方を解説
ncdc
1
540
#Zenoh 完全に理解した 〜組込み純情篇〜
takasehideki
1
470
サーバレスでモバイルアプリ開発! NTTコム「ビジネスdアプリ」のアーキテクチャ / The architecture of business d app
nttcom
10
180
DroidKaigi 2024 たすけて!ViewModel
mhidaka
4
410
強いチームを夢見て-PMからSREに転身して1年の振り返り / 20240906_bengo4_sre
bengo4com
2
820
Namespace, Now and Then
tagomoris
0
190
手軽に始める? おうちサーバーのすゝめ
nyagasan
0
190
CRTO/CRTL/OSEPの比較・勉強法とAV/EDRの検知実験
chayakonanaika
1
1k
Tricentisにおけるテスト自動化へのAI活用ご紹介/20240910Shunsuke Katakura
shift_evolve
0
150
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
30
2.2k
The Straight Up "How To Draw Better" Workshop
denniskardys
230
130k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
How to name files
jennybc
75
98k
How to train your dragon (web standard)
notwaldorf
85
5.6k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
24
3.9k
The Invisible Side of Design
smashingmag
295
50k
Code Review Best Practice
trishagee
62
16k
What’s in a name? Adding method to the madness
productmarketing
PRO
21
3k
KATA
mclloyd
27
13k
Design by the Numbers
sachag
277
19k
Optimizing for Happiness
mojombo
375
69k
Transcript
αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱࢹ גࣜձࣾͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo
ࣗݾհ גࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ͖: Rust, Haskell, Go, Scala,
Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
αʔόʔϨεͷίϯϙʔωϯτΛͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋
αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γࢹ͍ͯ͠Δਓ ✋
ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ࢹɾཧ͍ͯ͠Δ͔Λ͓͑͠·͢
࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerelʹ͍ͭͯ
Mackerel mackerel.io
Mackerel SaaSͷαʔόʔࢹɾཧπʔϧ ͯͳࣾͷࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛߘ͢Δpushܕ νϟοτπʔϧߏཧπʔϧͱΈ߹Θͤ Slack, HipChat, etc. Chef, Ansible,
etc.
Mackerelུ֓ਤ CPUɾMemory༻ OSใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ
Mackerelͷάϥϑ ϝτϦοΫͷσʔλ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετใϝλσʔλ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
࣌ܥྻσʔλϕʔε
࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹσʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •
ߴղ૾ɾظอଘɾେྔϝτϦοΫ 㱺 ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕͍ͬͯ·ͨ͠
GraphiteΛӡ༻͢Δ্Ͱͷ՝ ϋʔυΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ • ϝτϦοΫͷظอଘ (1m:25h 㱺
1m:400d+) σʔλϩετੑͷ͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ ΫϥυΛ׆༻ͯ͠ӡ༻ίετݮ͍ͨ͠ Application servers mackerel.io
৽࣌ܥྻDBͰୡ͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυΣΞௐୡίετ͑Δ σʔλϩετੑ
৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετݮ • ϚωʔδυɾαʔϏεΛ͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ͍͑ͨ
• RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 Ҡಈ σʔλϩετੑ • Amazon Kinesis Streams
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛΈ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
Aug. 2017 ৽࣌ܥྻDBʹҠߦ
diamondߏਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯߗ͍) DynamoDB Kinesis Steams S3 Lambda
Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ
diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ͍ॻ͖ࠐΈίετΛ࠷దԽ •
ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠σʔλKinesis StreamsʹΔ • োൃੜ͔࣌Β࠶ॲཧͰ͖Δ
• backup༻ʹผͷLambdaΛ͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো࣌ͷόοΫΞοϓܥ
Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛΜͰ͍Δ • ॳElastiCacheΛఆ͍ͯͨ͠ •
εέʔϧ {Ξοϓ,Ξτ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕ҰఆݸҎ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cronεέʔϧͤ͞ʹ͍͘ͷͰجຊతʹΘͳ͍ Lambda Redis cluster
Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫMapʹอଘ •
BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacityجຊతʹҰఆ • Read capacityΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
Lambda ίʔυߏɾdeploy ϩδοΫlibσΟϨΫτϦʹΓग़͢ • function handlerΛ࠷খʹอͭ (clientೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy
• build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlinkḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά
τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓ஈ͕ߴ͍
ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ topnetstattcpdumpଧͯͳ͍ʂ
͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏
CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch
sizeΛ૿ͨ͠ͱ͖ͷdurationͷԠ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷͣ • ઢܗԠͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧత (͔) • Ұൠʹ…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ • I/O͕ࢧతͳΒCPU্͕͕ͬͯͦ͜·ͰԼ͕Βͳ͍ͣ • యܕతͳϦΫΤετͷlatencyduration͔ΒײతʹΘ͔Γͦ͏
ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ) • खݩͰܭଌ͢Δ • sjsp
(Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿͢ (CPUεϖοΫ্) • ΞϧΰϦζϜɾॲཧ༰ͷվળ
sjsp ࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥβʹґଘ͠ͳ͍)
• Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ͑Δ CPU boundͳॏ͍ॲཧ͕͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐΔ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ •
ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿͢
AWS X-Ray ϚωʔδυαʔϏεͷϦΫΤετͷใΛऩूɾੳ AWSͷSDKʹΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ߹ʹΛൃݟͰ͖Δ const
AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));
֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerel plugins ͯͳʹ͓͚ΔAWSαʔϏεͷࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch
Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫMackerelʹू ϗετཧɾΞϥʔτɾ௨ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ ͨ͘͞Μ͋Δ
diamondߏਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL
expired Application servers diamond-reader mackerel.io diamond-writer
Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key
γϟʔυΛݻఆ͍ͨ࣌͠ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
Kinesis Streamsͷࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ
• ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞτ • γϟʔυ͕Γͳ͍ • γϟʔυΛ૿ͯ͠ throughputΛ্͛Δ Read delay I/O bytes
Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφجຊతʹ࠶ར༻͞ΕΔ • deployઃఆมߋͰ࡞Γ͞ΕΔ •
clientglobalมͰӬଓԽ ͦΕͧΕͷίϯςφಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤containerhandlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ ྉۚମܥ
Lambdaͷࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճɾΤϥʔճ Τϥʔճ্͕͕͍ͬͯΔ • CloudWatch
LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿ͦ͏ • CPUεϖοΫྉ͕ۚ͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughputΓͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shardΛ૿͢
Lambdaͷࢹ ฏۉಉ࣌ॲཧ = 1ؒͷ࣮ߦճ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)
≤ Kinesis shard time 1 minute container processing idle Mackerel advanced graph Shardͷ7ׂҎԼ(҆)ʹ͑Δ ShardΛ૿͢ͷ͕͔͔࣌ؒΔ ੵ͕՝ۚର ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥυ൛ͷࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ
{Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ
DynamoDBͷࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •
LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
Redisͷࢹ CPU % Redis used memory Redis processed commands CPU
%্͕͕͍ͬͯΔ • Redis1 core͔͑͠ͳ͍ • readΫΤϦΛslaveʹ͚Δ • nodeΛՃͯ͠resharding memory༻ྔ • ϋʔυΣΞͷݶք͕͋Δ • key͕ফ͑ͯOSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers
diamond-reader
diamond-readerͷࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO
࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
ॲཧ ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ / 1m • RedisɾDynamoDBʹՃͨ͠ϝτϦοΫ • CloudWatchʹJSONͰใΛग़͢ •
ผͷLambda functionͰҰؒͷϩάΛूܭ (cron trigger) 㱺 Mackerelʹߘ • ॲཧ͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ༻ͷΩʔΛҰ͝ͱʹ࡞ͬͯincrͯ͠Α͍
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host
End-to-end ࢹ ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹߘɾࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2Eॏཁʂ
ࢹΛ࡞Δ ίϯϙʔωϯτΛΈ߹Θͤͨ࣌ શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱͲ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷతͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾࢹ͠·͠ΐ͏
ʮࢹΛ͢Δʯͱ ϝτϦοΫʹᮢΛઃఆ͢Δ͜ͱ ͚ͩͰͳ͍
ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ Lambdaॲཧ࣌ؒ ߹ܭॲཧ DynamoDB Write E2E
MackerelͷάϥϑϘʔυ
ࢹ͢Δͱ͍͏͜ͱ ฏৗঢ়ଶΛΓ ఆٛ͢Δͱ͍͏͜ͱ
·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ࢹͷجૅฏৗঢ়ଶΛΔ͜ͱ ܥશମͷՄ༻ੑΛࢹ͠Α͏ ʮࢹΛ࡞Δʯ
We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ
iPhoneɾAndroidΞϓϦΤϯδχΞ
Thank you!