Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf...
Search
Ken Hamada
November 03, 2017
Technology
18
30k
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
サーバレスアーキテクチャによる時系列データベースの構築と監視
Ken Hamada
November 03, 2017
Tweet
Share
More Decks by Ken Hamada
See All by Ken Hamada
組織と技術の両輪で開発を加速させるkintoneチームの取り組み / JJUG CCC 2022 Fall Cybozu kintone
itchyny
1
2.5k
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
itchyny
0
5.7k
Other Decks in Technology
See All in Technology
ファッションコーディネートアプリ「WEAR」における、Vertex AI Vector Searchを利用したレコメンド機能の開発・運用で得られたノウハウの紹介
zozotech
PRO
0
300
[OCI Technical Deep Dive] OracleのAI戦略(2025年8月5日開催)
oracle4engineer
PRO
1
170
AI時代の大規模データ活用とセキュリティ戦略
ken5scal
0
110
Serverless Meetup #21
yoshidashingo
1
120
AIエージェントを現場で使う / 2025.08.07 著者陣に聞く!現場で活用するためのAIエージェント実践入門(Findyランチセッション)
smiyawaki0820
6
1k
Backlog AI アシスタントが切り開く未来
vvatanabe
1
130
AWS DDoS攻撃防御の最前線
ryutakondo
1
150
2025新卒研修・HTML/CSS #弁護士ドットコム
bengo4com
3
13k
Claude Codeから我々が学ぶべきこと
oikon48
10
2.8k
S3 Glacier のデータを Athena からクエリしようとしたらどうなるのか/try-to-query-s3-glacier-from-athena
emiki
0
220
Segment Anything Modelの最新動向:SAM2とその発展系
tenten0727
0
750
Oracle Exadata Database Service on Cloud@Customer X11M (ExaDB-C@C) サービス概要
oracle4engineer
PRO
2
6.3k
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
462
33k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Speed Design
sergeychernyshev
32
1.1k
Bash Introduction
62gerente
614
210k
Building Adaptive Systems
keathley
43
2.7k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
Designing for Performance
lara
610
69k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.4k
Music & Morning Musume
bryan
46
6.7k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Transcript
αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱࢹ גࣜձࣾͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo
ࣗݾհ גࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ͖: Rust, Haskell, Go, Scala,
Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
αʔόʔϨεͷίϯϙʔωϯτΛͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋
αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γࢹ͍ͯ͠Δਓ ✋
ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ࢹɾཧ͍ͯ͠Δ͔Λ͓͑͠·͢
࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerelʹ͍ͭͯ
Mackerel mackerel.io
Mackerel SaaSͷαʔόʔࢹɾཧπʔϧ ͯͳࣾͷࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛߘ͢Δpushܕ νϟοτπʔϧߏཧπʔϧͱΈ߹Θͤ Slack, HipChat, etc. Chef, Ansible,
etc.
Mackerelུ֓ਤ CPUɾMemory༻ OSใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ
Mackerelͷάϥϑ ϝτϦοΫͷσʔλ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετใϝλσʔλ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
࣌ܥྻσʔλϕʔε
࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹσʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •
ߴղ૾ɾظอଘɾେྔϝτϦοΫ 㱺 ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕͍ͬͯ·ͨ͠
GraphiteΛӡ༻͢Δ্Ͱͷ՝ ϋʔυΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ • ϝτϦοΫͷظอଘ (1m:25h 㱺
1m:400d+) σʔλϩετੑͷ͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ ΫϥυΛ׆༻ͯ͠ӡ༻ίετݮ͍ͨ͠ Application servers mackerel.io
৽࣌ܥྻDBͰୡ͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυΣΞௐୡίετ͑Δ σʔλϩετੑ
৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετݮ • ϚωʔδυɾαʔϏεΛ͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ͍͑ͨ
• RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 Ҡಈ σʔλϩετੑ • Amazon Kinesis Streams
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛΈ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
Aug. 2017 ৽࣌ܥྻDBʹҠߦ
diamondߏਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯߗ͍) DynamoDB Kinesis Steams S3 Lambda
Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ
diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ͍ॻ͖ࠐΈίετΛ࠷దԽ •
ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠σʔλKinesis StreamsʹΔ • োൃੜ͔࣌Β࠶ॲཧͰ͖Δ
• backup༻ʹผͷLambdaΛ͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো࣌ͷόοΫΞοϓܥ
Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛΜͰ͍Δ • ॳElastiCacheΛఆ͍ͯͨ͠ •
εέʔϧ {Ξοϓ,Ξτ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕ҰఆݸҎ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cronεέʔϧͤ͞ʹ͍͘ͷͰجຊతʹΘͳ͍ Lambda Redis cluster
Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫMapʹอଘ •
BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacityجຊతʹҰఆ • Read capacityΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
Lambda ίʔυߏɾdeploy ϩδοΫlibσΟϨΫτϦʹΓग़͢ • function handlerΛ࠷খʹอͭ (clientೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy
• build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlinkḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά
τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓ஈ͕ߴ͍
ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ topnetstattcpdumpଧͯͳ͍ʂ
͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏
CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch
sizeΛ૿ͨ͠ͱ͖ͷdurationͷԠ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷͣ • ઢܗԠͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧత (͔) • Ұൠʹ…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ • I/O͕ࢧతͳΒCPU্͕͕ͬͯͦ͜·ͰԼ͕Βͳ͍ͣ • యܕతͳϦΫΤετͷlatencyduration͔ΒײతʹΘ͔Γͦ͏
ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ) • खݩͰܭଌ͢Δ • sjsp
(Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿͢ (CPUεϖοΫ্) • ΞϧΰϦζϜɾॲཧ༰ͷվળ
sjsp ࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥβʹґଘ͠ͳ͍)
• Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ͑Δ CPU boundͳॏ͍ॲཧ͕͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐΔ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ •
ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿͢
AWS X-Ray ϚωʔδυαʔϏεͷϦΫΤετͷใΛऩूɾੳ AWSͷSDKʹΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ߹ʹΛൃݟͰ͖Δ const
AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));
֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerel plugins ͯͳʹ͓͚ΔAWSαʔϏεͷࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch
Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫMackerelʹू ϗετཧɾΞϥʔτɾ௨ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ ͨ͘͞Μ͋Δ
diamondߏਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL
expired Application servers diamond-reader mackerel.io diamond-writer
Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key
γϟʔυΛݻఆ͍ͨ࣌͠ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
Kinesis Streamsͷࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ
• ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞτ • γϟʔυ͕Γͳ͍ • γϟʔυΛ૿ͯ͠ throughputΛ্͛Δ Read delay I/O bytes
Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφجຊతʹ࠶ར༻͞ΕΔ • deployઃఆมߋͰ࡞Γ͞ΕΔ •
clientglobalมͰӬଓԽ ͦΕͧΕͷίϯςφಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤containerhandlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ ྉۚମܥ
Lambdaͷࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճɾΤϥʔճ Τϥʔճ্͕͕͍ͬͯΔ • CloudWatch
LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿ͦ͏ • CPUεϖοΫྉ͕ۚ͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughputΓͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shardΛ૿͢
Lambdaͷࢹ ฏۉಉ࣌ॲཧ = 1ؒͷ࣮ߦճ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)
≤ Kinesis shard time 1 minute container processing idle Mackerel advanced graph Shardͷ7ׂҎԼ(҆)ʹ͑Δ ShardΛ૿͢ͷ͕͔͔࣌ؒΔ ੵ͕՝ۚର ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥυ൛ͷࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ
{Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ
DynamoDBͷࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •
LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
Redisͷࢹ CPU % Redis used memory Redis processed commands CPU
%্͕͕͍ͬͯΔ • Redis1 core͔͑͠ͳ͍ • readΫΤϦΛslaveʹ͚Δ • nodeΛՃͯ͠resharding memory༻ྔ • ϋʔυΣΞͷݶք͕͋Δ • key͕ফ͑ͯOSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers
diamond-reader
diamond-readerͷࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO
࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
ॲཧ ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ / 1m • RedisɾDynamoDBʹՃͨ͠ϝτϦοΫ • CloudWatchʹJSONͰใΛग़͢ •
ผͷLambda functionͰҰؒͷϩάΛूܭ (cron trigger) 㱺 Mackerelʹߘ • ॲཧ͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ༻ͷΩʔΛҰ͝ͱʹ࡞ͬͯincrͯ͠Α͍
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host
End-to-end ࢹ ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹߘɾࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2Eॏཁʂ
ࢹΛ࡞Δ ίϯϙʔωϯτΛΈ߹Θͤͨ࣌ શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱͲ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷతͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾࢹ͠·͠ΐ͏
ʮࢹΛ͢Δʯͱ ϝτϦοΫʹᮢΛઃఆ͢Δ͜ͱ ͚ͩͰͳ͍
ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ Lambdaॲཧ࣌ؒ ߹ܭॲཧ DynamoDB Write E2E
MackerelͷάϥϑϘʔυ
ࢹ͢Δͱ͍͏͜ͱ ฏৗঢ়ଶΛΓ ఆٛ͢Δͱ͍͏͜ͱ
·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ࢹͷجૅฏৗঢ়ଶΛΔ͜ͱ ܥશମͷՄ༻ੑΛࢹ͠Α͏ ʮࢹΛ࡞Δʯ
We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ
iPhoneɾAndroidΞϓϦΤϯδχΞ
Thank you!