Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Testing with microservices in merpay
Search
kazegusuri
April 14, 2018
Programming
10
10k
Testing with microservices in merpay
Go Conference 2018 Spring
kazegusuri
April 14, 2018
Tweet
Share
More Decks by kazegusuri
See All by kazegusuri
go-sqlite3を使ってCloud Spannerエミュレーターを作ってみた / Cloud Spanner emulator with go-sqlite3
kazegusuri
5
6.5k
handy-spanner GCPUG
kazegusuri
4
1.8k
Open SKT: メルペイ開発の裏側 / builderscon tokyo 2019 Open SKT
kazegusuri
22
27k
Keep watching and extending features of gRPC
kazegusuri
3
2.5k
Real World Mercari API Architecture
kazegusuri
1
6.1k
gRPC and REST with gRPC in practice
kazegusuri
19
7.8k
Fluentdで始めるPrometheus / Prometheus Tokyo Meetup #1
kazegusuri
1
1.8k
GRPCの実践と現状での利点欠点 / Go Conference 2016 Spring
kazegusuri
44
32k
OutputとBufferedOutputの間の何か
kazegusuri
2
3.2k
Other Decks in Programming
See All in Programming
SwiftUI API Design Lessons
niw
1
320
プロダクト横断分析に役立つ、事前集計しないサマリーテーブル設計
hanon52_
2
500
監視 やばい
syossan27
11
10k
SwiftDataのカスタムデータストアを試してみた
1mash0
0
130
Unlock the Potential of Swift Code Generation
rockname
0
280
インプロセスQAにおいて大事にしていること / In-process QA Meetup
medley
0
100
ASP.NETアプリケーションのモダナイゼーションについて
tomokusaba
0
200
DevOpsDaysTokyo2025社内副業で他部門へ_越境_して見えた価値再定義最大1か月のリードタイムを10分に短縮したDevOps実践.pdf
susumutomita
0
110
RubyKaigi Dev Meeting 2025
tenderlove
1
830
カオスに立ち向かう小規模チームの装備の選択〜フルスタックTSという装備の強み _ 弱み〜/Choosing equipment for a small team facing chaos ~ Strengths and weaknesses of full-stack TS~
bitkey
1
120
Glue Studio Visual ETL実装時にハマったこと
kokitamura
0
100
Memory API : Patterns, Performance et Cas d'Utilisation
josepaumard
1
150
Featured
See All Featured
Six Lessons from altMBA
skipperchong
28
3.7k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
104
19k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.3k
How STYLIGHT went responsive
nonsquared
100
5.5k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.4k
How GitHub (no longer) Works
holman
314
140k
Practical Orchestrator
shlominoach
187
11k
How to Think Like a Performance Engineer
csswizardry
23
1.5k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Embracing the Ebb and Flow
colly
85
4.7k
VelocityConf: Rendering Performance Case Studies
addyosmani
329
24k
Transcript
Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices
in merpay
Agenda • ࣗݾհ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ
Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
ࣗݾհ • ࠤ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,
Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ͖ͳ͜ͱ • gRPC
@kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷརɾܽ
ϝϧΧϦʹ͓͚ΔgRPC • 2015ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜ΕܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ
None
None
ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes
(GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
ܾࡁͷٕज़తͳ͠͞ͱָ͠͞ • ܾࡁ͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
• ΤϯδχΞϦϯάͰղܾ
ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰτϥϯβΫγϣϯͰ݁
• Microservices ArchitectureͰඞવతʹτϥϯβΫγϣϯ • ݁Ռ߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛཧ • ঢ়ଶભҠϞσϧ
• ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ࠶։Մೳ
ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3
4 Init 1 2 3 4 Fin
Payment Point Sales Microservices Payment Provider 1 2 3 4
Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
ਓྨ͕ਖ਼࣮͘͢͠Δͷ͍͠
Ͳ͏ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞ • ֤ঢ়ଶຖʹ͋Δ͖ঢ়ଶ(σʔλͷଘࡏɺ)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •
͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
Failure Injection • ֎෦௨৴ʹରͯ͠ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏεͷHTTPϦΫΤετ • ෦αʔϏεͷgRPCϦΫΤετ
• ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷྃޙ •
Commitͷྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕ࠐΊͯศར!
Failure Injection (MySQL)
Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
net/http.RoundTripper ʹΑΔ࣮
Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
Client InterceptorʹΑΔ࣮ • ·ͩಋೖͰ͖͍ͯͳ͍
Random or Deterministic? • Ͳ͏͍͏݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬Λ͋͛Δͱޭ͠ͳ͘ͳΓςετ͕࣌ؒ͘ͳΔ •
Τϥʔͷൃੜܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұൃੜͨ͠ύεͰ2ͱൃੜͤ͞ͳ͍
αʔϏεͷϦΫΤετͷςετ • ςετͰ֎෦ɾ෦αʔϏεΛ͏ͷ͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock୯७ͳUnitςετʹྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ
• ༧ଌෆೳͳঢ়ଶͰͷςετ
Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ • σʔλϕʔεͳͲʹґଘͤͣʹߴʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δͷ͋Δ • Google
Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞͢Δ • αʔϏεɺpackage, interface
Google Cloud Pub/Sub ͷ Fake
Conformance Test for Fake • FakeͷৼΔ͍͕ຊདྷͷ࣮ͱဃ͍ͯ͠ͳ͍͔ • ظ͢ΔৼΔ͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
None
ςετͷछྨ • Unit Test / Integration Test • E2E Test
• Chaos Test • QA
Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ
• ґଘinterfaceશͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏεͷଓFakeΛར༻ • σʔλϕʔεͷڞ༗͕
σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕ • ͳΒDBΛςετຖʹཱͯΕྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞(instanceͰͳ͍) •
1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳΫΤϦΛ·Δ͝ͱ͛ΒΕΔ • ࡞ʹ͔͔Δ࣌ؒ1ඵ΄Ͳ
E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ͛ͯৼΔ͍Λςετ • ՄೳͳݶΓґଘαʔϏεىಈ͢Δ • docker-composeͰڥߏங • ςετؒͷಠཱੑςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
• ͜͜·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετڝ߹͢ΔΑ͏ͳϦΫΤετ
Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network
conditions • Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ͍Ͳ͜Ζ͕͍͠ • ࠓͷͱ͜ΖԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
QA (Quality Assurance) • PostmanΛͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • PostmanͷςετέʔεGitHubͰཧͯ͠CI •
QAͷਓ͕ࣗߦ͍ͬͯΔ!
·ͱΊ • ϚΠΫϩαʔϏε͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόάݟ͚ͭ͘͢ •
MockΑΓFake
We’re Hiring! https://www.merpay.com/jp/careers/index.html