Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Testing with microservices in merpay
Search
kazegusuri
April 14, 2018
Programming
10
10k
Testing with microservices in merpay
Go Conference 2018 Spring
kazegusuri
April 14, 2018
Tweet
Share
More Decks by kazegusuri
See All by kazegusuri
go-sqlite3を使ってCloud Spannerエミュレーターを作ってみた / Cloud Spanner emulator with go-sqlite3
kazegusuri
5
6.6k
handy-spanner GCPUG
kazegusuri
4
1.9k
Open SKT: メルペイ開発の裏側 / builderscon tokyo 2019 Open SKT
kazegusuri
22
27k
Keep watching and extending features of gRPC
kazegusuri
3
2.5k
Real World Mercari API Architecture
kazegusuri
1
6.2k
gRPC and REST with gRPC in practice
kazegusuri
19
7.9k
Fluentdで始めるPrometheus / Prometheus Tokyo Meetup #1
kazegusuri
1
1.9k
GRPCの実践と現状での利点欠点 / Go Conference 2016 Spring
kazegusuri
44
32k
OutputとBufferedOutputの間の何か
kazegusuri
2
3.3k
Other Decks in Programming
See All in Programming
複雑なドメインに挑む.pdf
yukisakai1225
5
1.2k
さようなら Date。 ようこそTemporal! 3年間先行利用して得られた知見の共有
8beeeaaat
3
1.5k
今だからこそ入門する Server-Sent Events (SSE)
nearme_tech
PRO
3
260
rage against annotate_predecessor
junk0612
0
170
MCPでVibe Working。そして、結局はContext Eng(略)/ Working with Vibe on MCP And Context Eng
rkaga
5
2.3k
概念モデル→論理モデルで気をつけていること
sunnyone
3
300
Amazon RDS 向けに提供されている MCP Server と仕組みを調べてみた/jawsug-okayama-2025-aurora-mcp
takahashiikki
1
120
Azure SRE Agentで運用は楽になるのか?
kkamegawa
0
2.6k
Cache Me If You Can
ryunen344
2
4k
CloudflareのChat Agent Starter Kitで簡単!AIチャットボット構築
syumai
2
520
私の後悔をAWS DMSで解決した話
hiramax
4
210
Kiroで始めるAI-DLC
kaonash
2
630
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
The Pragmatic Product Professional
lauravandoore
36
6.9k
Unsuck your backbone
ammeep
671
58k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.6k
Designing for humans not robots
tammielis
253
25k
The Straight Up "How To Draw Better" Workshop
denniskardys
236
140k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.2k
A designer walks into a library…
pauljervisheath
207
24k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Agile that works and the tools we love
rasmusluckow
330
21k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
930
Transcript
Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices
in merpay
Agenda • ࣗݾհ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ
Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
ࣗݾհ • ࠤ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,
Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ͖ͳ͜ͱ • gRPC
@kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷརɾܽ
ϝϧΧϦʹ͓͚ΔgRPC • 2015ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜ΕܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ
None
None
ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes
(GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
ܾࡁͷٕज़తͳ͠͞ͱָ͠͞ • ܾࡁ͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
• ΤϯδχΞϦϯάͰղܾ
ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰτϥϯβΫγϣϯͰ݁
• Microservices ArchitectureͰඞવతʹτϥϯβΫγϣϯ • ݁Ռ߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛཧ • ঢ়ଶભҠϞσϧ
• ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ࠶։Մೳ
ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3
4 Init 1 2 3 4 Fin
Payment Point Sales Microservices Payment Provider 1 2 3 4
Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
ਓྨ͕ਖ਼࣮͘͢͠Δͷ͍͠
Ͳ͏ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞ • ֤ঢ়ଶຖʹ͋Δ͖ঢ়ଶ(σʔλͷଘࡏɺ)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •
͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
Failure Injection • ֎෦௨৴ʹରͯ͠ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏεͷHTTPϦΫΤετ • ෦αʔϏεͷgRPCϦΫΤετ
• ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷྃޙ •
Commitͷྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕ࠐΊͯศར!
Failure Injection (MySQL)
Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
net/http.RoundTripper ʹΑΔ࣮
Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
Client InterceptorʹΑΔ࣮ • ·ͩಋೖͰ͖͍ͯͳ͍
Random or Deterministic? • Ͳ͏͍͏݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬Λ͋͛Δͱޭ͠ͳ͘ͳΓςετ͕࣌ؒ͘ͳΔ •
Τϥʔͷൃੜܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұൃੜͨ͠ύεͰ2ͱൃੜͤ͞ͳ͍
αʔϏεͷϦΫΤετͷςετ • ςετͰ֎෦ɾ෦αʔϏεΛ͏ͷ͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock୯७ͳUnitςετʹྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ
• ༧ଌෆೳͳঢ়ଶͰͷςετ
Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ • σʔλϕʔεͳͲʹґଘͤͣʹߴʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δͷ͋Δ • Google
Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞͢Δ • αʔϏεɺpackage, interface
Google Cloud Pub/Sub ͷ Fake
Conformance Test for Fake • FakeͷৼΔ͍͕ຊདྷͷ࣮ͱဃ͍ͯ͠ͳ͍͔ • ظ͢ΔৼΔ͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
None
ςετͷछྨ • Unit Test / Integration Test • E2E Test
• Chaos Test • QA
Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ
• ґଘinterfaceશͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏεͷଓFakeΛར༻ • σʔλϕʔεͷڞ༗͕
σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕ • ͳΒDBΛςετຖʹཱͯΕྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞(instanceͰͳ͍) •
1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳΫΤϦΛ·Δ͝ͱ͛ΒΕΔ • ࡞ʹ͔͔Δ࣌ؒ1ඵ΄Ͳ
E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ͛ͯৼΔ͍Λςετ • ՄೳͳݶΓґଘαʔϏεىಈ͢Δ • docker-composeͰڥߏங • ςετؒͷಠཱੑςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
• ͜͜·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετڝ߹͢ΔΑ͏ͳϦΫΤετ
Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network
conditions • Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ͍Ͳ͜Ζ͕͍͠ • ࠓͷͱ͜ΖԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
QA (Quality Assurance) • PostmanΛͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • PostmanͷςετέʔεGitHubͰཧͯ͠CI •
QAͷਓ͕ࣗߦ͍ͬͯΔ!
·ͱΊ • ϚΠΫϩαʔϏε͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόάݟ͚ͭ͘͢ •
MockΑΓFake
We’re Hiring! https://www.merpay.com/jp/careers/index.html