Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Testing with microservices in merpay
Search
kazegusuri
April 14, 2018
Programming
10
10k
Testing with microservices in merpay
Go Conference 2018 Spring
kazegusuri
April 14, 2018
Tweet
Share
More Decks by kazegusuri
See All by kazegusuri
go-sqlite3を使ってCloud Spannerエミュレーターを作ってみた / Cloud Spanner emulator with go-sqlite3
kazegusuri
5
6.6k
handy-spanner GCPUG
kazegusuri
4
1.9k
Open SKT: メルペイ開発の裏側 / builderscon tokyo 2019 Open SKT
kazegusuri
22
27k
Keep watching and extending features of gRPC
kazegusuri
3
2.5k
Real World Mercari API Architecture
kazegusuri
1
6.2k
gRPC and REST with gRPC in practice
kazegusuri
19
7.9k
Fluentdで始めるPrometheus / Prometheus Tokyo Meetup #1
kazegusuri
1
1.9k
GRPCの実践と現状での利点欠点 / Go Conference 2016 Spring
kazegusuri
44
32k
OutputとBufferedOutputの間の何か
kazegusuri
2
3.3k
Other Decks in Programming
See All in Programming
構造化・自動化・ガードレール - Vibe Coding実践記 -
tonegawa07
0
150
코딩 에이전트 체크리스트: Claude Code ver.
nacyot
0
1k
MCPで実現できる、Webサービス利用体験について
syumai
7
2.1k
可変性を制する設計: 構造と振る舞いから考える概念モデリングとその実装
a_suenami
7
940
Streamlitで実現できるようになったこと、実現してくれたこと
ayumu_yamaguchi
2
220
Gemini CLI のはじめ方
ttnyt8701
1
100
効率的な開発手段として VRTを活用する
ishkawa
1
180
Hack Claude Code with Claude Code
choplin
8
2.8k
Strands Agents で実現する名刺解析アーキテクチャ
omiya0555
1
110
SQLアンチパターン第2版 データベースプログラミングで陥りがちな失敗とその対策 / Intro to SQL Antipatterns 2nd
twada
PRO
32
8.7k
それ CLI フレームワークがなくてもできるよ / Building CLI Tools Without Frameworks
orgachem
PRO
6
1.5k
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
0
140
Featured
See All Featured
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
How GitHub (no longer) Works
holman
314
140k
The Pragmatic Product Professional
lauravandoore
35
6.8k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Documentation Writing (for coders)
carmenintech
72
4.9k
RailsConf 2023
tenderlove
30
1.2k
We Have a Design System, Now What?
morganepeng
53
7.7k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Designing for Performance
lara
610
69k
How to Ace a Technical Interview
jacobian
278
23k
KATA
mclloyd
30
14k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
Transcript
Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices
in merpay
Agenda • ࣗݾհ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ
Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
ࣗݾհ • ࠤ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,
Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ͖ͳ͜ͱ • gRPC
@kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷརɾܽ
ϝϧΧϦʹ͓͚ΔgRPC • 2015ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜ΕܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ
None
None
ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes
(GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
ܾࡁͷٕज़తͳ͠͞ͱָ͠͞ • ܾࡁ͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
• ΤϯδχΞϦϯάͰղܾ
ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰτϥϯβΫγϣϯͰ݁
• Microservices ArchitectureͰඞવతʹτϥϯβΫγϣϯ • ݁Ռ߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛཧ • ঢ়ଶભҠϞσϧ
• ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ࠶։Մೳ
ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3
4 Init 1 2 3 4 Fin
Payment Point Sales Microservices Payment Provider 1 2 3 4
Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
ਓྨ͕ਖ਼࣮͘͢͠Δͷ͍͠
Ͳ͏ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞ • ֤ঢ়ଶຖʹ͋Δ͖ঢ়ଶ(σʔλͷଘࡏɺ)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •
͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
Failure Injection • ֎෦௨৴ʹରͯ͠ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏεͷHTTPϦΫΤετ • ෦αʔϏεͷgRPCϦΫΤετ
• ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷྃޙ •
Commitͷྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕ࠐΊͯศར!
Failure Injection (MySQL)
Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
net/http.RoundTripper ʹΑΔ࣮
Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
Client InterceptorʹΑΔ࣮ • ·ͩಋೖͰ͖͍ͯͳ͍
Random or Deterministic? • Ͳ͏͍͏݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬Λ͋͛Δͱޭ͠ͳ͘ͳΓςετ͕࣌ؒ͘ͳΔ •
Τϥʔͷൃੜܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұൃੜͨ͠ύεͰ2ͱൃੜͤ͞ͳ͍
αʔϏεͷϦΫΤετͷςετ • ςετͰ֎෦ɾ෦αʔϏεΛ͏ͷ͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock୯७ͳUnitςετʹྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ
• ༧ଌෆೳͳঢ়ଶͰͷςετ
Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ • σʔλϕʔεͳͲʹґଘͤͣʹߴʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δͷ͋Δ • Google
Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞͢Δ • αʔϏεɺpackage, interface
Google Cloud Pub/Sub ͷ Fake
Conformance Test for Fake • FakeͷৼΔ͍͕ຊདྷͷ࣮ͱဃ͍ͯ͠ͳ͍͔ • ظ͢ΔৼΔ͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
None
ςετͷछྨ • Unit Test / Integration Test • E2E Test
• Chaos Test • QA
Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ
• ґଘinterfaceશͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏεͷଓFakeΛར༻ • σʔλϕʔεͷڞ༗͕
σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕ • ͳΒDBΛςετຖʹཱͯΕྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞(instanceͰͳ͍) •
1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳΫΤϦΛ·Δ͝ͱ͛ΒΕΔ • ࡞ʹ͔͔Δ࣌ؒ1ඵ΄Ͳ
E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ͛ͯৼΔ͍Λςετ • ՄೳͳݶΓґଘαʔϏεىಈ͢Δ • docker-composeͰڥߏங • ςετؒͷಠཱੑςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
• ͜͜·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετڝ߹͢ΔΑ͏ͳϦΫΤετ
Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network
conditions • Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ͍Ͳ͜Ζ͕͍͠ • ࠓͷͱ͜ΖԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
QA (Quality Assurance) • PostmanΛͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • PostmanͷςετέʔεGitHubͰཧͯ͠CI •
QAͷਓ͕ࣗߦ͍ͬͯΔ!
·ͱΊ • ϚΠΫϩαʔϏε͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόάݟ͚ͭ͘͢ •
MockΑΓFake
We’re Hiring! https://www.merpay.com/jp/careers/index.html