Testing with microservices in merpay

Go Conference 2018 Spring

1. Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices in merpay
2. Agenda • ࣗݾ঺հ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ͸ Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
3. ࣗݾ঺հ • ࠤ໺ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh, Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ޷͖ͳ͜ͱ • gRPC
4. @kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷར఺ɾܽ఺
5. ϝϧΧϦʹ͓͚ΔgRPC • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ΁
8. ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes (GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
9. ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞ • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ • ΤϯδχΞϦϯάͰղܾ
10. ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁ • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭ • ݁Ռ੔߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
11. ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ • ঢ়ଶભҠϞσϧ • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ
12. ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3 4 Init 1 2 3 4 Fin
13. Payment Point Sales Microservices Payment Provider 1 2 3 4 Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
14. ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠
15. Ͳ͏΍ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒ • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ • ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
16. ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
17. Failure Injection • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏε΁ͷHTTPϦΫΤετ • ಺෦αʔϏε΁ͷgRPCϦΫΤετ • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
18. Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷ׬ྃޙ • Commitͷ׬ྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕࢓ࠐΊͯศར!
19. Failure Injection (MySQL)
20. Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ • net/http.RoundTripper ʹΑΔ࣮૷
21. Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ • Client InterceptorʹΑΔ࣮૷ • ·ͩಋೖͰ͖͍ͯͳ͍
22. Random or Deterministic? • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ • Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍
23. αʔϏε΁ͷϦΫΤετͷςετ • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock͸୯७ͳUnitςετʹ͸ྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ • ༧ଌෆೳͳঢ়ଶͰͷςετ
24. Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷ • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ • Google Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ • αʔϏεɺpackage, interface
25. Google Cloud Pub/Sub ͷ Fake
26. Conformance Test for Fake • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔ • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
28. ςετͷछྨ • Unit Test / Integration Test • E2E Test • Chaos Test • QA
29. Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻ • σʔλϕʔεͷڞ༗͕໰୊
30. σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊ • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍) • 1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ
31. E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ • docker-composeͰ؀ڥߏங • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ
32. Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network conditions • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠ • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
33. QA (Quality Assurance) • PostmanΛ࢖ͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI • QAͷਓࣗ਎͕ߦ͍ͬͯΔ!
34. ·ͱΊ • ϚΠΫϩαʔϏε͸೉͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢ • MockΑΓ΋Fake
35. We’re Hiring! https://www.merpay.com/jp/careers/index.html