Go Conference 2018 Spring
Go Conference 2018 SpringMasahiro Sano (@kazegusuri)Testing with microservices in merpay
View Slide
Agenda• ࣗݾհ• ϝϧϖΠͷܾࡁγεςϜ• ςετϑϨʔϜϫʔΫ• ςετͷछྨεϥΠυ Speaker Deck ʹ͋Γ·͢ʂhttps://speakerdeck.com/kazegusuri
ࣗݾհ• ࠤ ਖ਼ߒ (@kazegusuri)• Mercari, Inc → Souzoh, Inc → merpay, Inc• Principal Software Engineer• Backend Engineer• GoͰܾࡁγεςϜશൠΛ։ൃத• ͖ͳ͜ͱ• gRPC
@kazegusuriGo Conference 2016 SpringGRPCͷ࣮ફͱݱঢ়Ͱͷརɾܽ
ϝϧΧϦʹ͓͚ΔgRPC• 2015ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ• ͜ΕܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ• ͦͯ͠ϝϧϖΠ
ϝϧϖΠͷٕज़ελοΫ• Go + gRPC• Microservices Architecture• Kubernetes (GKE) + (EKS?)• ͦͷଞ• CircleCI, Spinnaker• Stackdriver Logging, Profiler• Datadog (Monitoring, Tracing)• Pagerduty, Sentry
ܾࡁͷٕज़తͳ͠͞ͱָ͠͞• ܾࡁ͍͠ (ͱݴΘΕΔ)• ܾࡁϓϩόΠμͱͷҰ؏ੑ• ΫϦςΟΧϧ• 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ• ΤϯδχΞϦϯάͰղܾ
ϚΠΫϩαʔϏεͱҰ؏ੑ• Microservices Architecture is super difficult!!• Monolithic ArchitectureͰτϥϯβΫγϣϯͰ݁• Microservices ArchitectureͰඞવతʹτϥϯβΫγϣϯ• ݁Ռ߹ੑ• Idempotency• Retry, retry, retry• Async, Message Queue
ϝϧϖΠͷܾࡁγεςϜ• Microservices Architecture• ػೳຖʹϚΠΫϩαʔϏεΛߏங• த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛཧ• ঢ়ଶભҠϞσϧ• ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ• ͲΜͳλΠϛϯάͰམͪͯ࠶։Մೳ
ϝϧϖΠͷܾࡁγεςϜPaymentPoint Sales MicroservicesPaymentProvider1234Init 1 2 3 4 Fin
PaymentPoint Sales MicroservicesPaymentProvider1234Init 1 2 3 4 FinKeep retrying with delayϝϧϖΠͷܾࡁγεςϜ
ਓྨ͕ਖ਼࣮͘͢͠Δͷ͍͠
Ͳ͏ͬͯςετ͢Δ͔• ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞• ֤ঢ়ଶຖʹ͋Δ͖ঢ়ଶ(σʔλͷଘࡏɺ)Λఆٛ• ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ• ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ• ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢Δ͔֬ೝ͢Δ
ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
Failure Injection• ֎෦௨৴ʹରͯ͠ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ• MySQL• ֎෦αʔϏεͷHTTPϦΫΤετ• ෦αʔϏεͷgRPCϦΫΤετ• ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
Failure Injection (MySQL)• ΤϥʔλΠϛϯά• Query• Execͷྃޙ• Commitͷྃޙ• શͯͷMySQL queryʹରͯ͠hook͢Δ• github.com/shogo82148/go-sql-proxy• ༷ʑͳhook͕ࠐΊͯศར!
Failure Injection (MySQL)
Failure Injection (HTTP)• ΤϥʔλΠϛϯά• HTTPϦΫΤετͷྃޙ• ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ• net/http.RoundTripper ʹΑΔ࣮
Failure Injection (gRPC)• ΤϥʔλΠϛϯά• gRPCϦΫΤετͷྃޙ• ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ• Client InterceptorʹΑΔ࣮• ·ͩಋೖͰ͖͍ͯͳ͍
Random or Deterministic?• Ͳ͏͍͏݅ͰΤϥʔΛൃੜͤ͞Δ͔• ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ• Τϥʔ֬Λ͋͛Δͱޭ͠ͳ͘ͳΓςετ͕࣌ؒ͘ͳΔ• Τϥʔͷൃੜܾఆతʹߦ͍͍ͨ• stacktrace(runtime.Callers) Ͱܾఆ• Ұൃੜͨ͠ύεͰ2ͱൃੜͤ͞ͳ͍
αʔϏεͷϦΫΤετͷςετ• ςετͰ֎෦ɾ෦αʔϏεΛ͏ͷ͍͠• ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ• Mock୯७ͳUnitςετʹྑ͍• ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯςετ• ༧ଌෆೳͳঢ়ଶͰͷςετ
Fake Implementation• αʔϏεͷ୯७Խ͞Ε࣮ͨ• σʔλϕʔεͳͲʹґଘͤͣʹߴʹಈ࡞• ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δͷ͋Δ• Google Cloud Pub/Sub• ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞͢Δ• αʔϏεɺpackage, interface
Google Cloud Pub/Sub ͷ Fake
Conformance Test for Fake• FakeͷৼΔ͍͕ຊདྷͷ࣮ͱဃ͍ͯ͠ͳ͍͔• ظ͢ΔৼΔ͍Λςετέʔεʹ͢Δ• ຊདྷͷ࣮ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
ςετͷछྨ• Unit Test / Integration Test• E2E Test• Chaos Test• QA
Unit and Integration Test• packageͰߦ͏ςετ• ݫີͳ୯ମςετʹ͍ͯ͠ͳ͍• ԼҐϨΠϠ·ͰؚΊͨςετ• ґଘinterfaceશͯΠϯδΣΫγϣϯ͢Δ• ґଘαʔϏεͷଓFakeΛར༻• σʔλϕʔεͷڞ༗͕
σʔλϕʔεͷςετؒͷಠཱੑ• ֤ςετͰDB·Ͱଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ• DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕• ͳΒDBΛςετຖʹཱͯΕྑ͍ʂ• TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞(instanceͰͳ͍)• 1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ• multiStatements=trueʹ͢ΔͱExecͰෳΫΤϦΛ·Δ͝ͱ͛ΒΕΔ• ࡞ʹ͔͔Δ࣌ؒ1ඵ΄Ͳ
E2E Test• ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ͛ͯৼΔ͍Λςετ• ՄೳͳݶΓґଘαʔϏεىಈ͢Δ• docker-composeͰڥߏங• ςετؒͷಠཱੑςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ• ͜͜·ͩਏ͍ͱ͜Ζ͕ଟ͍• E2EςετͰෳࡶͳςετΛ࣮ࢪ• ಉ࣌ϦΫΤετڝ߹͢ΔΑ͏ͳϦΫΤετ
Chaos Test (Trying)• github.com/Shopify/toxiproxy• framework for simulating network conditions• Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ• gRPCͩͱ͍Ͳ͜Ζ͕͍͠• ࠓͷͱ͜ΖԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ༧ଌʹར༻
QA (Quality Assurance)• PostmanΛͬͨAPIςετ• JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻• PostmanͷςετέʔεGitHubͰཧͯ͠CI• QAͷਓ͕ࣗߦ͍ͬͯΔ!
·ͱΊ• ϚΠΫϩαʔϏε͍͠• ςετϑϨʔϜϫʔΫͰςετ߲ͷύλʔϯԽ• Failure Injection TestͰΤοδέʔεͷόάݟ͚ͭ͘͢• MockΑΓFake
We’re Hiring!https://www.merpay.com/jp/careers/index.html