Testing with microservices in merpay

Transcript

1. Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices

   in merpay

2. Agenda • ࣗݾ঺հ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ͸

   Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri

3. ࣗݾ঺հ • ࠤ໺ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,

   Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ޷͖ͳ͜ͱ • gRPC

4. @kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷར఺ɾܽ఺

5. ϝϧΧϦʹ͓͚ΔgRPC • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ΁

6. None
7. None

8. ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes

   (GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry

9. ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞ • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ

   • ΤϯδχΞϦϯάͰղܾ

10. ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁

    • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭ • ݁Ռ੔߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue

11. ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ • ঢ়ଶભҠϞσϧ

    • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ

12. ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3

    4 Init 1 2 3 4 Fin

13. Payment Point Sales Microservices Payment Provider 1 2 3 4

    Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ

14. ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠

15. Ͳ͏΍ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒ • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •

    ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ

16. ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ

17. Failure Injection • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏε΁ͷHTTPϦΫΤετ • ಺෦αʔϏε΁ͷgRPCϦΫΤετ

    • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ

18. Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷ׬ྃޙ •

    Commitͷ׬ྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕࢓ࠐΊͯศར!

19. Failure Injection (MySQL)

20. Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ •

    net/http.RoundTripper ʹΑΔ࣮૷

21. Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ •

    Client InterceptorʹΑΔ࣮૷ • ·ͩಋೖͰ͖͍ͯͳ͍

22. Random or Deterministic? • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ •

    Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍

23. αʔϏε΁ͷϦΫΤετͷςετ • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock͸୯७ͳUnitςετʹ͸ྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ

    • ༧ଌෆೳͳঢ়ଶͰͷςετ

24. Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷ • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ • Google

    Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ • αʔϏεɺpackage, interface

25. Google Cloud Pub/Sub ͷ Fake

26. Conformance Test for Fake • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔ • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ

27. None

28. ςετͷछྨ • Unit Test / Integration Test • E2E Test

    • Chaos Test • QA

29. Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ

    • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻ • σʔλϕʔεͷڞ༗͕໰୊

30. σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊ • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍) •

    1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ

31. E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ • docker-composeͰ؀ڥߏங • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ

    • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ

32. Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network

    conditions • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠ • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻

33. QA (Quality Assurance) • PostmanΛ࢖ͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI •

    QAͷਓࣗ਎͕ߦ͍ͬͯΔ!

34. ·ͱΊ • ϚΠΫϩαʔϏε͸೉͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢ •

    MockΑΓ΋Fake

35. We’re Hiring! https://www.merpay.com/jp/careers/index.html