Testing with microservices in merpay

Transcript

1. 1.

   Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices

   in merpay
2. 2.

   Agenda • ࣗݾ঺հ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ͸

   Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
3. 3.

   ࣗݾ঺հ • ࠤ໺ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,

   Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ޷͖ͳ͜ͱ • gRPC
4. 4.

   @kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷར఺ɾܽ఺

5. 5.

   ϝϧΧϦʹ͓͚ΔgRPC • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ΁

6. 6.

   None
7. 7.

   None
8. 8.

   ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes

   (GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
9. 9.

   ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞ • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ

   • ΤϯδχΞϦϯάͰղܾ
10. 10.

    ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁

    • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭ • ݁Ռ੔߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
11. 11.

    ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ • ঢ়ଶભҠϞσϧ

    • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ
12. 12.

    ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3

    4 Init 1 2 3 4 Fin
13. 13.

    Payment Point Sales Microservices Payment Provider 1 2 3 4

    Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
14. 14.

    ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠

15. 15.

    Ͳ͏΍ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒ • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •

    ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
16. 16.

    ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ

17. 17.

    Failure Injection • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏε΁ͷHTTPϦΫΤετ • ಺෦αʔϏε΁ͷgRPCϦΫΤετ

    • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
18. 18.

    Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷ׬ྃޙ •

    Commitͷ׬ྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕࢓ࠐΊͯศར!
19. 19.

    Failure Injection (MySQL)

20. 20.

    Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ •

    net/http.RoundTripper ʹΑΔ࣮૷
21. 21.

    Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ •

    Client InterceptorʹΑΔ࣮૷ • ·ͩಋೖͰ͖͍ͯͳ͍
22. 22.

    Random or Deterministic? • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ •

    Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍
23. 23.

    αʔϏε΁ͷϦΫΤετͷςετ • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock͸୯७ͳUnitςετʹ͸ྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ

    • ༧ଌෆೳͳঢ়ଶͰͷςετ
24. 24.

    Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷ • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ • Google

    Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ • αʔϏεɺpackage, interface
25. 25.

    Google Cloud Pub/Sub ͷ Fake

26. 26.

    Conformance Test for Fake • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔ • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ

27. 27.

    None
28. 28.

    ςετͷछྨ • Unit Test / Integration Test • E2E Test

    • Chaos Test • QA
29. 29.

    Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ

    • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻ • σʔλϕʔεͷڞ༗͕໰୊
30. 30.

    σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊ • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍) •

    1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ
31. 31.

    E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ • docker-composeͰ؀ڥߏங • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ

    • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ
32. 32.

    Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network

    conditions • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠ • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
33. 33.

    QA (Quality Assurance) • PostmanΛ࢖ͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI •

    QAͷਓࣗ਎͕ߦ͍ͬͯΔ!
34. 34.

    ·ͱΊ • ϚΠΫϩαʔϏε͸೉͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢ •

    MockΑΓ΋Fake
35. 35.

    We’re Hiring! https://www.merpay.com/jp/careers/index.html