Testing with microservices in merpay

Go Conference 2018 Spring

Transcript

1. Go Conference 2018 Spring
   Masahiro Sano (@kazegusuri)
   Testing with microservices in merpay
   

   View Slide

2. Agenda
   • ࣗݾ঺հ
   • ϝϧϖΠͷܾࡁγεςϜ
   • ςετϑϨʔϜϫʔΫ
   • ςετͷछྨ
   εϥΠυ͸ Speaker Deck ʹ͋Γ·͢ʂ
   https://speakerdeck.com/kazegusuri
   

   View Slide

3. ࣗݾ঺հ
   • ࠤ໺ ਖ਼ߒ (@kazegusuri)
   • Mercari, Inc → Souzoh, Inc → merpay, Inc
   • Principal Software Engineer
   • Backend Engineer
   • GoͰܾࡁγεςϜશൠΛ։ൃத
   • ޷͖ͳ͜ͱ
   • gRPC
   

   View Slide

4. @kazegusuri
   Go Conference 2016 Spring
   GRPCͷ࣮ફͱ
   ݱঢ়Ͱͷར఺ɾܽ఺
   

   View Slide

5. ϝϧΧϦʹ͓͚ΔgRPC
   • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ
   • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ
   • ͦͯ͠ϝϧϖΠ΁
   

   View Slide

6. View Slide

7. View Slide

8. ϝϧϖΠͷٕज़ελοΫ
   • Go + gRPC
   • Microservices Architecture
   • Kubernetes (GKE) + (EKS?)
   • ͦͷଞ
   • CircleCI, Spinnaker
   • Stackdriver Logging, Profiler
   • Datadog (Monitoring, Tracing)
   • Pagerduty, Sentry
   

   View Slide

9. ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞
   • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ)
   • ܾࡁϓϩόΠμͱͷҰ؏ੑ
   • ΫϦςΟΧϧ
   • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
   • ΤϯδχΞϦϯάͰղܾ
   

   View Slide

10. ϚΠΫϩαʔϏεͱҰ؏ੑ
    • Microservices Architecture is super difficult!!
    • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁
    • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭
    • ݁Ռ੔߹ੑ
    • Idempotency
    • Retry, retry, retry
    • Async, Message Queue
    

    View Slide

11. ϝϧϖΠͷܾࡁγεςϜ
    • Microservices Architecture
    • ػೳຖʹϚΠΫϩαʔϏεΛߏங
    • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ
    • ঢ়ଶભҠϞσϧ
    • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ
    • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ
    

    View Slide

12. ϝϧϖΠͷܾࡁγεςϜ
    Payment
    Point Sales Microservices
    Payment
    Provider
    1
    2
    3
    4
    Init 1 2 3 4 Fin
    

    View Slide

13. Payment
    Point Sales Microservices
    Payment
    Provider
    1
    2
    3
    4
    Init 1 2 3 4 Fin
    Keep retrying with delay
    ϝϧϖΠͷܾࡁγεςϜ
    

    View Slide

14. ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠
    

    View Slide

15. Ͳ͏΍ͬͯςετ͢Δ͔
    • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒
    • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ
    • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ
    • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ
    • ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢
    Δ͔֬ೝ͢Δ
    

    View Slide

16. ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
    

    View Slide

17. Failure Injection
    • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ
    • MySQL
    • ֎෦αʔϏε΁ͷHTTPϦΫΤετ
    • ಺෦αʔϏε΁ͷgRPCϦΫΤετ
    • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
    

    View Slide

18. Failure Injection (MySQL)
    • ΤϥʔλΠϛϯά
    • Query
    • Execͷ׬ྃޙ
    • Commitͷ׬ྃޙ
    • શͯͷMySQL queryʹରͯ͠hook͢Δ
    • github.com/shogo82148/go-sql-proxy
    • ༷ʑͳhook͕࢓ࠐΊͯศར!
    

    View Slide

19. Failure Injection (MySQL)
    

    View Slide

20. Failure Injection (HTTP)
    • ΤϥʔλΠϛϯά
    • HTTPϦΫΤετͷ׬ྃޙ
    • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ
    • net/http.RoundTripper ʹΑΔ࣮૷
    

    View Slide

21. Failure Injection (gRPC)
    • ΤϥʔλΠϛϯά
    • gRPCϦΫΤετͷ׬ྃޙ
    • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ
    • Client InterceptorʹΑΔ࣮૷
    • ·ͩಋೖͰ͖͍ͯͳ͍
    

    View Slide

22. Random or Deterministic?
    • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔
    • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ
    • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ
    • Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ
    • stacktrace(runtime.Callers) Ͱܾఆ
    • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍
    

    View Slide

23. αʔϏε΁ͷϦΫΤετͷςετ
    • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠
    • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ
    • Mock͸୯७ͳUnitςετʹ͸ྑ͍
    • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ
    ςετ
    • ༧ଌෆೳͳঢ়ଶͰͷςετ
    

    View Slide

24. Fake Implementation
    • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷
    • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞
    • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ
    • Google Cloud Pub/Sub
    • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ
    • αʔϏεɺpackage, interface
    

    View Slide

25. Google Cloud Pub/Sub ͷ Fake
    

    View Slide

26. Conformance Test for Fake
    • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔
    • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ
    • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
    

    View Slide

27. View Slide

28. ςετͷछྨ
    • Unit Test / Integration Test
    • E2E Test
    • Chaos Test
    • QA
    

    View Slide

29. Unit and Integration Test
    • packageͰߦ͏ςετ
    • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍
    • ԼҐϨΠϠ·ͰؚΊͨςετ
    • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ
    • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻
    • σʔλϕʔεͷڞ༗͕໰୊
    

    View Slide

30. σʔλϕʔεͷςετؒͷಠཱੑ
    • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ
    • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊
    • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ
    • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍)
    • 1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ
    • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ
    • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ
    

    View Slide

31. E2E Test
    • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ
    • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ
    • docker-composeͰ؀ڥߏங
    • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
    • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍
    • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ
    • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ
    

    View Slide

32. Chaos Test (Trying)
    • github.com/Shopify/toxiproxy
    • framework for simulating network conditions
    • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ
    • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠
    • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ
    ༧ଌʹར༻
    

    View Slide

33. QA (Quality Assurance)
    • PostmanΛ࢖ͬͨAPIςετ
    • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻
    • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI
    • QAͷਓࣗ਎͕ߦ͍ͬͯΔ!
    

    View Slide

34. ·ͱΊ
    • ϚΠΫϩαʔϏε͸೉͍͠
    • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ
    • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢
    • MockΑΓ΋Fake
    

    View Slide

35. We’re Hiring!
    https://www.merpay.com/jp/careers/index.html
    

    View Slide