Testing with microservices in merpay

Testing with microservices in merpay

Go Conference 2018 Spring

59c0cc69a8ad4ca8d26d752b3b795b55?s=128

kazegusuri

April 14, 2018
Tweet

Transcript

  1. Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices

    in merpay
  2. Agenda • ࣗݾ঺հ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ͸

    Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
  3. ࣗݾ঺հ • ࠤ໺ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,

    Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ޷͖ͳ͜ͱ • gRPC
  4. @kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷར఺ɾܽ఺

  5. ϝϧΧϦʹ͓͚ΔgRPC • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ΁

  6. None
  7. None
  8. ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes

    (GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
  9. ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞ • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ

    • ΤϯδχΞϦϯάͰղܾ
  10. ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁

    • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭ • ݁Ռ੔߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
  11. ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ • ঢ়ଶભҠϞσϧ

    • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ
  12. ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3

    4 Init 1 2 3 4 Fin
  13. Payment Point Sales Microservices Payment Provider 1 2 3 4

    Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
  14. ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠

  15. Ͳ͏΍ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒ • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •

    ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
  16. ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ

  17. Failure Injection • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏε΁ͷHTTPϦΫΤετ • ಺෦αʔϏε΁ͷgRPCϦΫΤετ

    • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
  18. Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷ׬ྃޙ •

    Commitͷ׬ྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕࢓ࠐΊͯศར!
  19. Failure Injection (MySQL)

  20. Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ •

    net/http.RoundTripper ʹΑΔ࣮૷
  21. Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ •

    Client InterceptorʹΑΔ࣮૷ • ·ͩಋೖͰ͖͍ͯͳ͍
  22. Random or Deterministic? • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ •

    Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍
  23. αʔϏε΁ͷϦΫΤετͷςετ • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock͸୯७ͳUnitςετʹ͸ྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ

    • ༧ଌෆೳͳঢ়ଶͰͷςετ
  24. Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷ • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ • Google

    Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ • αʔϏεɺpackage, interface
  25. Google Cloud Pub/Sub ͷ Fake

  26. Conformance Test for Fake • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔ • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ

  27. None
  28. ςετͷछྨ • Unit Test / Integration Test • E2E Test

    • Chaos Test • QA
  29. Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ

    • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻ • σʔλϕʔεͷڞ༗͕໰୊
  30. σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊ • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍) •

    1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ
  31. E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ • docker-composeͰ؀ڥߏங • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ

    • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ
  32. Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network

    conditions • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠ • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
  33. QA (Quality Assurance) • PostmanΛ࢖ͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI •

    QAͷਓࣗ਎͕ߦ͍ͬͯΔ!
  34. ·ͱΊ • ϚΠΫϩαʔϏε͸೉͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢ •

    MockΑΓ΋Fake
  35. We’re Hiring! https://www.merpay.com/jp/careers/index.html