Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing with microservices in merpay

Testing with microservices in merpay

Go Conference 2018 Spring

kazegusuri

April 14, 2018
Tweet

More Decks by kazegusuri

Other Decks in Programming

Transcript

  1. Go Conference 2018 Spring
    Masahiro Sano (@kazegusuri)
    Testing with microservices in merpay

    View Slide

  2. Agenda
    • ࣗݾ঺հ
    • ϝϧϖΠͷܾࡁγεςϜ
    • ςετϑϨʔϜϫʔΫ
    • ςετͷछྨ
    εϥΠυ͸ Speaker Deck ʹ͋Γ·͢ʂ
    https://speakerdeck.com/kazegusuri

    View Slide

  3. ࣗݾ঺հ
    • ࠤ໺ ਖ਼ߒ (@kazegusuri)
    • Mercari, Inc → Souzoh, Inc → merpay, Inc
    • Principal Software Engineer
    • Backend Engineer
    • GoͰܾࡁγεςϜશൠΛ։ൃத
    • ޷͖ͳ͜ͱ
    • gRPC

    View Slide

  4. @kazegusuri
    Go Conference 2016 Spring
    GRPCͷ࣮ફͱ
    ݱঢ়Ͱͷར఺ɾܽ఺

    View Slide

  5. ϝϧΧϦʹ͓͚ΔgRPC
    • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ
    • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ
    • ͦͯ͠ϝϧϖΠ΁

    View Slide

  6. View Slide

  7. View Slide

  8. ϝϧϖΠͷٕज़ελοΫ
    • Go + gRPC
    • Microservices Architecture
    • Kubernetes (GKE) + (EKS?)
    • ͦͷଞ
    • CircleCI, Spinnaker
    • Stackdriver Logging, Profiler
    • Datadog (Monitoring, Tracing)
    • Pagerduty, Sentry

    View Slide

  9. ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞
    • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ)
    • ܾࡁϓϩόΠμͱͷҰ؏ੑ
    • ΫϦςΟΧϧ
    • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
    • ΤϯδχΞϦϯάͰղܾ

    View Slide

  10. ϚΠΫϩαʔϏεͱҰ؏ੑ
    • Microservices Architecture is super difficult!!
    • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁
    • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭
    • ݁Ռ੔߹ੑ
    • Idempotency
    • Retry, retry, retry
    • Async, Message Queue

    View Slide

  11. ϝϧϖΠͷܾࡁγεςϜ
    • Microservices Architecture
    • ػೳຖʹϚΠΫϩαʔϏεΛߏங
    • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ
    • ঢ়ଶભҠϞσϧ
    • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ
    • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ

    View Slide

  12. ϝϧϖΠͷܾࡁγεςϜ
    Payment
    Point Sales Microservices
    Payment
    Provider
    1
    2
    3
    4
    Init 1 2 3 4 Fin

    View Slide

  13. Payment
    Point Sales Microservices
    Payment
    Provider
    1
    2
    3
    4
    Init 1 2 3 4 Fin
    Keep retrying with delay
    ϝϧϖΠͷܾࡁγεςϜ

    View Slide

  14. ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠

    View Slide

  15. Ͳ͏΍ͬͯςετ͢Δ͔
    • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒
    • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ
    • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ
    • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ
    • ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢
    Δ͔֬ೝ͢Δ

    View Slide

  16. ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ

    View Slide

  17. Failure Injection
    • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ
    • MySQL
    • ֎෦αʔϏε΁ͷHTTPϦΫΤετ
    • ಺෦αʔϏε΁ͷgRPCϦΫΤετ
    • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ

    View Slide

  18. Failure Injection (MySQL)
    • ΤϥʔλΠϛϯά
    • Query
    • Execͷ׬ྃޙ
    • Commitͷ׬ྃޙ
    • શͯͷMySQL queryʹରͯ͠hook͢Δ
    • github.com/shogo82148/go-sql-proxy
    • ༷ʑͳhook͕࢓ࠐΊͯศར!

    View Slide

  19. Failure Injection (MySQL)

    View Slide

  20. Failure Injection (HTTP)
    • ΤϥʔλΠϛϯά
    • HTTPϦΫΤετͷ׬ྃޙ
    • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ
    • net/http.RoundTripper ʹΑΔ࣮૷

    View Slide

  21. Failure Injection (gRPC)
    • ΤϥʔλΠϛϯά
    • gRPCϦΫΤετͷ׬ྃޙ
    • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ
    • Client InterceptorʹΑΔ࣮૷
    • ·ͩಋೖͰ͖͍ͯͳ͍

    View Slide

  22. Random or Deterministic?
    • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔
    • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ
    • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ
    • Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ
    • stacktrace(runtime.Callers) Ͱܾఆ
    • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍

    View Slide

  23. αʔϏε΁ͷϦΫΤετͷςετ
    • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠
    • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ
    • Mock͸୯७ͳUnitςετʹ͸ྑ͍
    • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ
    ςετ
    • ༧ଌෆೳͳঢ়ଶͰͷςετ

    View Slide

  24. Fake Implementation
    • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷
    • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞
    • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ
    • Google Cloud Pub/Sub
    • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ
    • αʔϏεɺpackage, interface

    View Slide

  25. Google Cloud Pub/Sub ͷ Fake

    View Slide

  26. Conformance Test for Fake
    • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔
    • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ
    • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ

    View Slide

  27. View Slide

  28. ςετͷछྨ
    • Unit Test / Integration Test
    • E2E Test
    • Chaos Test
    • QA

    View Slide

  29. Unit and Integration Test
    • packageͰߦ͏ςετ
    • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍
    • ԼҐϨΠϠ·ͰؚΊͨςετ
    • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ
    • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻
    • σʔλϕʔεͷڞ༗͕໰୊

    View Slide

  30. σʔλϕʔεͷςετؒͷಠཱੑ
    • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ
    • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊
    • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ
    • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍)
    • 1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ
    • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ
    • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ

    View Slide

  31. E2E Test
    • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ
    • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ
    • docker-composeͰ؀ڥߏங
    • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
    • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍
    • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ
    • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ

    View Slide

  32. Chaos Test (Trying)
    • github.com/Shopify/toxiproxy
    • framework for simulating network conditions
    • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ
    • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠
    • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ
    ༧ଌʹར༻

    View Slide

  33. QA (Quality Assurance)
    • PostmanΛ࢖ͬͨAPIςετ
    • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻
    • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI
    • QAͷਓࣗ਎͕ߦ͍ͬͯΔ!

    View Slide

  34. ·ͱΊ
    • ϚΠΫϩαʔϏε͸೉͍͠
    • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ
    • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢
    • MockΑΓ΋Fake

    View Slide

  35. We’re Hiring!
    https://www.merpay.com/jp/careers/index.html

    View Slide