Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing with microservices in merpay

Testing with microservices in merpay

Go Conference 2018 Spring

kazegusuri

April 14, 2018
Tweet

More Decks by kazegusuri

Other Decks in Programming

Transcript

  1. Go Conference 2018 Spring
    Masahiro Sano (@kazegusuri)
    Testing with microservices in merpay

    View full-size slide

  2. Agenda
    • ࣗݾ঺հ
    • ϝϧϖΠͷܾࡁγεςϜ
    • ςετϑϨʔϜϫʔΫ
    • ςετͷछྨ
    εϥΠυ͸ Speaker Deck ʹ͋Γ·͢ʂ
    https://speakerdeck.com/kazegusuri

    View full-size slide

  3. ࣗݾ঺հ
    • ࠤ໺ ਖ਼ߒ (@kazegusuri)
    • Mercari, Inc → Souzoh, Inc → merpay, Inc
    • Principal Software Engineer
    • Backend Engineer
    • GoͰܾࡁγεςϜશൠΛ։ൃத
    • ޷͖ͳ͜ͱ
    • gRPC

    View full-size slide

  4. @kazegusuri
    Go Conference 2016 Spring
    GRPCͷ࣮ફͱ
    ݱঢ়Ͱͷར఺ɾܽ఺

    View full-size slide

  5. ϝϧΧϦʹ͓͚ΔgRPC
    • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ
    • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ
    • ͦͯ͠ϝϧϖΠ΁

    View full-size slide

  6. ϝϧϖΠͷٕज़ελοΫ
    • Go + gRPC
    • Microservices Architecture
    • Kubernetes (GKE) + (EKS?)
    • ͦͷଞ
    • CircleCI, Spinnaker
    • Stackdriver Logging, Profiler
    • Datadog (Monitoring, Tracing)
    • Pagerduty, Sentry

    View full-size slide

  7. ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞
    • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ)
    • ܾࡁϓϩόΠμͱͷҰ؏ੑ
    • ΫϦςΟΧϧ
    • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
    • ΤϯδχΞϦϯάͰղܾ

    View full-size slide

  8. ϚΠΫϩαʔϏεͱҰ؏ੑ
    • Microservices Architecture is super difficult!!
    • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁
    • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭
    • ݁Ռ੔߹ੑ
    • Idempotency
    • Retry, retry, retry
    • Async, Message Queue

    View full-size slide

  9. ϝϧϖΠͷܾࡁγεςϜ
    • Microservices Architecture
    • ػೳຖʹϚΠΫϩαʔϏεΛߏங
    • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ
    • ঢ়ଶભҠϞσϧ
    • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ
    • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ

    View full-size slide

  10. ϝϧϖΠͷܾࡁγεςϜ
    Payment
    Point Sales Microservices
    Payment
    Provider
    1
    2
    3
    4
    Init 1 2 3 4 Fin

    View full-size slide

  11. Payment
    Point Sales Microservices
    Payment
    Provider
    1
    2
    3
    4
    Init 1 2 3 4 Fin
    Keep retrying with delay
    ϝϧϖΠͷܾࡁγεςϜ

    View full-size slide

  12. ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠

    View full-size slide

  13. Ͳ͏΍ͬͯςετ͢Δ͔
    • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒
    • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ
    • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ
    • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ
    • ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢
    Δ͔֬ೝ͢Δ

    View full-size slide

  14. ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ

    View full-size slide

  15. Failure Injection
    • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ
    • MySQL
    • ֎෦αʔϏε΁ͷHTTPϦΫΤετ
    • ಺෦αʔϏε΁ͷgRPCϦΫΤετ
    • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ

    View full-size slide

  16. Failure Injection (MySQL)
    • ΤϥʔλΠϛϯά
    • Query
    • Execͷ׬ྃޙ
    • Commitͷ׬ྃޙ
    • શͯͷMySQL queryʹରͯ͠hook͢Δ
    • github.com/shogo82148/go-sql-proxy
    • ༷ʑͳhook͕࢓ࠐΊͯศར!

    View full-size slide

  17. Failure Injection (MySQL)

    View full-size slide

  18. Failure Injection (HTTP)
    • ΤϥʔλΠϛϯά
    • HTTPϦΫΤετͷ׬ྃޙ
    • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ
    • net/http.RoundTripper ʹΑΔ࣮૷

    View full-size slide

  19. Failure Injection (gRPC)
    • ΤϥʔλΠϛϯά
    • gRPCϦΫΤετͷ׬ྃޙ
    • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ
    • Client InterceptorʹΑΔ࣮૷
    • ·ͩಋೖͰ͖͍ͯͳ͍

    View full-size slide

  20. Random or Deterministic?
    • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔
    • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ
    • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ
    • Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ
    • stacktrace(runtime.Callers) Ͱܾఆ
    • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍

    View full-size slide

  21. αʔϏε΁ͷϦΫΤετͷςετ
    • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠
    • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ
    • Mock͸୯७ͳUnitςετʹ͸ྑ͍
    • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ
    ςετ
    • ༧ଌෆೳͳঢ়ଶͰͷςετ

    View full-size slide

  22. Fake Implementation
    • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷
    • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞
    • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ
    • Google Cloud Pub/Sub
    • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ
    • αʔϏεɺpackage, interface

    View full-size slide

  23. Google Cloud Pub/Sub ͷ Fake

    View full-size slide

  24. Conformance Test for Fake
    • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔
    • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ
    • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ

    View full-size slide

  25. ςετͷछྨ
    • Unit Test / Integration Test
    • E2E Test
    • Chaos Test
    • QA

    View full-size slide

  26. Unit and Integration Test
    • packageͰߦ͏ςετ
    • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍
    • ԼҐϨΠϠ·ͰؚΊͨςετ
    • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ
    • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻
    • σʔλϕʔεͷڞ༗͕໰୊

    View full-size slide

  27. σʔλϕʔεͷςετؒͷಠཱੑ
    • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ
    • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊
    • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ
    • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍)
    • 1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ
    • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ
    • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ

    View full-size slide

  28. E2E Test
    • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ
    • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ
    • docker-composeͰ؀ڥߏங
    • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
    • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍
    • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ
    • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ

    View full-size slide

  29. Chaos Test (Trying)
    • github.com/Shopify/toxiproxy
    • framework for simulating network conditions
    • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ
    • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠
    • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ
    ༧ଌʹར༻

    View full-size slide

  30. QA (Quality Assurance)
    • PostmanΛ࢖ͬͨAPIςετ
    • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻
    • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI
    • QAͷਓࣗ਎͕ߦ͍ͬͯΔ!

    View full-size slide

  31. ·ͱΊ
    • ϚΠΫϩαʔϏε͸೉͍͠
    • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ
    • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢
    • MockΑΓ΋Fake

    View full-size slide

  32. We’re Hiring!
    https://www.merpay.com/jp/careers/index.html

    View full-size slide