Testing with microservices in merpay

by kazegusuri

Slide 1

Slide 1 text

Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices in merpay

Slide 2

Slide 2 text

Agenda • ࣗݾ঺հ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ͸ Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri

Slide 3

Slide 3 text

ࣗݾ঺հ • ࠤ໺ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh, Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ޷͖ͳ͜ͱ • gRPC

Slide 4

Slide 4 text

@kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷར఺ɾܽ఺

Slide 5

Slide 5 text

ϝϧΧϦʹ͓͚ΔgRPC • 2015೥຤ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜Ε΋ܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ΁

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes (GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry

Slide 9

Slide 9 text

ܾࡁͷٕज़తͳ೉͠͞ͱָ͠͞ • ܾࡁ͸೉͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ • ΤϯδχΞϦϯάͰղܾ

Slide 10

Slide 10 text

ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰ͸τϥϯβΫγϣϯ಺Ͱ׬݁ • Microservices ArchitectureͰ͸ඞવతʹτϥϯβΫγϣϯ͸෼཭ • ݁Ռ੔߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue

Slide 11

Slide 11 text

ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛ؅ཧ • ঢ়ଶભҠϞσϧ • ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ΋࠶։Մೳ

Slide 12

Slide 12 text

ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3 4 Init 1 2 3 4 Fin

Slide 13

Slide 13 text

Payment Point Sales Microservices Payment Provider 1 2 3 4 Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ

Slide 14

Slide 14 text

ਓྨ͕ਖ਼࣮͘͠૷͢Δͷ͸೉͍͠

Slide 15

Slide 15 text

Ͳ͏΍ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞੒ • ֤ঢ়ଶຖʹ͋Δ΂͖ঢ়ଶ(σʔλͷଘࡏɺ஋)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ • ͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ

Slide 16

Slide 16 text

ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ

Slide 17

Slide 17 text

Failure Injection • ֎෦௨৴ʹରͯ͠೚ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏε΁ͷHTTPϦΫΤετ • ಺෦αʔϏε΁ͷgRPCϦΫΤετ • ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ

Slide 18

Slide 18 text

Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷ׬ྃޙ • Commitͷ׬ྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕࢓ࠐΊͯศར!

Slide 19

Slide 19 text

Failure Injection (MySQL)

Slide 20

Slide 20 text

Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ • net/http.RoundTripper ʹΑΔ࣮૷

Slide 21

Slide 21 text

Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷ׬ྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴͸શͯhook͢Δ • Client InterceptorʹΑΔ࣮૷ • ·ͩಋೖͰ͖͍ͯͳ͍

Slide 22

Slide 22 text

Random or Deterministic? • Ͳ͏͍͏৚݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬཰Λ͋͛Δͱ੒ޭ͠ͳ͘ͳΓςετ͕࣌ؒ௕͘ͳΔ • Τϥʔͷൃੜ͸ܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұ౓ൃੜͨ͠ύεͰ͸2౓ͱൃੜͤ͞ͳ͍

Slide 23

Slide 23 text

αʔϏε΁ͷϦΫΤετͷςετ • ςετͰ֎෦ɾ಺෦αʔϏεΛ௚઀࢖͏ͷ͸೉͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock͸୯७ͳUnitςετʹ͸ྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ • ༧ଌෆೳͳঢ়ଶͰͷςετ

Slide 24

Slide 24 text

Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ૷ • σʔλϕʔεͳͲʹґଘͤͣʹߴ଎ʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δ΋ͷ΋͋Δ • Google Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞੒͢Δ • αʔϏεɺpackage, interface

Slide 25

Slide 25 text

Google Cloud Pub/Sub ͷ Fake

Slide 26

Slide 26 text

Conformance Test for Fake • FakeͷৼΔ෣͍͕ຊདྷͷ࣮૷ͱဃ཭͍ͯ͠ͳ͍͔ • ظ଴͢ΔৼΔ෣͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮૷ͱFakeͷ྆ํʹର࣮ͯ͠ߦ

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

ςετͷछྨ • Unit Test / Integration Test • E2E Test • Chaos Test • QA

Slide 29

Slide 29 text

Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͸͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ • ґଘinterface͸શͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏε΁ͷ઀ଓ͸FakeΛར༻ • σʔλϕʔεͷڞ༗͕໰୊

Slide 30

Slide 30 text

σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱ઀ଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕໰୊ • ͳΒDBΛςετຖʹཱͯΕ͹ྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞੒(instanceͰ͸ͳ͍) • 1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳ਺ΫΤϦΛ·Δ͝ͱ౤͛ΒΕΔ • ࡞੒ʹ͔͔Δ࣌ؒ͸1ඵ΄Ͳ

Slide 31

Slide 31 text

E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ౤͛ͯৼΔ෣͍Λςετ • ՄೳͳݶΓґଘαʔϏε΋ىಈ͢Δ • docker-composeͰ؀ڥߏங • ςετؒͷಠཱੑ͸ςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ • ͜͜͸·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰ΋ෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετ΍ڝ߹͢ΔΑ͏ͳϦΫΤετ

Slide 32

Slide 32 text

Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network conditions • ஗Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ࢖͍Ͳ͜Ζ͕೉͍͠ • ࠓͷͱ͜Ζ͸஗ԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻

Slide 33

Slide 33 text

QA (Quality Assurance) • PostmanΛ࢖ͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • Postmanͷςετέʔε΋GitHubͰ؅ཧͯ͠CI • QAͷਓࣗ਎͕ߦ͍ͬͯΔ!

Slide 34

Slide 34 text

·ͱΊ • ϚΠΫϩαʔϏε͸೉͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲໨ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόά΋ݟ͚ͭ΍͘͢ • MockΑΓ΋Fake

Slide 35

Slide 35 text

We’re Hiring! https://www.merpay.com/jp/careers/index.html