Software Engineer, Infrastructure

Slide 1

Slide 1 text

Tatsuhiko Kubo@cubicdaiya mercari.go#2 2018/08/10 Software Engineer, Infrastructure

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo Principal Software Engineer, SRE @ Mercari, Inc.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Agenda • Software Engineer, Infrastructure ϙδγϣϯͷ঺հ • ϝϧΧϦͷཪଆͰՔಇ͢ΔGoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ

Slide 5

Slide 5 text

Software Engineer, Infrastructure https://careers.mercari.com/job/

Slide 6

Slide 6 text

Software Engineer, Infrastructure https://careers.mercari.com/job/

Slide 7

Slide 7 text

Software Engineer, Infrastructure • Role • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΔߴ͍ύϑΥʔϚϯε΍৴པੑͷߴ͍Πϯϑϥε τϥΫνϟͷ࣮ݱ • ΞϓϦέʔγϣϯ΍ΠϯϑϥपΓͰڞ௨͢Δٕज़త՝୊Λղܾ͢ΔͨΊͷπʔϧ΍ϛ υϧ΢ΣΞͷ։ൃɾӡ༻ • Software Engineer, Site Reliability (SRE)ͱͷҧ͍ • ਖ਼௚ʹݴ͏ͱɺͦ͜·Ͱ໌֬ʹ۠ผ͞ΕΔΘ͚Ͱ͸ͳ͍ʢॴଐ͸SREνʔϜʣ • ҰํͰɺSREΑΓ΋։ൃۀ຿ͷൺॏ͕େ͖͍ʢ7:3͔8:2͘Β͍ʣ

Slide 8

Slide 8 text

Software Engineer, Infrastructure • ։ൃݴޠ͸Go͕த৺ • ωοτϫʔΫ΍γεςϜϓϩάϥϛϯάدΓͷ஌ࣝ΍εΩϧ͕ཁٻ͞ΕΔ৔໘͕ଟΊ • ΠϯϑϥετϥΫνϟͷج൫΍Τοδέʔε޲͚ͷαʔόαΠυϛυϧ΢ΣΞͷ ։ൃɾӡ༻ • ֤छΦϖϨʔγϣϯͷࣗಈԽ΍ϞχλϦϯά޲͚πʔϧ౳ͷ։ൃɾϝϯς • ύϑΥʔϚϯενϡʔχϯά΍ϘτϧωοΫͷௐࠪɾվળ౳ɺ໨ͷલʹ͋Δٕज़త՝ ୊Λιϑτ΢ΣΞΤϯδχΞϦϯάͷྗʹΑͬͯൃݟɾղܾ͢Δͷ͕ϛογϣϯʢ͜ Ε͸SRE΋Ұॹʣ

Slide 9

Slide 9 text

ಥવͰ͕͢ɺϚΠΫϩαʔϏε͕ྲྀߦΓͰ͢Ͷ

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

https://tech.mercari.com/entry/2018/07/11/180500

Slide 12

Slide 12 text

https://tech.mercari.com/entry/2018/07/11/180500

Slide 13

Slide 13 text

࣮ࡍͷͱ͜Ζɺ • ϝϧΧϦͰGo͕ར༻͞Ε࢝Ίͨͷ͸2014೥͔Β • αʔϏεͷػೳ։ൃͰ͸౰͔࣌ΒPHP͕ϝΠϯͩͬͨ • ҰํɺཪଆͰՔಇ͢Δπʔϧ΍ϛυϧ΢ΣΞͳΜ͔͸ ౰͔࣌ΒGo΍Node.js΋ͪΒ΄Β • ࢝·Γ͸ӡ༻޲͚πʔϧͷ։ൃ΍ύϑΥʔϚϯε͕ཁٻ͞ΕΔՕॴͰͷར༻ • GKE+GoΛத৺ʹਾ͑ͯϚΠΫϩαʔϏεԽΛਪਐ <- ΠϚίί

Slide 14

Slide 14 text

ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨPHP • PHP͸7ʹͳͬͯେ෼଎͘ͳͬͨ • CPU࢖༻཰͸൒෼ҎԼ • ॲཧ͕࣌ؒ਺े%ݮ • ͔͠͠ɺ୯ମੑೳ͕޲্͚ͨͩ͠Ͱ΢ΟʔΫϙΠϯτ͸ͦͷ·· • γϯάϧεϨουͳͷͰϚϧνίΞΛ׆͔ͤͳ͍ • ඇಉظॲཧ͕ۤख PHP5.6 -> PHP7.1 Response time avg PHP5.6 -> PHP7.1

Slide 15

Slide 15 text

• े෼ͳੑೳ • ϐʔΫ࣌Ͱඵؒ਺ઍreq/secΛࡹ͍ͯΔ ͱ͋ΔGoͷHTTP APIαʔόͷ CPU࢖༻཰ͷάϥϑˠ • ϨεϙϯελΠϜ͸avgͰ4msɺ90~99percentileͰ10~20msલޙ • ඇಉظॲཧ΍ฒߦੑͷߴ͍ϓϩάϥϜͷ࣮ݱ͕Մೳ • ΰϧʔνϯ΍νϟωϧʹΑΔڧྗͳαϙʔτ ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨGo

Slide 16

Slide 16 text

• γϯάϧόΠφϦͳͷͰ഑෍ָ͕ • γεςϜӡ༻ऀతʹָ͘͢͝ • ύοέʔδϯά΋ָ πʔϧ։ൃݴޠͱͯ͠ͷGo

Slide 17

Slide 17 text

• DockerͰͲ͔ͬʔΜ • RPM • πʔϧ΍ϛυϧ΢ΣΞྨ͸ΞϓϦέʔγϣϯͱൺ΂ͯͦΜͳʹසൟʹߋ ৽͠ͳ͍ͷͰ΄ͱΜͲͷ৔߹͸͜ΕͰे෼ ϝϧΧϦͰͷ࣮ࡍͷ഑෍ํ๏

Slide 18

Slide 18 text

DockerͱMakeʹΑΔRPMύοέʔδͷϏϧυγεςϜ https://tech.mercari.com/entry/2016/08/15/163219

Slide 19

Slide 19 text

ෳ਺ͷOS޲͚ʹDockerΛىಈͯ͠Ϗϧυ

Slide 20

Slide 20 text

࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package & upload to S3

Slide 21

Slide 21 text

࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package & upload to S3 JP US UK aws s3 sync yum repo yum repo yum repo

Slide 22

Slide 22 text

ϝϧΧϦͷཪଆͰՔಇ͢Δ GoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ

Slide 23

Slide 23 text

slackboard

Slide 24

Slide 24 text

slackboard • Slack Incoming WebhookϓϩΩγ݉ΫϥΠΞϯτ • github.com/cubicdaiya/slackboard • Slack΁ͷ௨஌ॲཧ΍ඞཁͳ৘ใΛҰݩ؅ཧԽ • ௨஌ઌ΍Webhook URLઃఆͷू໿ • ΫϥΠΞϯτϓϩάϥϜͷ౷ҰʹΑΔϝϯςφϏϦςΟͷ޲্ • ౰࣌(2014೥ࠒ)ࣾ಺ͰPHP΍Ruby΍PythonͷSlackΫϥΠΞϯτ͕ཚཱͯͨ͠

Slide 25

Slide 25 text

slackboard conﬁguration [core] # listen port port = “29800” # URL for Incoming Webhooks slack_url = “https://hooks.slack.com/services/…” [log] # access log path or redirector access_log = “stdout” # error log path or redirector error_log = “stderr” # log level level = “error”

Slide 26

Slide 26 text

Usage of slackboard-cli $ slackboard -c /path/to/slackboard.toml Server Client $ echo mercari | slackboard-cli -c test -s 127.0.0.1:29800

Slide 27

Slide 27 text

Usage of slackboard-log $ slackboard -c /path/to/slackboard.toml Server Client $ slackboard-log -c test -s 127.0.0.1:29800 — ls notfound

Slide 28

Slide 28 text

࣮ࡍͷγεςϜߏ੒ by slackboard asia region us region eu region eu region asia region us region Cloud Load Balancing GCP TMBDLCPBSE TMBDLCPBSE TMBDLCPBSE

Slide 29

Slide 29 text

ͳ͓ɺϝϧΧϦͰ࣮ࡍʹ࢖ΘΕ͍ͯΔΫϥΠΞϯτ͸Perl੡ͷslacklog $ slacklog -c test --log-channel test -- ls notfound

Slide 30

Slide 30 text

pvpool

Slide 31

Slide 31 text

pvpool • ϝϧΧϦͷ঎඼Ӿཡ਺ΛΧ΢ϯτΞοϓ͢ΔͨΊͷ࢓૊Έ • ࢦఆ͞Εͨ঎඼IDͷӾཡ਺ͷΧ΢ϯτΞοϓ͓Αͼ औಘͷͨΊͷAPIΛఏڙ • ※ ը໘͸։ൃ༻ͷ΋ͷͰ͢

Slide 32

Slide 32 text

঎඼Ӿཡ਺Χ΢ϯτΞοϓͷϫʔΫϩʔυͱٕज़తͳཁ݅ • ঎඼Ӿཡʹؔ͢ΔAPI΁ͷϦΫΤετ਺͸ϝϧΧϦͷશAPIதTOP3ʹೖΔ • ϝϧΧϦAPI΁ͷશϦΫΤετ਺͸ϐʔΫ࣌Ͱ໿56,000req/sec • ੑೳཁ݅ • ग़དྷΔݶΓ௿஗ԆͰԠ౴Ͱ͖ΔAPIΛఏڙ͢Δʢ໨҆ɿ਺ᶡඵҎ಺ʣ • ग़དྷΔݶΓϦΞϧλΠϜʹӾཡ਺Λ࠷৽ͷঢ়ଶʹ൓ө͢Δʢ໨҆ɿ਺ඵ Ҏ಺ʣ

Slide 33

Slide 33 text

System architecture of pvpool • nginx • pvpoold • Powered by Go (net/http) • MySQL pvpoold pvpoold nginx MySQL pvpool.local by consul DNS : HTTP protocol : MySQL protocol The gopher was designed by Renée French

Slide 34

Slide 34 text

ఏڙ͢ΔAPI͸2ͭ • POST /items/pvc • ༩͑ΒΕͨ঎඼IDຖͷPVΛΧ΢ϯτΞοϓ • POST /items/pv • ༩͑ΒΕͨ঎඼IDຖͷPVΛฦ͢ • ঎඼ID͸ෳ਺ࢦఆՄೳ

Slide 35

Slide 35 text

pvpoold Internals POST /items/pvc { "item_ids" [ ... ] } PVCHandler Sharding Item IDs MySQL Slot Slot Slot Slot Slot Slot Pooling query Flushing periodically synchronous processing asynchronous processing

Slide 36

Slide 36 text

pvpoold Internals • PVCHandler • HTTPϋϯυϥؔ਺ • Slot • Χ΢ϯτΞοϓ༻ϓϩηεϢχοτ by ΰϧʔνϯ

Slide 37

Slide 37 text

pvpoold Internals • Slot಺ͷొ৔ਓ෺ • Slot -> chan types.ItemID • PVMap -> map[types.ItemID]int • Storer -> Slot͔Β঎඼IDΛϑΣονͯ͠PVMap಺ͷ஋Λߋ৽ • Flusher -> ఆظతʹPVMap಺ͷ஋ΛऔΓग़ͯ͠MySQLʹߋ৽ΫΤϦΛൃߦ • Storer ͱ Flusher ΋·ͨΰϧʔνϯ

Slide 38

Slide 38 text

Slot overview Slot Storer Flusher PVMap dequeue item ID Store pv per item Fetch pv per item periodically MySQL Issue SQL to count up

Slide 39

Slide 39 text

Storer func (slot *PVCSlot) storer() { for { itemID := <- slot.Slot slot.Lock() slot.PVMap[itemID]++ slot.Unock() } }

Slide 40

Slide 40 text

Flusher func (slot *PVCSlot) ﬂusher(interval time.Duration) { ticker := time.NewTicker(interval) for { <- ticker.C slot.Lock() if err := slot.ﬂush(); err != nil { // transaction // error handling } slot.Unlock() } }

Slide 41

Slide 41 text

StorerͱFlusherͷ໾ׂ • Storer • MySQL΁ൃߦ͢ΔSQLΛ঎඼IDຖʹ·ͱΊΔ • eg. UPDATE pv = pv + 5 instead of ﬁve of UPDATE pv = pv +1 • Flusher • ҰఆִؒͰMySQLʹߋ৽༻SQLΛൃߦ͢Δ • ঎඼͕Ӿཡ͞ΕΔ౓ʹUPDATE͕૸ΔͷΛආ͚ͯෛՙΛ࿨Β͛Δ

Slide 42

Slide 42 text

go-httpstats

Slide 43

Slide 43 text

go-httpstats • ʢػೳ͕গͳ͍ͱ͍͏ҙຯͰʣখن໛ͳHTTPαʔό޲͚ͷϝτϦΫεऔಘϥΠϒϥϦ • ϦΫΤετ਺ʢτʔλϧͱεςʔλείʔυຖʣ • ϨεϙϯελΠϜʢmaxɺminɺavgɺpercentileຖʣ • Not published • ։ൃͨ͠ͷ͸ࠓ೥৽ଔೖࣾͷSoftware Engineer, Infrastructure • ͢Ͱʹ͋Δ΋ͷΛ୳͚ͨ͠Ͳɺͪΐ͏Ͳ͍͍΋ͷ͕ͳ͔ͬͨͷͰࣗ࡞

Slide 44

Slide 44 text

Getting Started - go-httpstats mw := stats.New() handler := mw.WrapHandleFunc( http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(“Hello, World”)) })) statsHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { d, err := mw.Data() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }) http.Handle(“/“, handler) http.Handle(“/stats”, statshandler) http.ListenAndServe(“:9999”, nil)

Slide 45

Slide 45 text

Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’ { “count”: 13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } }

Slide 46

Slide 46 text

Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }

Slide 47

Slide 47 text

Sampling factor const ( // DefaultBufferSize is buf size sampling response time for percentile and average DefaultBufferSize = 1000 // DefaultSamplingFactor is factor to sample response time for percentile and average DefaultSamplingFactor = 1 ) // … // mw := stats.New() // NewCapa returns a new statistics structure // buffer size of HTTP requests is allocated length of bufsize // If bufsize is less than 2, NewCapa returns error mw, err := stats.NewCapa(bufsize, factor)

Slide 48

Slide 48 text

Introducing sampling factor • percentileͷܭࢉ͸ͦͦ͜͜ॏ͍ • avgͷܭࢉͱҧͬͯιʔτ͕ඞཁ • ϨεϙϯελΠϜܥ͸౷ܭσʔλͳͷͰඞͣ͠΋શ෦औΔඞཁ͸ͳ͍ • ϫʔΫϩʔυͷܹ͍͠ՕॴͩͱόοϑΝॻ͖ࠐΈͷMutexڝ߹͕ܹͯ͘͠ ໰୊ʹͳͬͨͷͰɺͦͷϫʔΫΞϥ΢ϯυͱͯ͠ಋೖ

Slide 49

Slide 49 text

࠷ऴతʹ͸MackerelͰάϥϑԽ $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’ { “count”: 13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } } $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }

Slide 50

Slide 50 text

mackerelϓϥάΠϯ΋GoͰ։ൃ • MackerelϓϥάΠϯ։ൃ༻ϔϧύʔΛ࢖͏ͱָ • github.com/mackerelip/go-mackerel-plugin • specϑΝΠϧ࡞੒ޙɺmake mackerel-plugin-nameͰRPMύοέʔδΛϏ ϧυ & ഑෍ • ฐࣾͷಠࣗMackerelϓϥάΠϯ͸Go & Perl੡ • https://speakerdeck.com/kazeburo/mackerel-day

Slide 51

Slide 51 text

mfc

Slide 52

Slide 52 text

mfc • In-house Fastly CLI tool at Mercari • Early development phase • ϝϧΧϦͰ͸2017೥͔ΒFastlyΛར༻த • ػೳͷ໢ཏੑࣗମ͸ͦΜͳʹߴ͘ͳ͍ • ඞཁʹͳͬͨΓɺ͋Δͱศརͦ͏ͳػೳΛ౎౓ΠϯςάϨʔτͯ͠Δײ͡ • ΏΔ;Θʹ։ൃͯ͠·͢

Slide 53

Slide 53 text

mfc conﬁguration $ cat ~/.fastly/conf.ini service = xxx apikey = yyy …

Slide 54

Slide 54 text

Usage of mfc • $ mfc Usage of mfc: config the utility for mfc configuration service the utility for fastly service acl the utility for fastly ACL (etc…) • ػೳྖҬຖʹαϒίϚϯυΛఆٛ • ACL, Service, Version౳ • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹ switch args[1] { case “config”: return config.NewCLI().Run(args) case “service”: return service.NewCLI().Run(args) case “acl”: return acl.NewCLI().Run(args) case “…” … } ಈ࡞Πϝʔδ

Slide 55

Slide 55 text

mfcͷػೳ(Ұ෦) $ mfc config -h Usage of config: mfc config list output mfc configuration $ mfc service -h Usage of service: mfc service vcldiff from_version to_version output difference between from and to mfc service versions output active and latest versions $ mfc acl -h Usage acl: mfc acl create create acl mfc acl remove remove acl mfc acl add add given entry to acl mfc acl del delete given entry in acl mfc acl show output acl mfc acl list output acl entries mfc acl sync sync acl with given provider file

Slide 56

Slide 56 text

imageﬂux-cli

Slide 57

Slide 57 text

imageflux-cli • In-house ImageFlux CLI tool for Mercari • github.com/mercari/imageflux-cli • ImageFlux ~ Ϋϥ΢υը૾ม׵αʔϏε • https://www.sakura.ad.jp/services/imageflux/ • Reference • ImageFluxΛར༻ͨ͠ը૾഑৴ͷ࠷దԽʙಈతϦαΠζͱWebPม׵ʙ • https://tech.mercari.com/entry/2018/01/30/161001

Slide 58

Slide 58 text

imageflux-cli configuration $ cat ~/.imageflux/conf.ini token = xxx

Slide 59

Slide 59 text

Usage of imageflux-cli $ imageflux-cli -h Usage: imageflux-cli cache.lookup -k $url imageflux-cli cache.delete -k $url imageflux-cli signature -s $secret -p $path

Slide 60

Slide 60 text

΄͔ʹ΋͍Ζ͍Ζ • Gaurun - Building high performance push notiﬁcation server in Go • https://speakerdeck.com/cubicdaiya/building-high-performance-push-notiﬁcation-server-in- go • chocon - Simple proxy server for persisting connection between upstream servers • https://speakerdeck.com/kazeburo/cloud-connect-the-world-as-a-glue-aws-dev-day-2017? slide=53 • Mercari API Gateway • https://go-talks.appspot.com/github.com/tcnksm/talks/2018/07/mercarigo/microservices-api- gateway.slide#1 • etc… ( There are too many works and products in Go at Mercari)

Slide 61

Slide 61 text

·ͱΊ • Software Engineer, Infrastructureͷ঺հ • ओʹGoͰαʔόαΠυϛυϧ΢ΣΞ΍ΒCLIπʔϧͷ։ൃΛ΍ͬͯ·͢ • ݱࡏ3ਓ(͏ͪ1ਓ͸SREͱ݉຿)

Slide 62

Slide 62 text

We are hiring Go Software Engineer! https://careers.mercari.com/job/