Software Engineer, Infrastructure

Software Engineer, Infrastructure

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

August 10, 2018
Tweet

Transcript

  1. Tatsuhiko Kubo@cubicdaiya mercari.go#2 2018/08/10 Software Engineer, Infrastructure

  2. @cubicdaiya / Tatsuhiko Kubo Principal Software Engineer, SRE @ Mercari,

    Inc.
  3. None
  4. Agenda • Software Engineer, Infrastructure ϙδγϣϯͷ঺հ • ϝϧΧϦͷཪଆͰՔಇ͢ΔGoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ

  5. Software Engineer, Infrastructure https://careers.mercari.com/job/

  6. Software Engineer, Infrastructure https://careers.mercari.com/job/

  7. Software Engineer, Infrastructure • Role • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΔߴ͍ύϑΥʔϚϯε΍৴པੑͷߴ͍Πϯϑϥε τϥΫνϟͷ࣮ݱ • ΞϓϦέʔγϣϯ΍ΠϯϑϥपΓͰڞ௨͢Δٕज़త՝୊Λղܾ͢ΔͨΊͷπʔϧ΍ϛ

    υϧ΢ΣΞͷ։ൃɾӡ༻ • Software Engineer, Site Reliability (SRE)ͱͷҧ͍ • ਖ਼௚ʹݴ͏ͱɺͦ͜·Ͱ໌֬ʹ۠ผ͞ΕΔΘ͚Ͱ͸ͳ͍ʢॴଐ͸SREνʔϜʣ • ҰํͰɺSREΑΓ΋։ൃۀ຿ͷൺॏ͕େ͖͍ʢ7:3͔8:2͘Β͍ʣ
  8. Software Engineer, Infrastructure • ։ൃݴޠ͸Go͕த৺ • ωοτϫʔΫ΍γεςϜϓϩάϥϛϯάدΓͷ஌ࣝ΍εΩϧ͕ཁٻ͞ΕΔ৔໘͕ଟΊ • ΠϯϑϥετϥΫνϟͷج൫΍Τοδέʔε޲͚ͷαʔόαΠυϛυϧ΢ΣΞͷ ։ൃɾӡ༻

    • ֤छΦϖϨʔγϣϯͷࣗಈԽ΍ϞχλϦϯά޲͚πʔϧ౳ͷ։ൃɾϝϯς • ύϑΥʔϚϯενϡʔχϯά΍ϘτϧωοΫͷௐࠪɾվળ౳ɺ໨ͷલʹ͋Δٕज़త՝ ୊Λιϑτ΢ΣΞΤϯδχΞϦϯάͷྗʹΑͬͯൃݟɾղܾ͢Δͷ͕ϛογϣϯʢ͜ Ε͸SRE΋Ұॹʣ
  9. ಥવͰ͕͢ɺϚΠΫϩαʔϏε͕ྲྀߦΓͰ͢Ͷ

  10. None
  11. https://tech.mercari.com/entry/2018/07/11/180500

  12. https://tech.mercari.com/entry/2018/07/11/180500

  13. ࣮ࡍͷͱ͜Ζɺ • ϝϧΧϦͰGo͕ར༻͞Ε࢝Ίͨͷ͸2014೥͔Β • αʔϏεͷػೳ։ൃͰ͸౰͔࣌ΒPHP͕ϝΠϯͩͬͨ • ҰํɺཪଆͰՔಇ͢Δπʔϧ΍ϛυϧ΢ΣΞͳΜ͔͸ ౰͔࣌ΒGo΍Node.js΋ͪ´ • ࢝·Γ͸ӡ༻޲͚πʔϧͷ։ൃ΍ύϑΥʔϚϯε͕ཁٻ͞ΕΔՕॴͰͷར༻

    • GKE+GoΛத৺ʹਾ͑ͯϚΠΫϩαʔϏεԽΛਪਐ <- ΠϚίί
  14. ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨPHP • PHP͸7ʹͳͬͯେ෼଎͘ͳͬͨ • CPU࢖༻཰͸൒෼ҎԼ • ॲཧ͕࣌ؒ਺े%ݮ • ͔͠͠ɺ୯ମੑೳ͕޲্͚ͨͩ͠Ͱ΢ΟʔΫϙΠϯτ͸ͦͷ·· •

    γϯάϧεϨουͳͷͰϚϧνίΞΛ׆͔ͤͳ͍ • ඇಉظॲཧ͕ۤख PHP5.6 -> PHP7.1 Response time avg PHP5.6 -> PHP7.1
  15. • े෼ͳੑೳ • ϐʔΫ࣌Ͱඵؒ਺ઍreq/secΛࡹ͍ͯΔ ͱ͋ΔGoͷHTTP APIαʔόͷ CPU࢖༻཰ͷάϥϑˠ • ϨεϙϯελΠϜ͸avgͰ4msɺ90~99percentileͰ10~20msલޙ •

    ඇಉظॲཧ΍ฒߦੑͷߴ͍ϓϩάϥϜͷ࣮ݱ͕Մೳ • ΰϧʔνϯ΍νϟωϧʹΑΔڧྗͳαϙʔτ ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨGo
  16. • γϯάϧόΠφϦͳͷͰ഑෍ָ͕ • γεςϜӡ༻ऀతʹָ͘͢͝ • ύοέʔδϯά΋ָ πʔϧ։ൃݴޠͱͯ͠ͷGo

  17. • DockerͰͲ͔ͬʔΜ • RPM • πʔϧ΍ϛυϧ΢ΣΞྨ͸ΞϓϦέʔγϣϯͱൺ΂ͯͦΜͳʹසൟʹߋ ৽͠ͳ͍ͷͰ΄ͱΜͲͷ৔߹͸͜ΕͰे෼ ϝϧΧϦͰͷ࣮ࡍͷ഑෍ํ๏

  18. DockerͱMakeʹΑΔRPMύοέʔδͷϏϧυγεςϜ https://tech.mercari.com/entry/2016/08/15/163219

  19. ෳ਺ͷOS޲͚ʹDockerΛىಈͯ͠Ϗϧυ

  20. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package &

    upload to S3
  21. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package &

    upload to S3 JP US UK aws s3 sync yum repo yum repo yum repo
  22. ϝϧΧϦͷཪଆͰՔಇ͢Δ GoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ

  23. slackboard

  24. slackboard • Slack Incoming WebhookϓϩΩγ݉ΫϥΠΞϯτ • github.com/cubicdaiya/slackboard • Slack΁ͷ௨஌ॲཧ΍ඞཁͳ৘ใΛҰݩ؅ཧԽ •

    ௨஌ઌ΍Webhook URLઃఆͷू໿ • ΫϥΠΞϯτϓϩάϥϜͷ౷ҰʹΑΔϝϯςφϏϦςΟͷ޲্ • ౰࣌(2014೥ࠒ)ࣾ಺ͰPHP΍Ruby΍PythonͷSlackΫϥΠΞϯτ͕ཚཱͯͨ͠
  25. slackboard configuration [core] # listen port port = “29800” #

    URL for Incoming Webhooks slack_url = “https://hooks.slack.com/services/…” [log] # access log path or redirector access_log = “stdout” # error log path or redirector error_log = “stderr” # log level level = “error”
  26. Usage of slackboard-cli $ slackboard -c /path/to/slackboard.toml Server Client $

    echo mercari | slackboard-cli -c test -s 127.0.0.1:29800
  27. Usage of slackboard-log $ slackboard -c /path/to/slackboard.toml Server Client $

    slackboard-log -c test -s 127.0.0.1:29800 — ls notfound
  28. ࣮ࡍͷγεςϜߏ੒ by slackboard asia region us region eu region eu

    region asia region us region Cloud Load Balancing GCP TMBDLCPBSE TMBDLCPBSE TMBDLCPBSE
  29. ͳ͓ɺϝϧΧϦͰ࣮ࡍʹ࢖ΘΕ͍ͯΔΫϥΠΞϯτ͸Perl੡ͷslacklog $ slacklog -c test --log-channel test -- ls notfound

  30. pvpool

  31. pvpool • ϝϧΧϦͷ঎඼Ӿཡ਺ΛΧ΢ϯτΞοϓ͢ΔͨΊͷ࢓૊Έ • ࢦఆ͞Εͨ঎඼IDͷӾཡ਺ͷΧ΢ϯτΞοϓ͓Αͼ औಘͷͨΊͷAPIΛఏڙ • ※ ը໘͸։ൃ༻ͷ΋ͷͰ͢

  32. ঎඼Ӿཡ਺Χ΢ϯτΞοϓͷϫʔΫϩʔυͱٕज़తͳཁ݅ • ঎඼Ӿཡʹؔ͢ΔAPI΁ͷϦΫΤετ਺͸ϝϧΧϦͷશAPIதTOP3ʹೖΔ • ϝϧΧϦAPI΁ͷશϦΫΤετ਺͸ϐʔΫ࣌Ͱ໿56,000req/sec • ੑೳཁ݅ • ग़དྷΔݶΓ௿஗ԆͰԠ౴Ͱ͖ΔAPIΛఏڙ͢Δʢ໨҆ɿ਺ᶡඵҎ಺ʣ •

    ग़དྷΔݶΓϦΞϧλΠϜʹӾཡ਺Λ࠷৽ͷঢ়ଶʹ൓ө͢Δʢ໨҆ɿ਺ඵ Ҏ಺ʣ
  33. System architecture of pvpool • nginx • pvpoold • Powered

    by Go (net/http) • MySQL pvpoold pvpoold nginx MySQL pvpool.local by consul DNS : HTTP protocol : MySQL protocol The gopher was designed by Renée French
  34. ఏڙ͢ΔAPI͸2ͭ • POST /items/pvc • ༩͑ΒΕͨ঎඼IDຖͷPVΛΧ΢ϯτΞοϓ • POST /items/pv •

    ༩͑ΒΕͨ঎඼IDຖͷPVΛฦ͢ • ঎඼ID͸ෳ਺ࢦఆՄೳ
  35. pvpoold Internals POST /items/pvc { "item_ids" [ ... ] }

    PVCHandler Sharding Item IDs MySQL Slot Slot Slot Slot Slot Slot Pooling query Flushing periodically synchronous processing asynchronous processing
  36. pvpoold Internals • PVCHandler • HTTPϋϯυϥؔ਺ • Slot • Χ΢ϯτΞοϓ༻ϓϩηεϢχοτ

    by ΰϧʔνϯ
  37. pvpoold Internals • Slot಺ͷొ৔ਓ෺ • Slot -> chan types.ItemID •

    PVMap -> map[types.ItemID]int • Storer -> Slot͔Β঎඼IDΛϑΣονͯ͠PVMap಺ͷ஋Λߋ৽ • Flusher -> ఆظతʹPVMap಺ͷ஋ΛऔΓग़ͯ͠MySQLʹߋ৽ΫΤϦΛൃߦ • Storer ͱ Flusher ΋·ͨΰϧʔνϯ
  38. Slot overview Slot Storer Flusher PVMap dequeue item ID Store

    pv per item Fetch pv per item periodically MySQL Issue SQL to count up
  39. Storer func (slot *PVCSlot) storer() { for { itemID :=

    <- slot.Slot slot.Lock() slot.PVMap[itemID]++ slot.Unock() } }
  40. Flusher func (slot *PVCSlot) flusher(interval time.Duration) { ticker := time.NewTicker(interval)

    for { <- ticker.C slot.Lock() if err := slot.flush(); err != nil { // transaction // error handling } slot.Unlock() } }
  41. StorerͱFlusherͷ໾ׂ • Storer • MySQL΁ൃߦ͢ΔSQLΛ঎඼IDຖʹ·ͱΊΔ • eg. UPDATE pv =

    pv + 5 instead of five of UPDATE pv = pv +1 • Flusher • ҰఆִؒͰMySQLʹߋ৽༻SQLΛൃߦ͢Δ • ঎඼͕Ӿཡ͞ΕΔ౓ʹUPDATE͕૸ΔͷΛආ͚ͯෛՙΛ࿨Β͛Δ
  42. go-httpstats

  43. go-httpstats • ʢػೳ͕গͳ͍ͱ͍͏ҙຯͰʣখن໛ͳHTTPαʔό޲͚ͷϝτϦΫεऔಘϥΠϒϥϦ • ϦΫΤετ਺ʢτʔλϧͱεςʔλείʔυຖʣ • ϨεϙϯελΠϜʢmaxɺminɺavgɺpercentileຖʣ • Not published

    • ։ൃͨ͠ͷ͸ࠓ೥৽ଔೖࣾͷSoftware Engineer, Infrastructure • ͢Ͱʹ͋Δ΋ͷΛ୳͚ͨ͠Ͳɺͪΐ͏Ͳ͍͍΋ͷ͕ͳ͔ͬͨͷͰࣗ࡞
  44. Getting Started - go-httpstats mw := stats.New() handler := mw.WrapHandleFunc(

    http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(“Hello, World”)) })) statsHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { d, err := mw.Data() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }) http.Handle(“/“, handler) http.Handle(“/stats”, statshandler) http.ListenAndServe(“:9999”, nil)
  45. Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq

    ‘.request’ { “count”: 13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } }
  46. Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq

    ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }
  47. Sampling factor const ( // DefaultBufferSize is buf size sampling

    response time for percentile and average DefaultBufferSize = 1000 // DefaultSamplingFactor is factor to sample response time for percentile and average DefaultSamplingFactor = 1 ) // … // mw := stats.New() // NewCapa returns a new statistics structure // buffer size of HTTP requests is allocated length of bufsize // If bufsize is less than 2, NewCapa returns error mw, err := stats.NewCapa(bufsize, factor)
  48. Introducing sampling factor • percentileͷܭࢉ͸ͦͦ͜͜ॏ͍ • avgͷܭࢉͱҧͬͯιʔτ͕ඞཁ • ϨεϙϯελΠϜܥ͸౷ܭσʔλͳͷͰඞͣ͠΋શ෦औΔඞཁ͸ͳ͍ •

    ϫʔΫϩʔυͷܹ͍͠ՕॴͩͱόοϑΝॻ͖ࠐΈͷMutexڝ߹͕ܹͯ͘͠ ໰୊ʹͳͬͨͷͰɺͦͷϫʔΫΞϥ΢ϯυͱͯ͠ಋೖ
  49. ࠷ऴతʹ͸MackerelͰάϥϑԽ $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’ { “count”:

    13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } } $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }
  50. mackerelϓϥάΠϯ΋GoͰ։ൃ • MackerelϓϥάΠϯ։ൃ༻ϔϧύʔΛ࢖͏ͱָ • github.com/mackerelip/go-mackerel-plugin • specϑΝΠϧ࡞੒ޙɺmake mackerel-plugin-nameͰRPMύοέʔδΛϏ ϧυ &

    ഑෍ • ฐࣾͷಠࣗMackerelϓϥάΠϯ͸Go & Perl੡ • https://speakerdeck.com/kazeburo/mackerel-day
  51. mfc

  52. mfc • In-house Fastly CLI tool at Mercari • Early

    development phase • ϝϧΧϦͰ͸2017೥͔ΒFastlyΛར༻த • ػೳͷ໢ཏੑࣗମ͸ͦΜͳʹߴ͘ͳ͍ • ඞཁʹͳͬͨΓɺ͋Δͱศརͦ͏ͳػೳΛ౎౓ΠϯςάϨʔτͯ͠Δײ͡ • ΏΔ;Θʹ։ൃͯ͠·͢
  53. mfc configuration $ cat ~/.fastly/conf.ini service = xxx apikey =

    yyy …
  54. Usage of mfc • $ mfc Usage of mfc: config

    the utility for mfc configuration service the utility for fastly service acl the utility for fastly ACL (etc…) • ػೳྖҬຖʹαϒίϚϯυΛఆٛ • ACL, Service, Version౳ • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹ switch args[1] { case “config”: return config.NewCLI().Run(args) case “service”: return service.NewCLI().Run(args) case “acl”: return acl.NewCLI().Run(args) case “…” … } ಈ࡞Πϝʔδ
  55. mfcͷػೳ(Ұ෦) $ mfc config -h Usage of config: mfc config

    list output mfc configuration $ mfc service -h Usage of service: mfc service vcldiff from_version to_version output difference between from and to mfc service versions output active and latest versions $ mfc acl -h Usage acl: mfc acl create create acl mfc acl remove remove acl mfc acl add add given entry to acl mfc acl del delete given entry in acl mfc acl show output acl mfc acl list output acl entries mfc acl sync sync acl with given provider file
  56. imageflux-cli

  57. imageflux-cli • In-house ImageFlux CLI tool for Mercari • github.com/mercari/imageflux-cli

    • ImageFlux ~ Ϋϥ΢υը૾ม׵αʔϏε • https://www.sakura.ad.jp/services/imageflux/ • Reference • ImageFluxΛར༻ͨ͠ը૾഑৴ͷ࠷దԽʙಈతϦαΠζͱWebPม׵ʙ • https://tech.mercari.com/entry/2018/01/30/161001
  58. imageflux-cli configuration $ cat ~/.imageflux/conf.ini token = xxx

  59. Usage of imageflux-cli $ imageflux-cli -h Usage: imageflux-cli cache.lookup -k

    $url imageflux-cli cache.delete -k $url imageflux-cli signature -s $secret -p $path
  60. ΄͔ʹ΋͍Ζ͍Ζ • Gaurun - Building high performance push notification server

    in Go • https://speakerdeck.com/cubicdaiya/building-high-performance-push-notification-server-in- go • chocon - Simple proxy server for persisting connection between upstream servers • https://speakerdeck.com/kazeburo/cloud-connect-the-world-as-a-glue-aws-dev-day-2017? slide=53 • Mercari API Gateway • https://go-talks.appspot.com/github.com/tcnksm/talks/2018/07/mercarigo/microservices-api- gateway.slide#1 • etc… ( There are too many works and products in Go at Mercari)
  61. ·ͱΊ • Software Engineer, Infrastructureͷ঺հ • ओʹGoͰαʔόαΠυϛυϧ΢ΣΞ΍ΒCLIπʔϧͷ։ൃΛ΍ͬͯ·͢ • ݱࡏ3ਓ(͏ͪ1ਓ͸SREͱ݉຿)

  62. We are hiring Go Software Engineer! https://careers.mercari.com/job/