Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Software Engineer, Infrastructure

Software Engineer, Infrastructure

Tatsuhiko Kubo

August 10, 2018
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Software Engineer, Infrastructure • Role • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΔߴ͍ύϑΥʔϚϯε΍৴པੑͷߴ͍Πϯϑϥε τϥΫνϟͷ࣮ݱ • ΞϓϦέʔγϣϯ΍ΠϯϑϥपΓͰڞ௨͢Δٕज़త՝୊Λղܾ͢ΔͨΊͷπʔϧ΍ϛ

    υϧ΢ΣΞͷ։ൃɾӡ༻ • Software Engineer, Site Reliability (SRE)ͱͷҧ͍ • ਖ਼௚ʹݴ͏ͱɺͦ͜·Ͱ໌֬ʹ۠ผ͞ΕΔΘ͚Ͱ͸ͳ͍ʢॴଐ͸SREνʔϜʣ • ҰํͰɺSREΑΓ΋։ൃۀ຿ͷൺॏ͕େ͖͍ʢ7:3͔8:2͘Β͍ʣ
  2. Software Engineer, Infrastructure • ։ൃݴޠ͸Go͕த৺ • ωοτϫʔΫ΍γεςϜϓϩάϥϛϯάدΓͷ஌ࣝ΍εΩϧ͕ཁٻ͞ΕΔ৔໘͕ଟΊ • ΠϯϑϥετϥΫνϟͷج൫΍Τοδέʔε޲͚ͷαʔόαΠυϛυϧ΢ΣΞͷ ։ൃɾӡ༻

    • ֤छΦϖϨʔγϣϯͷࣗಈԽ΍ϞχλϦϯά޲͚πʔϧ౳ͷ։ൃɾϝϯς • ύϑΥʔϚϯενϡʔχϯά΍ϘτϧωοΫͷௐࠪɾվળ౳ɺ໨ͷલʹ͋Δٕज़త՝ ୊Λιϑτ΢ΣΞΤϯδχΞϦϯάͷྗʹΑͬͯൃݟɾղܾ͢Δͷ͕ϛογϣϯʢ͜ Ε͸SRE΋Ұॹʣ
  3. • े෼ͳੑೳ • ϐʔΫ࣌Ͱඵؒ਺ઍreq/secΛࡹ͍ͯΔ ͱ͋ΔGoͷHTTP APIαʔόͷ CPU࢖༻཰ͷάϥϑˠ • ϨεϙϯελΠϜ͸avgͰ4msɺ90~99percentileͰ10~20msલޙ •

    ඇಉظॲཧ΍ฒߦੑͷߴ͍ϓϩάϥϜͷ࣮ݱ͕Մೳ • ΰϧʔνϯ΍νϟωϧʹΑΔڧྗͳαϙʔτ ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨGo
  4. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package &

    upload to S3 JP US UK aws s3 sync yum repo yum repo yum repo
  5. slackboard • Slack Incoming WebhookϓϩΩγ݉ΫϥΠΞϯτ • github.com/cubicdaiya/slackboard • Slack΁ͷ௨஌ॲཧ΍ඞཁͳ৘ใΛҰݩ؅ཧԽ •

    ௨஌ઌ΍Webhook URLઃఆͷू໿ • ΫϥΠΞϯτϓϩάϥϜͷ౷ҰʹΑΔϝϯςφϏϦςΟͷ޲্ • ౰࣌(2014೥ࠒ)ࣾ಺ͰPHP΍Ruby΍PythonͷSlackΫϥΠΞϯτ͕ཚཱͯͨ͠
  6. slackboard configuration [core] # listen port port = “29800” #

    URL for Incoming Webhooks slack_url = “https://hooks.slack.com/services/…” [log] # access log path or redirector access_log = “stdout” # error log path or redirector error_log = “stderr” # log level level = “error”
  7. Usage of slackboard-cli $ slackboard -c /path/to/slackboard.toml Server Client $

    echo mercari | slackboard-cli -c test -s 127.0.0.1:29800
  8. Usage of slackboard-log $ slackboard -c /path/to/slackboard.toml Server Client $

    slackboard-log -c test -s 127.0.0.1:29800 — ls notfound
  9. ࣮ࡍͷγεςϜߏ੒ by slackboard asia region us region eu region eu

    region asia region us region Cloud Load Balancing GCP TMBDLCPBSE TMBDLCPBSE TMBDLCPBSE
  10. System architecture of pvpool • nginx • pvpoold • Powered

    by Go (net/http) • MySQL pvpoold pvpoold nginx MySQL pvpool.local by consul DNS : HTTP protocol : MySQL protocol The gopher was designed by Renée French
  11. pvpoold Internals POST /items/pvc { "item_ids" [ ... ] }

    PVCHandler Sharding Item IDs MySQL Slot Slot Slot Slot Slot Slot Pooling query Flushing periodically synchronous processing asynchronous processing
  12. pvpoold Internals • Slot಺ͷొ৔ਓ෺ • Slot -> chan types.ItemID •

    PVMap -> map[types.ItemID]int • Storer -> Slot͔Β঎඼IDΛϑΣονͯ͠PVMap಺ͷ஋Λߋ৽ • Flusher -> ఆظతʹPVMap಺ͷ஋ΛऔΓग़ͯ͠MySQLʹߋ৽ΫΤϦΛൃߦ • Storer ͱ Flusher ΋·ͨΰϧʔνϯ
  13. Slot overview Slot Storer Flusher PVMap dequeue item ID Store

    pv per item Fetch pv per item periodically MySQL Issue SQL to count up
  14. Storer func (slot *PVCSlot) storer() { for { itemID :=

    <- slot.Slot slot.Lock() slot.PVMap[itemID]++ slot.Unock() } }
  15. Flusher func (slot *PVCSlot) flusher(interval time.Duration) { ticker := time.NewTicker(interval)

    for { <- ticker.C slot.Lock() if err := slot.flush(); err != nil { // transaction // error handling } slot.Unlock() } }
  16. StorerͱFlusherͷ໾ׂ • Storer • MySQL΁ൃߦ͢ΔSQLΛ঎඼IDຖʹ·ͱΊΔ • eg. UPDATE pv =

    pv + 5 instead of five of UPDATE pv = pv +1 • Flusher • ҰఆִؒͰMySQLʹߋ৽༻SQLΛൃߦ͢Δ • ঎඼͕Ӿཡ͞ΕΔ౓ʹUPDATE͕૸ΔͷΛආ͚ͯෛՙΛ࿨Β͛Δ
  17. Getting Started - go-httpstats mw := stats.New() handler := mw.WrapHandleFunc(

    http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(“Hello, World”)) })) statsHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { d, err := mw.Data() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }) http.Handle(“/“, handler) http.Handle(“/stats”, statshandler) http.ListenAndServe(“:9999”, nil)
  18. Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq

    ‘.request’ { “count”: 13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } }
  19. Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq

    ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }
  20. Sampling factor const ( // DefaultBufferSize is buf size sampling

    response time for percentile and average DefaultBufferSize = 1000 // DefaultSamplingFactor is factor to sample response time for percentile and average DefaultSamplingFactor = 1 ) // … // mw := stats.New() // NewCapa returns a new statistics structure // buffer size of HTTP requests is allocated length of bufsize // If bufsize is less than 2, NewCapa returns error mw, err := stats.NewCapa(bufsize, factor)
  21. Introducing sampling factor • percentileͷܭࢉ͸ͦͦ͜͜ॏ͍ • avgͷܭࢉͱҧͬͯιʔτ͕ඞཁ • ϨεϙϯελΠϜܥ͸౷ܭσʔλͳͷͰඞͣ͠΋શ෦औΔඞཁ͸ͳ͍ •

    ϫʔΫϩʔυͷܹ͍͠ՕॴͩͱόοϑΝॻ͖ࠐΈͷMutexڝ߹͕ܹͯ͘͠ ໰୊ʹͳͬͨͷͰɺͦͷϫʔΫΞϥ΢ϯυͱͯ͠ಋೖ
  22. ࠷ऴతʹ͸MackerelͰάϥϑԽ $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’ { “count”:

    13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } } $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }
  23. mfc

  24. mfc • In-house Fastly CLI tool at Mercari • Early

    development phase • ϝϧΧϦͰ͸2017೥͔ΒFastlyΛར༻த • ػೳͷ໢ཏੑࣗମ͸ͦΜͳʹߴ͘ͳ͍ • ඞཁʹͳͬͨΓɺ͋Δͱศརͦ͏ͳػೳΛ౎౓ΠϯςάϨʔτͯ͠Δײ͡ • ΏΔ;Θʹ։ൃͯ͠·͢
  25. Usage of mfc • $ mfc Usage of mfc: config

    the utility for mfc configuration service the utility for fastly service acl the utility for fastly ACL (etc…) • ػೳྖҬຖʹαϒίϚϯυΛఆٛ • ACL, Service, Version౳ • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹ switch args[1] { case “config”: return config.NewCLI().Run(args) case “service”: return service.NewCLI().Run(args) case “acl”: return acl.NewCLI().Run(args) case “…” … } ಈ࡞Πϝʔδ
  26. mfcͷػೳ(Ұ෦) $ mfc config -h Usage of config: mfc config

    list output mfc configuration $ mfc service -h Usage of service: mfc service vcldiff from_version to_version output difference between from and to mfc service versions output active and latest versions $ mfc acl -h Usage acl: mfc acl create create acl mfc acl remove remove acl mfc acl add add given entry to acl mfc acl del delete given entry in acl mfc acl show output acl mfc acl list output acl entries mfc acl sync sync acl with given provider file
  27. imageflux-cli • In-house ImageFlux CLI tool for Mercari • github.com/mercari/imageflux-cli

    • ImageFlux ~ Ϋϥ΢υը૾ม׵αʔϏε • https://www.sakura.ad.jp/services/imageflux/ • Reference • ImageFluxΛར༻ͨ͠ը૾഑৴ͷ࠷దԽʙಈతϦαΠζͱWebPม׵ʙ • https://tech.mercari.com/entry/2018/01/30/161001
  28. Usage of imageflux-cli $ imageflux-cli -h Usage: imageflux-cli cache.lookup -k

    $url imageflux-cli cache.delete -k $url imageflux-cli signature -s $secret -p $path
  29. ΄͔ʹ΋͍Ζ͍Ζ • Gaurun - Building high performance push notification server

    in Go • https://speakerdeck.com/cubicdaiya/building-high-performance-push-notification-server-in- go • chocon - Simple proxy server for persisting connection between upstream servers • https://speakerdeck.com/kazeburo/cloud-connect-the-world-as-a-glue-aws-dev-day-2017? slide=53 • Mercari API Gateway • https://go-talks.appspot.com/github.com/tcnksm/talks/2018/07/mercarigo/microservices-api- gateway.slide#1 • etc… ( There are too many works and products in Go at Mercari)