Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Software Engineer, Infrastructure

Software Engineer, Infrastructure

Tatsuhiko Kubo

August 10, 2018
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Tatsuhiko Kubo@cubicdaiya
    mercari.go#2 2018/08/10
    Software Engineer, Infrastructure

    View Slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Principal Software Engineer, SRE @ Mercari, Inc.

    View Slide

  3. View Slide

  4. Agenda
    • Software Engineer, Infrastructure ϙδγϣϯͷ঺հ

    • ϝϧΧϦͷཪଆͰՔಇ͢ΔGoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ

    View Slide

  5. Software Engineer, Infrastructure
    https://careers.mercari.com/job/

    View Slide

  6. Software Engineer, Infrastructure
    https://careers.mercari.com/job/

    View Slide

  7. Software Engineer, Infrastructure
    • Role

    • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΔߴ͍ύϑΥʔϚϯε΍৴པੑͷߴ͍Πϯϑϥε
    τϥΫνϟͷ࣮ݱ

    • ΞϓϦέʔγϣϯ΍ΠϯϑϥपΓͰڞ௨͢Δٕज़త՝୊Λղܾ͢ΔͨΊͷπʔϧ΍ϛ
    υϧ΢ΣΞͷ։ൃɾӡ༻

    • Software Engineer, Site Reliability (SRE)ͱͷҧ͍

    • ਖ਼௚ʹݴ͏ͱɺͦ͜·Ͱ໌֬ʹ۠ผ͞ΕΔΘ͚Ͱ͸ͳ͍ʢॴଐ͸SREνʔϜʣ

    • ҰํͰɺSREΑΓ΋։ൃۀ຿ͷൺॏ͕େ͖͍ʢ7:3͔8:2͘Β͍ʣ

    View Slide

  8. Software Engineer, Infrastructure
    • ։ൃݴޠ͸Go͕த৺

    • ωοτϫʔΫ΍γεςϜϓϩάϥϛϯάدΓͷ஌ࣝ΍εΩϧ͕ཁٻ͞ΕΔ৔໘͕ଟΊ

    • ΠϯϑϥετϥΫνϟͷج൫΍Τοδέʔε޲͚ͷαʔόαΠυϛυϧ΢ΣΞͷ
    ։ൃɾӡ༻

    • ֤छΦϖϨʔγϣϯͷࣗಈԽ΍ϞχλϦϯά޲͚πʔϧ౳ͷ։ൃɾϝϯς

    • ύϑΥʔϚϯενϡʔχϯά΍ϘτϧωοΫͷௐࠪɾվળ౳ɺ໨ͷલʹ͋Δٕज़త՝
    ୊Λιϑτ΢ΣΞΤϯδχΞϦϯάͷྗʹΑͬͯൃݟɾղܾ͢Δͷ͕ϛογϣϯʢ͜
    Ε͸SRE΋Ұॹʣ

    View Slide

  9. ಥવͰ͕͢ɺϚΠΫϩαʔϏε͕ྲྀߦΓͰ͢Ͷ

    View Slide

  10. View Slide

  11. https://tech.mercari.com/entry/2018/07/11/180500

    View Slide

  12. https://tech.mercari.com/entry/2018/07/11/180500

    View Slide

  13. ࣮ࡍͷͱ͜Ζɺ
    • ϝϧΧϦͰGo͕ར༻͞Ε࢝Ίͨͷ͸2014೥͔Β

    • αʔϏεͷػೳ։ൃͰ͸౰͔࣌ΒPHP͕ϝΠϯͩͬͨ

    • ҰํɺཪଆͰՔಇ͢Δπʔϧ΍ϛυϧ΢ΣΞͳΜ͔͸
    ౰͔࣌ΒGo΍Node.js΋ͪΒ΄Β

    • ࢝·Γ͸ӡ༻޲͚πʔϧͷ։ൃ΍ύϑΥʔϚϯε͕ཁٻ͞ΕΔՕॴͰͷར༻

    • GKE+GoΛத৺ʹਾ͑ͯϚΠΫϩαʔϏεԽΛਪਐ <- ΠϚίί

    View Slide

  14. ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨPHP
    • PHP͸7ʹͳͬͯେ෼଎͘ͳͬͨ

    • CPU࢖༻཰͸൒෼ҎԼ

    • ॲཧ͕࣌ؒ਺े%ݮ

    • ͔͠͠ɺ୯ମੑೳ͕޲্͚ͨͩ͠Ͱ΢ΟʔΫϙΠϯτ͸ͦͷ··

    • γϯάϧεϨουͳͷͰϚϧνίΞΛ׆͔ͤͳ͍

    • ඇಉظॲཧ͕ۤख
    PHP5.6 -> PHP7.1
    Response time avg
    PHP5.6 -> PHP7.1

    View Slide

  15. • े෼ͳੑೳ

    • ϐʔΫ࣌Ͱඵؒ਺ઍreq/secΛࡹ͍ͯΔ
    ͱ͋ΔGoͷHTTP APIαʔόͷ
    CPU࢖༻཰ͷάϥϑˠ

    • ϨεϙϯελΠϜ͸avgͰ4msɺ90~99percentileͰ10~20msલޙ

    • ඇಉظॲཧ΍ฒߦੑͷߴ͍ϓϩάϥϜͷ࣮ݱ͕Մೳ

    • ΰϧʔνϯ΍νϟωϧʹΑΔڧྗͳαϙʔτ
    ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨGo

    View Slide

  16. • γϯάϧόΠφϦͳͷͰ഑෍ָ͕

    • γεςϜӡ༻ऀతʹָ͘͢͝

    • ύοέʔδϯά΋ָ
    πʔϧ։ൃݴޠͱͯ͠ͷGo

    View Slide

  17. • DockerͰͲ͔ͬʔΜ

    • RPM

    • πʔϧ΍ϛυϧ΢ΣΞྨ͸ΞϓϦέʔγϣϯͱൺ΂ͯͦΜͳʹසൟʹߋ
    ৽͠ͳ͍ͷͰ΄ͱΜͲͷ৔߹͸͜ΕͰे෼
    ϝϧΧϦͰͷ࣮ࡍͷ഑෍ํ๏

    View Slide

  18. DockerͱMakeʹΑΔRPMύοέʔδͷϏϧυγεςϜ
    https://tech.mercari.com/entry/2016/08/15/163219

    View Slide

  19. ෳ਺ͷOS޲͚ʹDockerΛىಈͯ͠Ϗϧυ

    View Slide

  20. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ
    Merge PR to chage spec Build package & upload to S3

    View Slide

  21. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ
    Merge PR to chage spec Build package & upload to S3
    JP US UK
    aws s3 sync
    yum repo yum repo yum repo

    View Slide

  22. ϝϧΧϦͷཪଆͰՔಇ͢Δ
    GoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ

    View Slide

  23. slackboard

    View Slide

  24. slackboard
    • Slack Incoming WebhookϓϩΩγ݉ΫϥΠΞϯτ

    • github.com/cubicdaiya/slackboard

    • Slack΁ͷ௨஌ॲཧ΍ඞཁͳ৘ใΛҰݩ؅ཧԽ

    • ௨஌ઌ΍Webhook URLઃఆͷू໿

    • ΫϥΠΞϯτϓϩάϥϜͷ౷ҰʹΑΔϝϯςφϏϦςΟͷ޲্

    • ౰࣌(2014೥ࠒ)ࣾ಺ͰPHP΍Ruby΍PythonͷSlackΫϥΠΞϯτ͕ཚཱͯͨ͠

    View Slide

  25. slackboard configuration
    [core]
    # listen port
    port = “29800”
    # URL for Incoming Webhooks
    slack_url = “https://hooks.slack.com/services/…”
    [log]
    # access log path or redirector
    access_log = “stdout”
    # error log path or redirector
    error_log = “stderr”
    # log level
    level = “error”

    View Slide

  26. Usage of slackboard-cli
    $ slackboard -c /path/to/slackboard.toml
    Server
    Client
    $ echo mercari | slackboard-cli -c test -s 127.0.0.1:29800

    View Slide

  27. Usage of slackboard-log
    $ slackboard -c /path/to/slackboard.toml
    Server
    Client
    $ slackboard-log -c test -s 127.0.0.1:29800 — ls notfound

    View Slide

  28. ࣮ࡍͷγεςϜߏ੒ by slackboard
    asia region
    us region
    eu region
    eu region
    asia region
    us region
    Cloud Load
    Balancing
    GCP
    TMBDLCPBSE
    TMBDLCPBSE
    TMBDLCPBSE

    View Slide

  29. ͳ͓ɺϝϧΧϦͰ࣮ࡍʹ࢖ΘΕ͍ͯΔΫϥΠΞϯτ͸Perl੡ͷslacklog
    $ slacklog -c test --log-channel test -- ls notfound

    View Slide

  30. pvpool

    View Slide

  31. pvpool
    • ϝϧΧϦͷ঎඼Ӿཡ਺ΛΧ΢ϯτΞοϓ͢ΔͨΊͷ࢓૊Έ

    • ࢦఆ͞Εͨ঎඼IDͷӾཡ਺ͷΧ΢ϯτΞοϓ͓Αͼ
    औಘͷͨΊͷAPIΛఏڙ

    • ※ ը໘͸։ൃ༻ͷ΋ͷͰ͢

    View Slide

  32. ঎඼Ӿཡ਺Χ΢ϯτΞοϓͷϫʔΫϩʔυͱٕज़తͳཁ݅
    • ঎඼Ӿཡʹؔ͢ΔAPI΁ͷϦΫΤετ਺͸ϝϧΧϦͷશAPIதTOP3ʹೖΔ

    • ϝϧΧϦAPI΁ͷશϦΫΤετ਺͸ϐʔΫ࣌Ͱ໿56,000req/sec

    • ੑೳཁ݅

    • ग़དྷΔݶΓ௿஗ԆͰԠ౴Ͱ͖ΔAPIΛఏڙ͢Δʢ໨҆ɿ਺ᶡඵҎ಺ʣ

    • ग़དྷΔݶΓϦΞϧλΠϜʹӾཡ਺Λ࠷৽ͷঢ়ଶʹ൓ө͢Δʢ໨҆ɿ਺ඵ
    Ҏ಺ʣ

    View Slide

  33. System architecture of pvpool
    • nginx

    • pvpoold

    • Powered by Go (net/http)

    • MySQL
    pvpoold
    pvpoold
    nginx MySQL
    pvpool.local
    by consul DNS
    : HTTP protocol
    : MySQL protocol
    The gopher was designed by Renée French

    View Slide

  34. ఏڙ͢ΔAPI͸2ͭ
    • POST /items/pvc
    • ༩͑ΒΕͨ঎඼IDຖͷPVΛΧ΢ϯτΞοϓ

    • POST /items/pv
    • ༩͑ΒΕͨ঎඼IDຖͷPVΛฦ͢

    • ঎඼ID͸ෳ਺ࢦఆՄೳ

    View Slide

  35. pvpoold Internals
    POST /items/pvc
    {
    "item_ids" [
    ...
    ]
    }
    PVCHandler
    Sharding
    Item IDs
    MySQL
    Slot
    Slot
    Slot
    Slot
    Slot
    Slot
    Pooling query
    Flushing
    periodically
    synchronous processing
    asynchronous processing

    View Slide

  36. pvpoold Internals
    • PVCHandler

    • HTTPϋϯυϥؔ਺

    • Slot

    • Χ΢ϯτΞοϓ༻ϓϩηεϢχοτ by ΰϧʔνϯ

    View Slide

  37. pvpoold Internals
    • Slot಺ͷొ৔ਓ෺

    • Slot -> chan types.ItemID

    • PVMap -> map[types.ItemID]int
    • Storer -> Slot͔Β঎඼IDΛϑΣονͯ͠PVMap಺ͷ஋Λߋ৽

    • Flusher -> ఆظతʹPVMap಺ͷ஋ΛऔΓग़ͯ͠MySQLʹߋ৽ΫΤϦΛൃߦ

    • Storer ͱ Flusher ΋·ͨΰϧʔνϯ

    View Slide

  38. Slot overview
    Slot
    Storer Flusher
    PVMap
    dequeue item ID
    Store pv per item Fetch pv per item
    periodically
    MySQL
    Issue SQL to count up

    View Slide

  39. Storer
    func (slot *PVCSlot) storer() {
    for {
    itemID := <- slot.Slot
    slot.Lock()
    slot.PVMap[itemID]++
    slot.Unock()
    }
    }

    View Slide

  40. Flusher
    func (slot *PVCSlot) flusher(interval time.Duration) {
    ticker := time.NewTicker(interval)
    for {
    <- ticker.C
    slot.Lock()
    if err := slot.flush(); err != nil { // transaction
    // error handling
    }
    slot.Unlock()
    }
    }

    View Slide

  41. StorerͱFlusherͷ໾ׂ
    • Storer
    • MySQL΁ൃߦ͢ΔSQLΛ঎඼IDຖʹ·ͱΊΔ

    • eg. UPDATE pv = pv + 5 instead of five of UPDATE pv = pv +1

    • Flusher
    • ҰఆִؒͰMySQLʹߋ৽༻SQLΛൃߦ͢Δ

    • ঎඼͕Ӿཡ͞ΕΔ౓ʹUPDATE͕૸ΔͷΛආ͚ͯෛՙΛ࿨Β͛Δ

    View Slide

  42. go-httpstats

    View Slide

  43. go-httpstats
    • ʢػೳ͕গͳ͍ͱ͍͏ҙຯͰʣখن໛ͳHTTPαʔό޲͚ͷϝτϦΫεऔಘϥΠϒϥϦ

    • ϦΫΤετ਺ʢτʔλϧͱεςʔλείʔυຖʣ

    • ϨεϙϯελΠϜʢmaxɺminɺavgɺpercentileຖʣ

    • Not published

    • ։ൃͨ͠ͷ͸ࠓ೥৽ଔೖࣾͷSoftware Engineer, Infrastructure

    • ͢Ͱʹ͋Δ΋ͷΛ୳͚ͨ͠Ͳɺͪΐ͏Ͳ͍͍΋ͷ͕ͳ͔ͬͨͷͰࣗ࡞

    View Slide

  44. Getting Started - go-httpstats
    mw := stats.New()
    handler := mw.WrapHandleFunc(
    http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte(“Hello, World”))
    }))
    statsHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    d, err := mw.Data()
    if err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
    }
    })
    http.Handle(“/“, handler)
    http.Handle(“/stats”, statshandler)
    http.ListenAndServe(“:9999”, nil)

    View Slide

  45. Getting Started - go-httpstats
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’
    {
    “count”: 13821031,
    “status_count”: {
    “200”: 13820000,
    “400”: 31,
    “401”: 0,
    “403”: 0,
    “404”: 1000,
    “500”: 0,

    }
    }

    View Slide

  46. Getting Started - go-httpstats
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’
    {
    “max_time”: 0.0238210,
    “min_time”: 0.00015,
    “average_time”: 0.0035,
    “percentiled_time”: {
    “90”: 0.0109,
    “95”: 0.0130,
    “99”: 0.0215
    }
    }

    View Slide

  47. Sampling factor
    const (
    // DefaultBufferSize is buf size sampling response time for percentile and average
    DefaultBufferSize = 1000
    // DefaultSamplingFactor is factor to sample response time for percentile and
    average
    DefaultSamplingFactor = 1
    )
    // …
    // mw := stats.New()
    // NewCapa returns a new statistics structure
    // buffer size of HTTP requests is allocated length of bufsize
    // If bufsize is less than 2, NewCapa returns error
    mw, err := stats.NewCapa(bufsize, factor)

    View Slide

  48. Introducing sampling factor
    • percentileͷܭࢉ͸ͦͦ͜͜ॏ͍

    • avgͷܭࢉͱҧͬͯιʔτ͕ඞཁ

    • ϨεϙϯελΠϜܥ͸౷ܭσʔλͳͷͰඞͣ͠΋શ෦औΔඞཁ͸ͳ͍

    • ϫʔΫϩʔυͷܹ͍͠ՕॴͩͱόοϑΝॻ͖ࠐΈͷMutexڝ߹͕ܹͯ͘͠
    ໰୊ʹͳͬͨͷͰɺͦͷϫʔΫΞϥ΢ϯυͱͯ͠ಋೖ

    View Slide

  49. ࠷ऴతʹ͸MackerelͰάϥϑԽ
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’
    {
    “count”: 13821031,
    “status_count”: {
    “200”: 13820000,
    “400”: 31,
    “401”: 0,
    “403”: 0,
    “404”: 1000,
    “500”: 0,

    }
    }
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’
    {
    “max_time”: 0.0238210,
    “min_time”: 0.00015,
    “average_time”: 0.0035,
    “percentiled_time”: {
    “90”: 0.0109,
    “95”: 0.0130,
    “99”: 0.0215
    }
    }

    View Slide

  50. mackerelϓϥάΠϯ΋GoͰ։ൃ
    • MackerelϓϥάΠϯ։ൃ༻ϔϧύʔΛ࢖͏ͱָ

    • github.com/mackerelip/go-mackerel-plugin

    • specϑΝΠϧ࡞੒ޙɺmake mackerel-plugin-nameͰRPMύοέʔδΛϏ
    ϧυ & ഑෍

    • ฐࣾͷಠࣗMackerelϓϥάΠϯ͸Go & Perl੡

    • https://speakerdeck.com/kazeburo/mackerel-day

    View Slide

  51. mfc

    View Slide

  52. mfc
    • In-house Fastly CLI tool at Mercari

    • Early development phase

    • ϝϧΧϦͰ͸2017೥͔ΒFastlyΛར༻த

    • ػೳͷ໢ཏੑࣗମ͸ͦΜͳʹߴ͘ͳ͍

    • ඞཁʹͳͬͨΓɺ͋Δͱศརͦ͏ͳػೳΛ౎౓ΠϯςάϨʔτͯ͠Δײ͡

    • ΏΔ;Θʹ։ൃͯ͠·͢

    View Slide

  53. mfc configuration
    $ cat ~/.fastly/conf.ini
    service = xxx
    apikey = yyy

    View Slide

  54. Usage of mfc

    $ mfc
    Usage of mfc:
    config
    the utility for mfc configuration
    service
    the utility for fastly service
    acl
    the utility for fastly ACL
    (etc…)
    • ػೳྖҬຖʹαϒίϚϯυΛఆٛ

    • ACL, Service, Version౳

    • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹

    switch args[1] {
    case “config”:
    return config.NewCLI().Run(args)
    case “service”:
    return service.NewCLI().Run(args)
    case “acl”:
    return acl.NewCLI().Run(args)
    case “…”

    }
    ಈ࡞Πϝʔδ

    View Slide

  55. mfcͷػೳ(Ұ෦)
    $ mfc config -h
    Usage of config:
    mfc config list
    output mfc configuration
    $ mfc service -h
    Usage of service:
    mfc service vcldiff from_version to_version
    output difference between from and to
    mfc service versions
    output active and latest versions
    $ mfc acl -h
    Usage acl:
    mfc acl create
    create acl
    mfc acl remove
    remove acl
    mfc acl add
    add given entry to acl
    mfc acl del
    delete given entry in acl
    mfc acl show
    output acl
    mfc acl list
    output acl entries
    mfc acl sync
    sync acl with given provider file

    View Slide

  56. imageflux-cli

    View Slide

  57. imageflux-cli
    • In-house ImageFlux CLI tool for Mercari

    • github.com/mercari/imageflux-cli

    • ImageFlux ~ Ϋϥ΢υը૾ม׵αʔϏε

    • https://www.sakura.ad.jp/services/imageflux/

    • Reference

    • ImageFluxΛར༻ͨ͠ը૾഑৴ͷ࠷దԽʙಈతϦαΠζͱWebPม׵ʙ

    • https://tech.mercari.com/entry/2018/01/30/161001

    View Slide

  58. imageflux-cli configuration
    $ cat ~/.imageflux/conf.ini
    token = xxx

    View Slide

  59. Usage of imageflux-cli
    $ imageflux-cli -h
    Usage:
    imageflux-cli cache.lookup -k $url
    imageflux-cli cache.delete -k $url
    imageflux-cli signature -s $secret -p $path

    View Slide

  60. ΄͔ʹ΋͍Ζ͍Ζ
    • Gaurun - Building high performance push notification server in Go

    • https://speakerdeck.com/cubicdaiya/building-high-performance-push-notification-server-in-
    go

    • chocon - Simple proxy server for persisting connection between upstream servers

    • https://speakerdeck.com/kazeburo/cloud-connect-the-world-as-a-glue-aws-dev-day-2017?
    slide=53

    • Mercari API Gateway

    • https://go-talks.appspot.com/github.com/tcnksm/talks/2018/07/mercarigo/microservices-api-
    gateway.slide#1

    • etc… ( There are too many works and products in Go at Mercari)

    View Slide

  61. ·ͱΊ
    • Software Engineer, Infrastructureͷ঺հ

    • ओʹGoͰαʔόαΠυϛυϧ΢ΣΞ΍ΒCLIπʔϧͷ։ൃΛ΍ͬͯ·͢

    • ݱࡏ3ਓ(͏ͪ1ਓ͸SREͱ݉຿)

    View Slide

  62. We are hiring Go Software Engineer!
    https://careers.mercari.com/job/

    View Slide