Software Engineer, Infrastructure

Transcript

1. Tatsuhiko Kubo@cubicdaiya
   mercari.go#2 2018/08/10
   Software Engineer, Infrastructure
   

   View Slide

2. @cubicdaiya / Tatsuhiko Kubo
   Principal Software Engineer, SRE @ Mercari, Inc.
   

   View Slide

3. View Slide

4. Agenda
   • Software Engineer, Infrastructure ϙδγϣϯͷ঺հ
   
   • ϝϧΧϦͷཪଆͰՔಇ͢ΔGoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ
   

   View Slide

5. Software Engineer, Infrastructure
   https://careers.mercari.com/job/
   

   View Slide

6. Software Engineer, Infrastructure
   https://careers.mercari.com/job/
   

   View Slide

7. Software Engineer, Infrastructure
   • Role
   
   • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΔߴ͍ύϑΥʔϚϯε΍৴པੑͷߴ͍Πϯϑϥε
   τϥΫνϟͷ࣮ݱ
   
   • ΞϓϦέʔγϣϯ΍ΠϯϑϥपΓͰڞ௨͢Δٕज़త՝୊Λղܾ͢ΔͨΊͷπʔϧ΍ϛ
   υϧ΢ΣΞͷ։ൃɾӡ༻
   
   • Software Engineer, Site Reliability (SRE)ͱͷҧ͍
   
   • ਖ਼௚ʹݴ͏ͱɺͦ͜·Ͱ໌֬ʹ۠ผ͞ΕΔΘ͚Ͱ͸ͳ͍ʢॴଐ͸SREνʔϜʣ
   
   • ҰํͰɺSREΑΓ΋։ൃۀ຿ͷൺॏ͕େ͖͍ʢ7:3͔8:2͘Β͍ʣ
   

   View Slide

8. Software Engineer, Infrastructure
   • ։ൃݴޠ͸Go͕த৺
   
   • ωοτϫʔΫ΍γεςϜϓϩάϥϛϯάدΓͷ஌ࣝ΍εΩϧ͕ཁٻ͞ΕΔ৔໘͕ଟΊ
   
   • ΠϯϑϥετϥΫνϟͷج൫΍Τοδέʔε޲͚ͷαʔόαΠυϛυϧ΢ΣΞͷ
   ։ൃɾӡ༻
   
   • ֤छΦϖϨʔγϣϯͷࣗಈԽ΍ϞχλϦϯά޲͚πʔϧ౳ͷ։ൃɾϝϯς
   
   • ύϑΥʔϚϯενϡʔχϯά΍ϘτϧωοΫͷௐࠪɾվળ౳ɺ໨ͷલʹ͋Δٕज़త՝
   ୊Λιϑτ΢ΣΞΤϯδχΞϦϯάͷྗʹΑͬͯൃݟɾղܾ͢Δͷ͕ϛογϣϯʢ͜
   Ε͸SRE΋Ұॹʣ
   

   View Slide

9. ಥવͰ͕͢ɺϚΠΫϩαʔϏε͕ྲྀߦΓͰ͢Ͷ
   

   View Slide

10. View Slide

11. https://tech.mercari.com/entry/2018/07/11/180500
    

    View Slide

12. https://tech.mercari.com/entry/2018/07/11/180500
    

    View Slide

13. ࣮ࡍͷͱ͜Ζɺ
    • ϝϧΧϦͰGo͕ར༻͞Ε࢝Ίͨͷ͸2014೥͔Β
    
    • αʔϏεͷػೳ։ൃͰ͸౰͔࣌ΒPHP͕ϝΠϯͩͬͨ
    
    • ҰํɺཪଆͰՔಇ͢Δπʔϧ΍ϛυϧ΢ΣΞͳΜ͔͸
    ౰͔࣌ΒGo΍Node.js΋ͪΒ΄Β
    
    • ࢝·Γ͸ӡ༻޲͚πʔϧͷ։ൃ΍ύϑΥʔϚϯε͕ཁٻ͞ΕΔՕॴͰͷར༻
    
    • GKE+GoΛத৺ʹਾ͑ͯϚΠΫϩαʔϏεԽΛਪਐ <- ΠϚίί
    

    View Slide

14. ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨPHP
    • PHP͸7ʹͳͬͯେ෼଎͘ͳͬͨ
    
    • CPU࢖༻཰͸൒෼ҎԼ
    
    • ॲཧ͕࣌ؒ਺े%ݮ
    
    • ͔͠͠ɺ୯ମੑೳ͕޲্͚ͨͩ͠Ͱ΢ΟʔΫϙΠϯτ͸ͦͷ··
    
    • γϯάϧεϨουͳͷͰϚϧνίΞΛ׆͔ͤͳ͍
    
    • ඇಉظॲཧ͕ۤख
    PHP5.6 -> PHP7.1
    Response time avg
    PHP5.6 -> PHP7.1
    

    View Slide

15. • े෼ͳੑೳ
    
    • ϐʔΫ࣌Ͱඵؒ਺ઍreq/secΛࡹ͍ͯΔ
    ͱ͋ΔGoͷHTTP APIαʔόͷ
    CPU࢖༻཰ͷάϥϑˠ
    
    • ϨεϙϯελΠϜ͸avgͰ4msɺ90~99percentileͰ10~20msલޙ
    
    • ඇಉظॲཧ΍ฒߦੑͷߴ͍ϓϩάϥϜͷ࣮ݱ͕Մೳ
    
    • ΰϧʔνϯ΍νϟωϧʹΑΔڧྗͳαϙʔτ
    ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨGo
    

    View Slide

16. • γϯάϧόΠφϦͳͷͰ഑෍ָ͕
    
    • γεςϜӡ༻ऀతʹָ͘͢͝
    
    • ύοέʔδϯά΋ָ
    πʔϧ։ൃݴޠͱͯ͠ͷGo
    

    View Slide

17. • DockerͰͲ͔ͬʔΜ
    
    • RPM
    
    • πʔϧ΍ϛυϧ΢ΣΞྨ͸ΞϓϦέʔγϣϯͱൺ΂ͯͦΜͳʹසൟʹߋ
    ৽͠ͳ͍ͷͰ΄ͱΜͲͷ৔߹͸͜ΕͰे෼
    ϝϧΧϦͰͷ࣮ࡍͷ഑෍ํ๏
    

    View Slide

18. DockerͱMakeʹΑΔRPMύοέʔδͷϏϧυγεςϜ
    https://tech.mercari.com/entry/2016/08/15/163219
    

    View Slide

19. ෳ਺ͷOS޲͚ʹDockerΛىಈͯ͠Ϗϧυ
    

    View Slide

20. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ
    Merge PR to chage spec Build package & upload to S3
    

    View Slide

21. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ
    Merge PR to chage spec Build package & upload to S3
    JP US UK
    aws s3 sync
    yum repo yum repo yum repo
    

    View Slide

22. ϝϧΧϦͷཪଆͰՔಇ͢Δ
    GoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ
    

    View Slide

23. slackboard
    

    View Slide

24. slackboard
    • Slack Incoming WebhookϓϩΩγ݉ΫϥΠΞϯτ
    
    • github.com/cubicdaiya/slackboard
    
    • Slack΁ͷ௨஌ॲཧ΍ඞཁͳ৘ใΛҰݩ؅ཧԽ
    
    • ௨஌ઌ΍Webhook URLઃఆͷू໿
    
    • ΫϥΠΞϯτϓϩάϥϜͷ౷ҰʹΑΔϝϯςφϏϦςΟͷ޲্
    
    • ౰࣌(2014೥ࠒ)ࣾ಺ͰPHP΍Ruby΍PythonͷSlackΫϥΠΞϯτ͕ཚཱͯͨ͠
    

    View Slide

25. slackboard configuration
    [core]
    # listen port
    port = “29800”
    # URL for Incoming Webhooks
    slack_url = “https://hooks.slack.com/services/…”
    [log]
    # access log path or redirector
    access_log = “stdout”
    # error log path or redirector
    error_log = “stderr”
    # log level
    level = “error”
    

    View Slide

26. Usage of slackboard-cli
    $ slackboard -c /path/to/slackboard.toml
    Server
    Client
    $ echo mercari | slackboard-cli -c test -s 127.0.0.1:29800
    

    View Slide

27. Usage of slackboard-log
    $ slackboard -c /path/to/slackboard.toml
    Server
    Client
    $ slackboard-log -c test -s 127.0.0.1:29800 — ls notfound
    

    View Slide

28. ࣮ࡍͷγεςϜߏ੒ by slackboard
    asia region
    us region
    eu region
    eu region
    asia region
    us region
    Cloud Load
    Balancing
    GCP
    TMBDLCPBSE
    TMBDLCPBSE
    TMBDLCPBSE
    

    View Slide

29. ͳ͓ɺϝϧΧϦͰ࣮ࡍʹ࢖ΘΕ͍ͯΔΫϥΠΞϯτ͸Perl੡ͷslacklog
    $ slacklog -c test --log-channel test -- ls notfound
    

    View Slide

30. pvpool
    

    View Slide

31. pvpool
    • ϝϧΧϦͷ঎඼Ӿཡ਺ΛΧ΢ϯτΞοϓ͢ΔͨΊͷ࢓૊Έ
    
    • ࢦఆ͞Εͨ঎඼IDͷӾཡ਺ͷΧ΢ϯτΞοϓ͓Αͼ
    औಘͷͨΊͷAPIΛఏڙ
    
    • ※ ը໘͸։ൃ༻ͷ΋ͷͰ͢
    

    View Slide

32. ঎඼Ӿཡ਺Χ΢ϯτΞοϓͷϫʔΫϩʔυͱٕज़తͳཁ݅
    • ঎඼Ӿཡʹؔ͢ΔAPI΁ͷϦΫΤετ਺͸ϝϧΧϦͷશAPIதTOP3ʹೖΔ
    
    • ϝϧΧϦAPI΁ͷશϦΫΤετ਺͸ϐʔΫ࣌Ͱ໿56,000req/sec
    
    • ੑೳཁ݅
    
    • ग़དྷΔݶΓ௿஗ԆͰԠ౴Ͱ͖ΔAPIΛఏڙ͢Δʢ໨҆ɿ਺ᶡඵҎ಺ʣ
    
    • ग़དྷΔݶΓϦΞϧλΠϜʹӾཡ਺Λ࠷৽ͷঢ়ଶʹ൓ө͢Δʢ໨҆ɿ਺ඵ
    Ҏ಺ʣ
    

    View Slide

33. System architecture of pvpool
    • nginx
    
    • pvpoold
    
    • Powered by Go (net/http)
    
    • MySQL
    pvpoold
    pvpoold
    nginx MySQL
    pvpool.local
    by consul DNS
    : HTTP protocol
    : MySQL protocol
    The gopher was designed by Renée French
    

    View Slide

34. ఏڙ͢ΔAPI͸2ͭ
    • POST /items/pvc
    • ༩͑ΒΕͨ঎඼IDຖͷPVΛΧ΢ϯτΞοϓ
    
    • POST /items/pv
    • ༩͑ΒΕͨ঎඼IDຖͷPVΛฦ͢
    
    • ঎඼ID͸ෳ਺ࢦఆՄೳ
    

    View Slide

35. pvpoold Internals
    POST /items/pvc
    {
    "item_ids" [
    ...
    ]
    }
    PVCHandler
    Sharding
    Item IDs
    MySQL
    Slot
    Slot
    Slot
    Slot
    Slot
    Slot
    Pooling query
    Flushing
    periodically
    synchronous processing
    asynchronous processing
    

    View Slide

36. pvpoold Internals
    • PVCHandler
    
    • HTTPϋϯυϥؔ਺
    
    • Slot
    
    • Χ΢ϯτΞοϓ༻ϓϩηεϢχοτ by ΰϧʔνϯ
    

    View Slide

37. pvpoold Internals
    • Slot಺ͷొ৔ਓ෺
    
    • Slot -> chan types.ItemID
    
    • PVMap -> map[types.ItemID]int
    • Storer -> Slot͔Β঎඼IDΛϑΣονͯ͠PVMap಺ͷ஋Λߋ৽
    
    • Flusher -> ఆظతʹPVMap಺ͷ஋ΛऔΓग़ͯ͠MySQLʹߋ৽ΫΤϦΛൃߦ
    
    • Storer ͱ Flusher ΋·ͨΰϧʔνϯ
    

    View Slide

38. Slot overview
    Slot
    Storer Flusher
    PVMap
    dequeue item ID
    Store pv per item Fetch pv per item
    periodically
    MySQL
    Issue SQL to count up
    

    View Slide

39. Storer
    func (slot *PVCSlot) storer() {
    for {
    itemID := <- slot.Slot
    slot.Lock()
    slot.PVMap[itemID]++
    slot.Unock()
    }
    }
    

    View Slide

40. Flusher
    func (slot *PVCSlot) flusher(interval time.Duration) {
    ticker := time.NewTicker(interval)
    for {
    <- ticker.C
    slot.Lock()
    if err := slot.flush(); err != nil { // transaction
    // error handling
    }
    slot.Unlock()
    }
    }
    

    View Slide

41. StorerͱFlusherͷ໾ׂ
    • Storer
    • MySQL΁ൃߦ͢ΔSQLΛ঎඼IDຖʹ·ͱΊΔ
    
    • eg. UPDATE pv = pv + 5 instead of five of UPDATE pv = pv +1
    
    • Flusher
    • ҰఆִؒͰMySQLʹߋ৽༻SQLΛൃߦ͢Δ
    
    • ঎඼͕Ӿཡ͞ΕΔ౓ʹUPDATE͕૸ΔͷΛආ͚ͯෛՙΛ࿨Β͛Δ
    

    View Slide

42. go-httpstats
    

    View Slide

43. go-httpstats
    • ʢػೳ͕গͳ͍ͱ͍͏ҙຯͰʣখن໛ͳHTTPαʔό޲͚ͷϝτϦΫεऔಘϥΠϒϥϦ
    
    • ϦΫΤετ਺ʢτʔλϧͱεςʔλείʔυຖʣ
    
    • ϨεϙϯελΠϜʢmaxɺminɺavgɺpercentileຖʣ
    
    • Not published
    
    • ։ൃͨ͠ͷ͸ࠓ೥৽ଔೖࣾͷSoftware Engineer, Infrastructure
    
    • ͢Ͱʹ͋Δ΋ͷΛ୳͚ͨ͠Ͳɺͪΐ͏Ͳ͍͍΋ͷ͕ͳ͔ͬͨͷͰࣗ࡞
    

    View Slide

44. Getting Started - go-httpstats
    mw := stats.New()
    handler := mw.WrapHandleFunc(
    http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte(“Hello, World”))
    }))
    statsHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    d, err := mw.Data()
    if err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
    }
    })
    http.Handle(“/“, handler)
    http.Handle(“/stats”, statshandler)
    http.ListenAndServe(“:9999”, nil)
    

    View Slide

45. Getting Started - go-httpstats
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’
    {
    “count”: 13821031,
    “status_count”: {
    “200”: 13820000,
    “400”: 31,
    “401”: 0,
    “403”: 0,
    “404”: 1000,
    “500”: 0,
    …
    }
    }
    

    View Slide

46. Getting Started - go-httpstats
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’
    {
    “max_time”: 0.0238210,
    “min_time”: 0.00015,
    “average_time”: 0.0035,
    “percentiled_time”: {
    “90”: 0.0109,
    “95”: 0.0130,
    “99”: 0.0215
    }
    }
    

    View Slide

47. Sampling factor
    const (
    // DefaultBufferSize is buf size sampling response time for percentile and average
    DefaultBufferSize = 1000
    // DefaultSamplingFactor is factor to sample response time for percentile and
    average
    DefaultSamplingFactor = 1
    )
    // …
    // mw := stats.New()
    // NewCapa returns a new statistics structure
    // buffer size of HTTP requests is allocated length of bufsize
    // If bufsize is less than 2, NewCapa returns error
    mw, err := stats.NewCapa(bufsize, factor)
    

    View Slide

48. Introducing sampling factor
    • percentileͷܭࢉ͸ͦͦ͜͜ॏ͍
    
    • avgͷܭࢉͱҧͬͯιʔτ͕ඞཁ
    
    • ϨεϙϯελΠϜܥ͸౷ܭσʔλͳͷͰඞͣ͠΋શ෦औΔඞཁ͸ͳ͍
    
    • ϫʔΫϩʔυͷܹ͍͠ՕॴͩͱόοϑΝॻ͖ࠐΈͷMutexڝ߹͕ܹͯ͘͠
    ໰୊ʹͳͬͨͷͰɺͦͷϫʔΫΞϥ΢ϯυͱͯ͠ಋೖ
    

    View Slide

49. ࠷ऴతʹ͸MackerelͰάϥϑԽ
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’
    {
    “count”: 13821031,
    “status_count”: {
    “200”: 13820000,
    “400”: 31,
    “401”: 0,
    “403”: 0,
    “404”: 1000,
    “500”: 0,
    …
    }
    }
    $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’
    {
    “max_time”: 0.0238210,
    “min_time”: 0.00015,
    “average_time”: 0.0035,
    “percentiled_time”: {
    “90”: 0.0109,
    “95”: 0.0130,
    “99”: 0.0215
    }
    }
    

    View Slide

50. mackerelϓϥάΠϯ΋GoͰ։ൃ
    • MackerelϓϥάΠϯ։ൃ༻ϔϧύʔΛ࢖͏ͱָ
    
    • github.com/mackerelip/go-mackerel-plugin
    
    • specϑΝΠϧ࡞੒ޙɺmake mackerel-plugin-nameͰRPMύοέʔδΛϏ
    ϧυ & ഑෍
    
    • ฐࣾͷಠࣗMackerelϓϥάΠϯ͸Go & Perl੡
    
    • https://speakerdeck.com/kazeburo/mackerel-day
    

    View Slide

51. mfc
    

    View Slide

52. mfc
    • In-house Fastly CLI tool at Mercari
    
    • Early development phase
    
    • ϝϧΧϦͰ͸2017೥͔ΒFastlyΛར༻த
    
    • ػೳͷ໢ཏੑࣗମ͸ͦΜͳʹߴ͘ͳ͍
    
    • ඞཁʹͳͬͨΓɺ͋Δͱศརͦ͏ͳػೳΛ౎౓ΠϯςάϨʔτͯ͠Δײ͡
    
    • ΏΔ;Θʹ։ൃͯ͠·͢
    

    View Slide

53. mfc configuration
    $ cat ~/.fastly/conf.ini
    service = xxx
    apikey = yyy
    …
    

    View Slide

54. Usage of mfc
    •
    $ mfc
    Usage of mfc:
    config
    the utility for mfc configuration
    service
    the utility for fastly service
    acl
    the utility for fastly ACL
    (etc…)
    • ػೳྖҬຖʹαϒίϚϯυΛఆٛ
    
    • ACL, Service, Version౳
    
    • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹
    
    switch args[1] {
    case “config”:
    return config.NewCLI().Run(args)
    case “service”:
    return service.NewCLI().Run(args)
    case “acl”:
    return acl.NewCLI().Run(args)
    case “…”
    …
    }
    ಈ࡞Πϝʔδ
    

    View Slide

55. mfcͷػೳ(Ұ෦)
    $ mfc config -h
    Usage of config:
    mfc config list
    output mfc configuration
    $ mfc service -h
    Usage of service:
    mfc service vcldiff from_version to_version
    output difference between from and to
    mfc service versions
    output active and latest versions
    $ mfc acl -h
    Usage acl:
    mfc acl create
    create acl
    mfc acl remove
    remove acl
    mfc acl add
    add given entry to acl
    mfc acl del
    delete given entry in acl
    mfc acl show
    output acl
    mfc acl list
    output acl entries
    mfc acl sync
    sync acl with given provider file
    

    View Slide

56. imageflux-cli
    

    View Slide

57. imageflux-cli
    • In-house ImageFlux CLI tool for Mercari
    
    • github.com/mercari/imageflux-cli
    
    • ImageFlux ~ Ϋϥ΢υը૾ม׵αʔϏε
    
    • https://www.sakura.ad.jp/services/imageflux/
    
    • Reference
    
    • ImageFluxΛར༻ͨ͠ը૾഑৴ͷ࠷దԽʙಈతϦαΠζͱWebPม׵ʙ
    
    • https://tech.mercari.com/entry/2018/01/30/161001
    

    View Slide

58. imageflux-cli configuration
    $ cat ~/.imageflux/conf.ini
    token = xxx
    

    View Slide

59. Usage of imageflux-cli
    $ imageflux-cli -h
    Usage:
    imageflux-cli cache.lookup -k $url
    imageflux-cli cache.delete -k $url
    imageflux-cli signature -s $secret -p $path
    

    View Slide

60. ΄͔ʹ΋͍Ζ͍Ζ
    • Gaurun - Building high performance push notification server in Go
    
    • https://speakerdeck.com/cubicdaiya/building-high-performance-push-notification-server-in-
    go
    
    • chocon - Simple proxy server for persisting connection between upstream servers
    
    • https://speakerdeck.com/kazeburo/cloud-connect-the-world-as-a-glue-aws-dev-day-2017?
    slide=53
    
    • Mercari API Gateway
    
    • https://go-talks.appspot.com/github.com/tcnksm/talks/2018/07/mercarigo/microservices-api-
    gateway.slide#1
    
    • etc… ( There are too many works and products in Go at Mercari)
    

    View Slide

61. ·ͱΊ
    • Software Engineer, Infrastructureͷ঺հ
    
    • ओʹGoͰαʔόαΠυϛυϧ΢ΣΞ΍ΒCLIπʔϧͷ։ൃΛ΍ͬͯ·͢
    
    • ݱࡏ3ਓ(͏ͪ1ਓ͸SREͱ݉຿)
    

    View Slide

62. We are hiring Go Software Engineer!
    https://careers.mercari.com/job/
    

    View Slide