࣮ྫͰֶͿGoϓϩάϥϜ։ൃ Tatsuhiko Kubo@cubicdaiya Gunosy.go#11@2015/03/11

ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in Infrastructure Engineering • Mercari, Inc. • Go / C / mruby / nginx / ngx_(lua|mruby) / h2o /etc…

࣮ྫͰֶͿGoϓϩάϥϜ։ൃ

ࠓճͷςʔϚ • ͜Ε·Ͱʹ։ൃͨ͠GoͷϓϩάϥϜΛྫʹ • ԿނGoͰ։ൃͨ͠ͷ͔ʁ • ࣮ࡍͷGoͷίʔυ • ֤GoϓϩάϥϜΛӡ༻ͯ͠Έͨײ૝ • ͳͲʹ͍ͭͯ࿩͠·͢

ࠓճ঺հ͢ΔGoϓϩάϥϜ • nginx-build • cachectl • slackboard • શ෦ϝϧΧϦͰ΋ར༻͍ͯ͠·͢ˢ • in-house program on ϝϧΧϦ

nginx-build

nginx-build • nginxͷϏϧυπʔϧ • nginxͷϏϧυʹ·ͭΘΔ໘౗ࣄΛࣗಈԽ • ιʔείʔυ΍ґଘϥΠϒϥϦͷμ΢ϯϩʔυ • 3rd PartyϞδϡʔϧͷμ΢ϯϩʔυɺ૊ΈࠐΈ • https://github.com/cubicdaiya/nginx-build

Why is nginx-build written in Go? • ࠷ॳ͸shͰॻ͍ͯͨ • ੲ͍ͨձࣾʹ͸શࣗಈԽsh͕͋ͬͨ • స৬ͯ͠࢖͑ͳ͘ͳͬͨ • ·ͨshॻ͘ͷͭΒ͍͠ɺϝϯς΋ͨ͘͠ͳ͍ • ͦ͏ͩɺGoͰॻ͜͏ʂ

Why is nginx-build written in Go? • ϙʔλϒϧ • ϝϯςָ͕ • ഑෍΋ָ • ΫϩείϯύΠϧ͕؆୯ • ฒྻॲཧ͕؆୯

nginx-buildͷ࣮ߦ෩ܠ $ nginx-build -d work -pcre -zlib -openssl nginx-build: 0.3.1 Compiler: gc go1.4.1 2015/03/10 02:04:23 Download pcre-8.36..... 2015/03/10 02:04:23 Download zlib-1.2.8..... 2015/03/10 02:04:23 Download openssl-1.0.2..... 2015/03/10 02:04:23 Download nginx-1.7.10..... 2015/03/10 02:04:28 Extract zlib-1.2.8.tar.gz..... 2015/03/10 02:04:28 Extract nginx-1.7.10.tar.gz..... 2015/03/10 02:04:35 Extract openssl-1.0.2.tar.gz..... 2015/03/10 02:04:39 Extract pcre-8.36.tar.gz..... 2015/03/10 02:04:39 Generate configure script for nginx-1.7.10..... 2015/03/10 02:04:39 Configure nginx-1.7.10..... 2015/03/10 02:04:45 Build nginx-1.7.10..... 2015/03/10 02:07:34 Complete building nginx! ɾ ɾ ɾ $ ฒྻͰ࣮ߦ͞Ε·͢

͜Μͳײ͡ parallels := 0 done := make(chan bool) (தུ) if *pcreStatic { parallels++ go downloadAndExtractParallel(&pcreBuilder, done) } if *openSSLStatic { parallels++ go downloadAndExtractParallel(&openSSLBuilder, done) } if *zlibStatic { parallels++ go downloadAndExtractParallel(&zlibBuilder, done) } parallels++ go downloadAndExtractParallel(&nginxBuilder, done) (தུ) for i := 0; i < parallels; i++ { <-done }

shΑΓͣͬͱָͰ͢

cachectl

cachectl • OSͷϖʔδΩϟογϡ֬ೝɾ࡟আπʔϧ • ແବͳϖʔδΩϟογϡΛղ์͢Δͷʹར༻ • cachectld͍ͬͯ͏σʔϞϯ΋෇͍ͯΔ • https://github.com/cubicdaiya/cachectl

Why is cachectl written in Go? • طଘͷπʔϧ͕࢖͍ʹ͔ͬͨ͘ • e.g. σΟϨΫτϦࢦఆ͕Ͱ͖ͳ͍ • σʔϞϯԽͯ͠ఆظతʹ࣮ߦ͔͚ͨͬͨ͠ͲɺͦΜͳػ ೳ࣋ͬͯΔπʔϧ͕ͳ͔ͬͨ • γεςϜίʔϧ͕cgo΍syscallͰ؆୯ʹݺͼग़ͤΔ • ϖʔδΩϟογϡͷ৘ใऔಘ΍ղ์ʹ͸mmap΍ mincoreɺposix_fadvise౳ͷγεςϜίʔϧ͕ඞཁ

cachectlͰϖʔδΩϟογϡͷ ར༻۩߹ΛνΣοΫ $ cachectl -f /var/log/nginx/access.log 2015/03/10 03:08:37 /var/log/nginx/ access.log 's pages in cache: 1323464/1323464 (100.0%) [filesize=5293852.1K, pagesize=4K] $ 5GBͷϩάϑΝΠϧͷத਎͕ શ෦ϖʔδΩϟογϡʹࡌͬͯΔ

ফͯ͠ΈΔ $ # ϖʔδΩϟογϡΛ࡟আ $ cachectl -f /var/log/nginx/access.log -op purge 2015/03/10 03:13:03 purging /var/log/nginx/ access.log 's page cache $ $ # ࠶౓ϖʔδΩϟογϡͷར༻۩߹ΛνΣοΫ $ cachectl -f /var/log/nginx/access.log 2015/03/10 03:13:07 /var/log/nginx/access.log 's pages in cache: 94/1325181 (0.0%) [filesize=5300722.2K, pagesize=4K] $ 5GBͷϝϞϦ͕ղ์͞Ε·ͨ͠

ࢦఆͨ͠ϑΝΠϧͷϖʔδΩϟογϡΛղ์͢ΔCͷؔ਺ /* #include #include #include #include int fadvise(const char *path, float r) { int fd; struct stat st; off_t l; fd = open(path, O_RDONLY); if(fd == -1) { return -1; } if(fstat(fd, &st) == -1) { goto error; } l = (off_t)(st.st_size * r); if(posix_fadvise(fd, 0, l, POSIX_FADV_DONTNEED) != 0) { goto error; } close(fd); return 1; error: close(fd); return -1; } */ import "C"

Go͔Βͷݺͼग़͠ func purgePages(fpath string, fsize int64, rate float64) error { if rate < 0.0 || rate > 1.0 { return errors.New(fmt.Sprintf("%f: rate should be less than 1.0\n", rate)) } cs := C.CString(fpath) defer C.free(unsafe.Pointer(cs)) result := C.fadvise(cs, C.float(rate)) if result == -1 { return errors.New(fmt.Sprintf("failed to purge page cache for %s", fpath)) } return nil } C.(ؔ਺໊)Ͱݺͼग़ͤΔ

cgoΛར༻͢Δࡍͷ஫ҙ఺ • CͳͷͰϝϞϦ؅ཧ͸ࣗલ • e.g. C.CStringͷ஋͸ޙͰC.free͢Δ • cgoͷίʔυ࣮ߦத͸goroutineͷίϯςΩετε Πον͕ޮ͔ͳ͍ • ίϯύΠϧ͕஗͘ͳΔ • ϙʔλϏϦςΟ͕Լ͕Δ

slackboard

slackboard • SlackϓϩΩγ • clientίϚϯυ(slackboard-cli)෇ଐ • https://github.com/cubicdaiya/slackboard

Why is slackboard written in Go? • net/httpͰ؆୯ʹHTTPαʔό/ΫϥΠΞϯτ͕ॻ ͚Δ • slackboard-cli͸શαʔόʹ഑ஔ͍ͨ͠ • GoͳΒ഑෍ָ͕ • LLΈ͍ͨʹ֤αʔόʹ࣮ߦ؀ڥ༻ҙ͠ͳ͍͍ͯ͘

࣮ߦΠϝʔδ

in-house programs on ϝϧΧϦ

Ωϟογϡੜ੒σʔϞϯ • ಛఆAPIͷϨεϙϯεͷΩϟογϡΛͻͨ͢Β࡞Γଓ ͚ΔσʔϞϯ • ΞΫηε਺͕ଟ͍ͷͰϢʔβ͕ΞΫηε͢Δલʹ ΩϟογϡΛ࡞੒͓͍ͯͯ͠ߴ଎Խ • 1ඵ͝ͱʹฒྻʹ݁ߏͳྔͷHTTPϦΫΤετΛૹΓଓ͚Δ • ݩʑRubyͰॻ͔Ε͍ͯͨ(࣮ߦαʔό͕ॏ͔ͬͨ)

ݩͷRubyίʔυ Parallel.map(params_list, :in_processes => in_processes) do |params| (தུ) res = http_request('GET', url, params) (தུ) end.compact params_listͷαΠζ͕େ͖͍ ↓ େྔʹfork͢ΔͷͰॏ͍

GoͰॻ͖௚ͨ͠ done := make(chan error) size := len(paramCombs.Params) var errors []error for i := 0; i < size; i++ { go makeCache(URL, paramCombs.Params[i], done) } for i := 0; i < size; i++ { err := <-done if err != nil { errors = append(errors, err) } } gorutine͸͍ܰʂ

࣮ߦαʔόͷάϥϑ(CPU) ϦϥΠτͨ͠λΠϛϯά

࣮ߦαʔόͷάϥϑ(LA) ϦϥΠτͨ͠λΠϛϯά ேԜΜͰΔ෦෼͸ϝϯςφϯε

ͦͷଞ • σʔϞϯԽ͢Δͱ͖͸supervisordΛར༻ • go-server-starter͔circusࢼ͍ͨ͠ • ύοέʔδ؅ཧ͸gom࢖ͬͯ·͢ • ϏϧυλεΫ͸Makefileʹॻ͍ͯ·͢

࣌ؒ଍Γͳͯ͘࡟ͬͨ΍ͭ • ϓογϡ௨஌αʔό in Go • https://speakerdeck.com/cubicdaiya/1000mo- daunrodoapuri-merukari-wozhi-eruji-shu? slide=57 • GoαʔόͷϞχλϦϯά • http://qiita.com/cubicdaiya/items/ b90c1ae147a72f4970b2

Ҏ্ͷϓϩάϥϜΛ GoͰॻ͍ͯӡ༻ͯ͠Έͨײ૝ • ૣ͍ɾ଎͍ɾ͍ܰ • ੜ࢈ੑߴ͍(શ෦1ʙ2࣌ؒͰ࠷ॳͷόʔδϣϯΛॻ͚ͨ) • े෼ʹύϑΥʔϚϯε͕Α͍ • ϑοτϓϦϯτ΋Α͍ • ඇৗʹ҆ఆ͍ͯ͠Δ • ։ൃऀʹ΋ӡ༻ऀʹ΋΍͍͞͠