実例で学ぶGoプログラム開発

Transcript

1. ࣮ྫͰֶͿGoϓϩάϥϜ։ൃ
   Tatsuhiko Kubo@cubicdaiya
   Gunosy.go#11@2015/03/11
   

   View Slide

2. ࣗݾ঺հ
   • ٱอୡ඙(Tatsuhiko Kubo)
   • bokko@cubicdaiya
   • Software Engineer in Infrastructure Engineering
   • Mercari, Inc.
   • Go / C / mruby / nginx / ngx_(lua|mruby) / h2o /etc…
   

   View Slide

3. ࣮ྫͰֶͿGoϓϩάϥϜ։ൃ
   

   View Slide

4. ࠓճͷςʔϚ
   • ͜Ε·Ͱʹ։ൃͨ͠GoͷϓϩάϥϜΛྫʹ
   • ԿނGoͰ։ൃͨ͠ͷ͔ʁ
   • ࣮ࡍͷGoͷίʔυ
   • ֤GoϓϩάϥϜΛӡ༻ͯ͠Έͨײ૝
   • ͳͲʹ͍ͭͯ࿩͠·͢
   

   View Slide

5. ࠓճ঺հ͢ΔGoϓϩάϥϜ
   • nginx-build
   • cachectl
   • slackboard
   • શ෦ϝϧΧϦͰ΋ར༻͍ͯ͠·͢ˢ
   • in-house program on ϝϧΧϦ
   

   View Slide

6. nginx-build
   

   View Slide

7. nginx-build
   • nginxͷϏϧυπʔϧ
   • nginxͷϏϧυʹ·ͭΘΔ໘౗ࣄΛࣗಈԽ
   • ιʔείʔυ΍ґଘϥΠϒϥϦͷμ΢ϯϩʔυ
   • 3rd PartyϞδϡʔϧͷμ΢ϯϩʔυɺ૊ΈࠐΈ
   • https://github.com/cubicdaiya/nginx-build
   

   View Slide

8. Why is nginx-build written in Go?
   • ࠷ॳ͸shͰॻ͍ͯͨ
   • ੲ͍ͨձࣾʹ͸શࣗಈԽsh͕͋ͬͨ
   • స৬ͯ͠࢖͑ͳ͘ͳͬͨ
   • ·ͨshॻ͘ͷͭΒ͍͠ɺϝϯς΋ͨ͘͠ͳ͍
   • ͦ͏ͩɺGoͰॻ͜͏ʂ
   

   View Slide

9. Why is nginx-build written in Go?
   • ϙʔλϒϧ
   • ϝϯςָ͕
   • ഑෍΋ָ
   • ΫϩείϯύΠϧ͕؆୯
   • ฒྻॲཧ͕؆୯
   

   View Slide

10. nginx-buildͷ࣮ߦ෩ܠ
    $ nginx-build -d work -pcre -zlib -openssl
    nginx-build: 0.3.1
    Compiler: gc go1.4.1
    2015/03/10 02:04:23 Download pcre-8.36.....
    2015/03/10 02:04:23 Download zlib-1.2.8.....
    2015/03/10 02:04:23 Download openssl-1.0.2.....
    2015/03/10 02:04:23 Download nginx-1.7.10.....
    2015/03/10 02:04:28 Extract zlib-1.2.8.tar.gz.....
    2015/03/10 02:04:28 Extract nginx-1.7.10.tar.gz.....
    2015/03/10 02:04:35 Extract openssl-1.0.2.tar.gz.....
    2015/03/10 02:04:39 Extract pcre-8.36.tar.gz.....
    2015/03/10 02:04:39 Generate configure script for nginx-1.7.10.....
    2015/03/10 02:04:39 Configure nginx-1.7.10.....
    2015/03/10 02:04:45 Build nginx-1.7.10.....
    2015/03/10 02:07:34 Complete building nginx!
    ɾ
    ɾ
    ɾ
    $
    ฒྻͰ࣮ߦ͞Ε·͢
    

    View Slide

11. ͜Μͳײ͡
    parallels := 0
    done := make(chan bool)
    (தུ)
    if *pcreStatic {
    parallels++
    go downloadAndExtractParallel(&pcreBuilder, done)
    }
    if *openSSLStatic {
    parallels++
    go downloadAndExtractParallel(&openSSLBuilder, done)
    }
    if *zlibStatic {
    parallels++
    go downloadAndExtractParallel(&zlibBuilder, done)
    }
    parallels++
    go downloadAndExtractParallel(&nginxBuilder, done)
    (தུ)
    for i := 0; i < parallels; i++ {
    <-done
    }
    

    View Slide

12. shΑΓͣͬͱָͰ͢
    

    View Slide

13. cachectl
    

    View Slide

14. cachectl
    • OSͷϖʔδΩϟογϡ֬ೝɾ࡟আπʔϧ
    • ແବͳϖʔδΩϟογϡΛղ์͢Δͷʹར༻
    • cachectld͍ͬͯ͏σʔϞϯ΋෇͍ͯΔ
    • https://github.com/cubicdaiya/cachectl
    

    View Slide

15. Why is cachectl written in Go?
    • طଘͷπʔϧ͕࢖͍ʹ͔ͬͨ͘
    • e.g. σΟϨΫτϦࢦఆ͕Ͱ͖ͳ͍
    • σʔϞϯԽͯ͠ఆظతʹ࣮ߦ͔͚ͨͬͨ͠ͲɺͦΜͳػ
    ೳ࣋ͬͯΔπʔϧ͕ͳ͔ͬͨ
    • γεςϜίʔϧ͕cgo΍syscallͰ؆୯ʹݺͼग़ͤΔ
    • ϖʔδΩϟογϡͷ৘ใऔಘ΍ղ์ʹ͸mmap΍
    mincoreɺposix_fadvise౳ͷγεςϜίʔϧ͕ඞཁ
    

    View Slide

16. cachectlͰϖʔδΩϟογϡͷ
    ར༻۩߹ΛνΣοΫ
    $ cachectl -f /var/log/nginx/access.log
    2015/03/10 03:08:37 /var/log/nginx/
    access.log 's pages in cache:
    1323464/1323464 (100.0%)
    [filesize=5293852.1K, pagesize=4K]
    $
    5GBͷϩάϑΝΠϧͷத਎͕
    શ෦ϖʔδΩϟογϡʹࡌͬͯΔ
    

    View Slide

17. ফͯ͠ΈΔ
    $ # ϖʔδΩϟογϡΛ࡟আ
    $ cachectl -f /var/log/nginx/access.log -op purge
    2015/03/10 03:13:03 purging /var/log/nginx/
    access.log 's page cache
    $
    $ # ࠶౓ϖʔδΩϟογϡͷར༻۩߹ΛνΣοΫ
    $ cachectl -f /var/log/nginx/access.log
    2015/03/10 03:13:07 /var/log/nginx/access.log 's
    pages in cache: 94/1325181 (0.0%)
    [filesize=5300722.2K, pagesize=4K]
    $
    5GBͷϝϞϦ͕ղ์͞Ε·ͨ͠
    

    View Slide

18. ࢦఆͨ͠ϑΝΠϧͷϖʔδΩϟογϡΛղ์͢ΔCͷؔ਺
    /*
    #include
    #include
    #include
    #include
    int fadvise(const char *path, float r)
    {
    int fd;
    struct stat st;
    off_t l;
    fd = open(path, O_RDONLY);
    if(fd == -1) {
    return -1;
    }
    if(fstat(fd, &st) == -1) {
    goto error;
    }
    l = (off_t)(st.st_size * r);
    if(posix_fadvise(fd, 0, l, POSIX_FADV_DONTNEED) != 0) {
    goto error;
    }
    close(fd);
    return 1;
    error:
    close(fd);
    return -1;
    }
    */
    import "C"
    

    View Slide

19. Go͔Βͷݺͼग़͠
    func purgePages(fpath string, fsize int64, rate float64) error {
    if rate < 0.0 || rate > 1.0 {
    return errors.New(fmt.Sprintf("%f: rate should be less than 1.0\n", rate))
    }
    cs := C.CString(fpath)
    defer C.free(unsafe.Pointer(cs))
    result := C.fadvise(cs, C.float(rate))
    if result == -1 {
    return errors.New(fmt.Sprintf("failed to purge page cache for %s", fpath))
    }
    return nil
    }
    C.(ؔ਺໊)Ͱݺͼग़ͤΔ
    

    View Slide

20. cgoΛར༻͢Δࡍͷ஫ҙ఺
    • CͳͷͰϝϞϦ؅ཧ͸ࣗલ
    • e.g. C.CStringͷ஋͸ޙͰC.free͢Δ
    • cgoͷίʔυ࣮ߦத͸goroutineͷίϯςΩετε
    Πον͕ޮ͔ͳ͍
    • ίϯύΠϧ͕஗͘ͳΔ
    • ϙʔλϏϦςΟ͕Լ͕Δ
    

    View Slide

21. slackboard
    

    View Slide

22. slackboard
    • SlackϓϩΩγ
    • clientίϚϯυ(slackboard-cli)෇ଐ
    • https://github.com/cubicdaiya/slackboard
    

    View Slide

23. Why is slackboard written in Go?
    • net/httpͰ؆୯ʹHTTPαʔό/ΫϥΠΞϯτ͕ॻ
    ͚Δ
    • slackboard-cli͸શαʔόʹ഑ஔ͍ͨ͠
    • GoͳΒ഑෍ָ͕
    • LLΈ͍ͨʹ֤αʔόʹ࣮ߦ؀ڥ༻ҙ͠ͳ͍͍ͯ͘
    

    View Slide

24. ࣮ߦΠϝʔδ
    

    View Slide

25. in-house programs on ϝϧΧϦ
    

    View Slide

26. Ωϟογϡੜ੒σʔϞϯ
    • ಛఆAPIͷϨεϙϯεͷΩϟογϡΛͻͨ͢Β࡞Γଓ
    ͚ΔσʔϞϯ
    • ΞΫηε਺͕ଟ͍ͷͰϢʔβ͕ΞΫηε͢Δલʹ
    ΩϟογϡΛ࡞੒͓͍ͯͯ͠ߴ଎Խ
    • 1ඵ͝ͱʹฒྻʹ݁ߏͳྔͷHTTPϦΫΤετΛૹΓଓ͚Δ
    • ݩʑRubyͰॻ͔Ε͍ͯͨ(࣮ߦαʔό͕ॏ͔ͬͨ)
    

    View Slide

27. ݩͷRubyίʔυ
    Parallel.map(params_list, :in_processes => in_processes) do |params|
    (தུ)
    res = http_request('GET', url, params)
    (தུ)
    end.compact
    params_listͷαΠζ͕େ͖͍
    ↓
    େྔʹfork͢ΔͷͰॏ͍
    

    View Slide

28. GoͰॻ͖௚ͨ͠
    done := make(chan error)
    size := len(paramCombs.Params)
    var errors []error
    for i := 0; i < size; i++ {
    go makeCache(URL, paramCombs.Params[i], done)
    }
    for i := 0; i < size; i++ {
    err := <-done
    if err != nil {
    errors = append(errors, err)
    }
    }
    gorutine͸͍ܰʂ
    

    View Slide

29. ࣮ߦαʔόͷάϥϑ(CPU)
    ϦϥΠτͨ͠λΠϛϯά
    

    View Slide

30. ࣮ߦαʔόͷάϥϑ(LA)
    ϦϥΠτͨ͠λΠϛϯά
    ேԜΜͰΔ෦෼͸ϝϯςφϯε
    

    View Slide

31. ͦͷଞ
    • σʔϞϯԽ͢Δͱ͖͸supervisordΛར༻
    • go-server-starter͔circusࢼ͍ͨ͠
    • ύοέʔδ؅ཧ͸gom࢖ͬͯ·͢
    • ϏϧυλεΫ͸Makefileʹॻ͍ͯ·͢
    

    View Slide

32. ࣌ؒ଍Γͳͯ͘࡟ͬͨ΍ͭ
    • ϓογϡ௨஌αʔό in Go
    • https://speakerdeck.com/cubicdaiya/1000mo-
    daunrodoapuri-merukari-wozhi-eruji-shu?
    slide=57
    • GoαʔόͷϞχλϦϯά
    • http://qiita.com/cubicdaiya/items/
    b90c1ae147a72f4970b2
    

    View Slide

33. Ҏ্ͷϓϩάϥϜΛ
    GoͰॻ͍ͯӡ༻ͯ͠Έͨײ૝
    • ૣ͍ɾ଎͍ɾ͍ܰ
    • ੜ࢈ੑߴ͍(શ෦1ʙ2࣌ؒͰ࠷ॳͷόʔδϣϯΛॻ͚ͨ)
    • े෼ʹύϑΥʔϚϯε͕Α͍
    • ϑοτϓϦϯτ΋Α͍
    • ඇৗʹ҆ఆ͍ͯ͠Δ
    • ։ൃऀʹ΋ӡ༻ऀʹ΋΍͍͞͠
    

    View Slide