Stretcher

Shibuya.go#2

Transcript

1. Stretcher
   Shibuya.go #2
   2016-03-22 @fujiwara
   

   View Slide

2. @fujiwara
   github.com/fujiwara
   sfujiwara.hatenablog.com
   ٕज़෦
   

   View Slide

3. What's Stretcher
   github.com/fujiwara/stretcher
   Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ
   

   View Slide

4. Inspired by
   github.com/sorah/mamiya
   &
   AWS CodeDeploy
   

   View Slide

5. ઃܭํ਑
   AWS͡Όͳͯ͘΋ಈ͘
   tar & rsync & ίϚϯυ࣮ߦ
   Consul / Serf ͱ͸ૄ݁߹
   GoͰॻ͘
   Ұͭͷ͜ͱΛ͏·͘΍Δ
   

   View Slide

6. Architecture
   

   View Slide

7. ಋೖ࣮੷
   ʙ100୆
   ιʔγϟϧήʔϜɺࣗࣾαʔϏεʙ10਺୆(਺λΠτϧ)
   ʙ100୆
   ੒௕ΛՃ଎͢Δ minne ͷٕज़ج൫ઓུ
   www.slideshare.net/hsbt/minne-54244702
   

   View Slide

8. Stretcherͷ΍Δ͜ͱ(1)
   1. Manifest URLΛඪ४ೖྗ͔ΒಡΉ
   JSON(Consul Event) ·ͨ͸ text 1ߦ(SerfͳͲ)
   2. ManifestΛऔಘ
   s3 or http(s) or file scheme
   3. Manifest(YAML)ʹैͬͯDeployॲཧ
   download & extract & rsync & command
   

   View Slide

9. Manifest?
   src: s3://example.com/app.tar.gz
   checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93
   dest: /home/stretcher/app
   commands:
   pre:
   - /path/to/notify_start
   post:
   - /path/to/restart_app
   success:
   - /path/to/notify_ok
   failure:
   - /path/to/notify_ng
   excludes:
   - "*.pid"
   - "*.socket"
   

   View Slide

10. Stretcherͷ΍Δ͜ͱ(2)
    1. Manifestʹهड़͞Εͨ src(tar)Λऔಘ
    2. pre ίϚϯυΛ࣮ߦ
    3. tarΛςϯϙϥϦσΟϨΫτϦʹల։
    4. dest σΟϨΫτϦʹ rsync
    5. post ίϚϯυΛ࣮ߦ
    6. success or failure ίϚϯυΛ࣮ߦ
    

    View Slide

11. Stretcherͷ΍Βͳ͍͜ͱ
    build, tar, S3΁ͷΞοϓϩʔυ, consul eventൃߦͳͲ͸؅׋֎
    github.com/pepabo/capistrano-stretcher
    Rundeckͱ࢖͏ྫ
    kikumoto.hatenablog.com/entry/2015/09/16/190036
    

    View Slide

12. Imprementation
    

    View Slide

13. Imprementation
    ͓͜ͱΘΓ: εϥΠυͷίʔυ͸ΤϥʔॲཧͷলུଟΊͰ͢
    

    View Slide

14. ن໛ײ
    ຊମ700ߦɺςετؚΊͯ1200ߦఔ౓ͷখ͞͞
    71 cmd/stretcher/main.go
    120 aws.go 66 aws_test.go
    115 command.go 101 command_test.go
    31 consul.go 62 consul_test.go
    212 manifest.go 245 manifest_test.go
    171 stretcher.go
    -------------------------------------------------
    720 total 474 total
    

    View Slide

15. go-latestͰ࠷৽όʔδϣϯ൑ผ
    $ stretcher -v
    version: v0.4.1
    build: 2016-02-15T18:36:48+0900
    0.4.1 is not latest, you should upgrade to 0.4.2
    

    View Slide

16. go-latestͰ࠷৽όʔδϣϯ൑ผ
    github.com/tcnksm/go-latest
    func checkLatest(version string) {
    version = fixVersionStr(version)
    githubTag := &latest.GithubTag{
    Owner: "fujiwara",
    Repository: "stretcher",
    FixVersionStrFunc: fixVersionStr,
    }
    res, err := latest.Check(githubTag, version)
    if err != nil {
    fmt.Println(err)
    return
    }
    if res.Outdated {
    fmt.Printf("%s is not latest, you should upgrade to %s\n", version, res.Current)
    }
    }
    

    View Slide

17. go-latestͰ࠷৽όʔδϣϯ൑ผ
    • GithubTag - GitHub্ͷλάͱൺֱ
    • JSON - JSON APIͷϨεϙϯεͱൺֱ
    {
    "version":"1.2.3",
    "message":"Fix crash bug when XXX"
    }
    • HTMLMeta - URLͰఏڙ͞ΕΔHTMLͷϝλλάͱൺֱ
    
    

    View Slide

18. versionͷ஋Λbuild࣌ʹຒΊࠐΉ
    Makefile
    GIT_VER := $(shell git describe --tags)
    packages:
    gox (ུ) -ldflags "-X main.version ${GIT_VER}"
    Gitͷtag͕ main.version ͷ஋ͱͯ͠ઃఆ͞ΕΔ
    

    View Slide

19. math/rand ͷSeedΛ͍͍ײ͡ʹॳظԽ͢Δ
    ίϚϯυϥΠϯΦϓγϣϯ -random-sleep Ͱద౰ʹSleep
    ىಈλΠϛϯάΛࢄΒ͢
    import crand "crypto/rand"
    import mrand "math/rand"
    // http://stackoverflow.com/questions/6181260/generating-random-numbers-in-go
    var s int64
    if err := binary.Read(crand.Reader, binary.LittleEndian, &s); err != nil {
    s = time.Now().UnixNano()
    }
    mrand.Seed(s)
    

    View Slide

20. ඪ४ೖྗͷParse
    • ConsulͰ͸JSON(഑ྻܗࣜ)͕౉͞ΕΔ
    • SerfͰ͸վߦऴ୺ςΩετ͕౉͞ΕΔ
    ࣗ෼Ͱશ෦ಡ·ͳ͍Ͱɺ͍͍ײ͡ʹ൑ผ͍ͨ͠
    

    View Slide

21. ඪ४ೖྗ͔ΒJSON ·ͨ͸ 1ߦΛಡΉ
    bufio.Reader.Peek() Ͱઌ಄Λ೷͍ͯಡΈ෼͚
    func parseEvents() (string, error) {
    reader := bufio.NewReader(os.Stdin)
    b, _ := reader.Peek(1)
    if string(b) == "[" {
    ev, err := ParseConsulEvents(reader)
    if err != nil {
    return "", err
    }
    return string(ev.Payload), nil
    } else {
    line, _, err := reader.ReadLine()
    return string(line), err
    }
    }
    

    View Slide

22. Consul event JSON
    Payloadʹ͸Base64͞Εͨ஋͕ೖ͍ͬͯΔ
    [
    {
    "ID": "1d6731a8-833c-1aff-94e5-aa5e5a77da9f"
    "Name": "deploy",
    "Payload": "czM6Ly9leGFtcGxlLmNvbS9wYXRoL3RvL3Rhci5neg==",
    "NodeFilter": "",
    "ServiceFilter": "",
    "TagFilter": "",
    "Version": 1,
    "LTime": 2,
    },
    ...
    ]
    

    View Slide

23. Consul event JSON ΛಡΉ
    type ConsulEvents []ConsulEvent
    type ConsulEvent struct {
    ID string `json:"ID"`
    Name string `json:"Name"`
    Payload []byte `json:"Payload"`
    LTime int `json:"LTime"`
    }
    []byte Ͱఆ͓ٛͯ͘͠ͱBase64Λdecodeͯ͘͠ΕΔ
    

    View Slide

24. S3, HTTP(S), File͔Βಉ͡Α͏ʹಡΉ
    Manifestͷ src ʹ͸4छྨͷURL scheme͕࢖༻Մೳ
    • s3://
    • http://
    • https://
    • file://
    io.ReadCloser interfaceΛѻ͏Α͏ʹઃܭͯ͠ந৅Խ
    

    View Slide

25. S3, HTTP(S), File͔Βಉ͡Α͏ʹಡΉ
    Manifestͷ src ʹ͸4छྨͷURL scheme͕࢖༻Մೳ
    • s3://
    • http://
    • https://
    • file://
    io.ReadCloser interfaceΛѻ͏Α͏ʹઃܭͯ͠ந৅Խ
    

    View Slide

26. getURL() - S3, HTTP(S), File͔ΒಡΉ
    stretcher.getURL() ͸ io.ReadCloser, error Λฦ͢
    func getURL(urlStr string) (io.ReadCloser, error) {
    u, _ := url.Parse(urlStr)
    switch u.Scheme {
    case "s3":
    return getS3(u)
    case "http", "https":
    return getHTTP(u)
    case "file":
    return getFile(u)
    default:
    return nil, fmt.Errorf("manifest URL scheme must be s3 or http(s) or file: %s", urlStr)
    }
    }
    

    View Slide

27. getS3() - S3͔Βऔಘ͢Δ
    github.com/AdRoll/goamz
    func getS3(u *url.URL) (io.ReadCloser, error) {
    // credentialͷಡΈ͜Έ(ུ)
    client := s3.New(AWSAuth, AWSRegion)
    bucket := client.Bucket(u.Host)
    return bucket.GetReader(u.Path)
    }
    // goamz/s3
    func (b *Bucket) GetReader(path string) (rc io.ReadCloser, err error) {
    

    View Slide

28. getHTTP() - HTTP(S)͔Βऔಘ͢Δ
    func getHTTP(u *url.URL) (io.ReadCloser, error) {
    req, _ := http.NewRequest("GET", u.String(), nil)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
    return nil, err
    }
    return resp.Body, nil
    }
    

    View Slide

29. getFile() - File͔Βऔಘ͢Δ
    func getFile(u *url.URL) (io.ReadCloser, error) {
    file, err := os.Open(u.Path)
    if err != nil {
    return nil, err
    }
    return file, nil
    }
    

    View Slide

30. srcΛಡΈࠐΜͰϑΝΠϧʹอଘ
    อଘ͠ͳ͕Β͍ͭͰʹchecksumܭࢉ
    func (m *Manifest) Deploy() error {
    src, _ := getURL(m.Src)
    defer src.Close()
    tmp, _ := ioutil.TempFile(os.TempDir(), "stretcher")
    defer os.Remove(tmp.Name())
    written, sum, err := m.copyAndCalcHash(tmp, src)
    

    View Slide

31. ϑΝΠϧΛอଘ͠ͳ͕Βchecksumܭࢉ
    Goຊମͷ io.CopyBuffer() + ϋογϡ஋ܭࢉ
    func (m *Manifest) copyAndCalcHash(dst io.Writer, src io.Reader) (int64, string, error) {
    h, _ := m.newHash() // ͨͱ͑͹ sha256.New()
    buf := make([]byte, 32*1024)
    for {
    nr, er := src.Read(buf) // bufʹಡΈࠐΜͰ
    if nr > 0 {
    h.Write(buf[0:nr]) // ϋογϡ஋Λܭࢉ
    nw, ew := dst.Write(buf[0:nr]) // dstʹ΋ॻ͘
    if nw > 0 {
    written += int64(nw)
    }
    

    View Slide

32. if ew != nil {
    err = ew
    break
    }
    if nr != nw {
    err = io.ErrShortWrite
    break
    }
    }
    if er == io.EOF {
    break
    }
    if er != nil {
    err = er
    break
    }
    }
    s := fmt.Sprintf("%x", h.Sum(nil))
    return written, s, err
    }
    

    View Slide

33. औಘ଎౓Λ͍͍ײ͡ʹϩάʹग़͢
    Wrote 218,761,853 bytes to /tmp/stretcher900495202 (in 4.187495 sec, 52MB/s)
    github.com/dustin/go-humanize
    begin := time.Now()
    // URLऔಘͱϑΝΠϧॻ͖ࠐΈ
    duration := float64(time.Now().Sub(begin).Nanoseconds()) / 1000000000
    log.Printf("Wrote %s bytes to %s (in %s sec, %s/s)",
    humanize.Comma(written),
    tmp.Name(),
    humanize.Ftoa(duration),
    humanize.Bytes(uint64(float64(written)/duration)),
    )
    

    View Slide

34. tar & rsyncͷ࣮ߦ
    os/exec.Cmd.CombinedOutput()
    ֎෦ίϚϯυ࣮ߦɺඪ४ग़ྗɺඪ४Τϥʔग़ྗΛ·ͱΊͯฦ͢
    out, err := exec.Command("tar", "xf", tmp.Name()).CombinedOutput()
    

    View Slide

35. ϢʔβఆٛίϚϯυ(pre,post)ͷ࣮ߦ
    sh -c ʹ౉࣮ͯ͠ߦ
    out, err := exec.Command("sh", "-c", c.String()).CombinedOutput()
    pipe, && || ͳͲ͕ॻ͚ΔͷͰศར
    ࣮ࡍͷྫ
    post:
    - /sbin/service zabbix-agent status || sleep 30 && sudo /sbin/service zabbix-agent start
    

    View Slide

36. ϢʔβఆٛίϚϯυ(success, failure)ͷ࣮ߦ
    stretcherͷ࣮ߦϩά͕ίϚϯυͷstdinʹ౉͞ΕΔ࢓༷
    package stretcher
    var LogBuffer bytes.Buffer // άϩʔόϧ
    func Init() {
    log.SetOutput(io.MultiWriter(os.Stderr, &LogBuffer))
    ඪ४logϞδϡʔϧͷग़ྗઌΛ os.Stderr ͱ LogBuffer ʹ
    ඪ४Τϥʔग़ྗ͠ͳ͕ΒίϚϯυʹ౉͢όοϑΝʹ΋ཷΊΔ
    

    View Slide

37. ϩάΛstdinʹ౉ͯ͠ίϚϯυ࣮ߦ
    buf := bytes.NewBuffer(src.Bytes())
    c.InvokePipe(buf)
    func (c CommandLine) InvokePipe(src io.Reader) error {
    cmd := exec.Command("sh", "-c", c.String())
    stdin, _ := cmd.StdinPipe()
    stdout, _ := cmd.StdoutPipe()
    stderr, _ := cmd.StderrPipe()
    cmd.Start()
    

    View Slide

38. ֎෦ίϚϯυͷstdin΁ϩάΛྲྀ͠ࠐΉ
    stdinΛಡ·ͳ͍৔߹͸EPIPE͕ൃੜ͢ΔͷͰແࢹ
    var wg sync.WaitGroup
    wg.Add(3)
    // src => cmd.stdin
    go func() {
    _, err := io.Copy(stdin, src)
    if e, ok := err.(*os.PathError); ok && e.Err == syscall.EPIPE {
    // ignore EPIPE
    } else if err != nil {
    log.Println("failed to write to STDIN of", c.String(), ":", err)
    }
    stdin.Close()
    wg.Done()
    }()
    

    View Slide

39. ֎෦ίϚϯυͷstdout,stderrΛಡΈࠐΉ
    sync.WaitGroupͰ଴ͪ߹Θͤͳ͍ͱ຤ඌ͕੾ΕΔ(͜ͱ͕͋Δ)
    // cmd.stdout => stretcher.stdout
    go func() {
    io.Copy(os.Stdout, stdout)
    stdout.Close()
    wg.Done()
    }()
    // cmd.stderr => stretcher.stderr
    go func() {
    io.Copy(os.Stderr, stderr)
    stderr.Close()
    wg.Done()
    }()
    wg.Wait()
    return cmd.Wait()
    }
    

    View Slide

40. ·ͱΊ
    interfaceΛ͏·͘࢖͏ͱ͖ͬ͢Γͨ͠ίʔυʹ
    ඪ४Ϟδϡʔϧ͸ๅͷࢁ
    stretcherʹ͍ͭͯͷ࣭໰ɺ͝ཁ๬͸͓ؾܰʹ @fujiwara ·Ͱ
    

    View Slide