Stretcher Shibuya.go #2 2016-03-22 @fujiwara

@fujiwara github.com/fujiwara sfujiwara.hatenablog.com ٕज़෦

What's Stretcher github.com/fujiwara/stretcher Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ

Inspired by github.com/sorah/mamiya & AWS CodeDeploy

ઃܭํ਑ AWS͡Όͳͯ͘΋ಈ͘ tar & rsync & ίϚϯυ࣮ߦ Consul / Serf ͱ͸ૄ݁߹ GoͰॻ͘ Ұͭͷ͜ͱΛ͏·͘΍Δ

Architecture

ಋೖ࣮੷ ʙ100୆ ιʔγϟϧήʔϜɺࣗࣾαʔϏεʙ10਺୆(਺λΠτϧ) ʙ100୆ ੒௕ΛՃ଎͢Δ minne ͷٕज़ج൫ઓུ www.slideshare.net/hsbt/minne-54244702

Stretcherͷ΍Δ͜ͱ(1) 1. Manifest URLΛඪ४ೖྗ͔ΒಡΉ JSON(Consul Event) ·ͨ͸ text 1ߦ(SerfͳͲ) 2. ManifestΛऔಘ s3 or http(s) or file scheme 3. Manifest(YAML)ʹैͬͯDeployॲཧ download & extract & rsync & command

Manifest? src: s3://example.com/app.tar.gz checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93 dest: /home/stretcher/app commands: pre: - /path/to/notify_start post: - /path/to/restart_app success: - /path/to/notify_ok failure: - /path/to/notify_ng excludes: - "*.pid" - "*.socket"

Stretcherͷ΍Δ͜ͱ(2) 1. Manifestʹهड़͞Εͨ src(tar)Λऔಘ 2. pre ίϚϯυΛ࣮ߦ 3. tarΛςϯϙϥϦσΟϨΫτϦʹల։ 4. dest σΟϨΫτϦʹ rsync 5. post ίϚϯυΛ࣮ߦ 6. success or failure ίϚϯυΛ࣮ߦ

Stretcherͷ΍Βͳ͍͜ͱ build, tar, S3΁ͷΞοϓϩʔυ, consul eventൃߦͳͲ͸؅׋֎ github.com/pepabo/capistrano-stretcher Rundeckͱ࢖͏ྫ kikumoto.hatenablog.com/entry/2015/09/16/190036

Imprementation

Imprementation ͓͜ͱΘΓ: εϥΠυͷίʔυ͸ΤϥʔॲཧͷলུଟΊͰ͢

ن໛ײ ຊମ700ߦɺςετؚΊͯ1200ߦఔ౓ͷখ͞͞ 71 cmd/stretcher/main.go 120 aws.go 66 aws_test.go 115 command.go 101 command_test.go 31 consul.go 62 consul_test.go 212 manifest.go 245 manifest_test.go 171 stretcher.go ------------------------------------------------- 720 total 474 total

go-latestͰ࠷৽όʔδϣϯ൑ผ $ stretcher -v version: v0.4.1 build: 2016-02-15T18:36:48+0900 0.4.1 is not latest, you should upgrade to 0.4.2

go-latestͰ࠷৽όʔδϣϯ൑ผ github.com/tcnksm/go-latest func checkLatest(version string) { version = fixVersionStr(version) githubTag := &latest.GithubTag{ Owner: "fujiwara", Repository: "stretcher", FixVersionStrFunc: fixVersionStr, } res, err := latest.Check(githubTag, version) if err != nil { fmt.Println(err) return } if res.Outdated { fmt.Printf("%s is not latest, you should upgrade to %s\n", version, res.Current) } }

go-latestͰ࠷৽όʔδϣϯ൑ผ • GithubTag - GitHub্ͷλάͱൺֱ • JSON - JSON APIͷϨεϙϯεͱൺֱ { "version":"1.2.3", "message":"Fix crash bug when XXX" } • HTMLMeta - URLͰఏڙ͞ΕΔHTMLͷϝλλάͱൺֱ

versionͷ஋Λbuild࣌ʹຒΊࠐΉ Makefile GIT_VER := $(shell git describe --tags) packages: gox (ུ) -ldflags "-X main.version ${GIT_VER}" Gitͷtag͕ main.version ͷ஋ͱͯ͠ઃఆ͞ΕΔ

math/rand ͷSeedΛ͍͍ײ͡ʹॳظԽ͢Δ ίϚϯυϥΠϯΦϓγϣϯ -random-sleep Ͱద౰ʹSleep ىಈλΠϛϯάΛࢄΒ͢ import crand "crypto/rand" import mrand "math/rand" // http://stackoverflow.com/questions/6181260/generating-random-numbers-in-go var s int64 if err := binary.Read(crand.Reader, binary.LittleEndian, &s); err != nil { s = time.Now().UnixNano() } mrand.Seed(s)

ඪ४ೖྗͷParse • ConsulͰ͸JSON(഑ྻܗࣜ)͕౉͞ΕΔ • SerfͰ͸վߦऴ୺ςΩετ͕౉͞ΕΔ ࣗ෼Ͱશ෦ಡ·ͳ͍Ͱɺ͍͍ײ͡ʹ൑ผ͍ͨ͠

ඪ४ೖྗ͔ΒJSON ·ͨ͸ 1ߦΛಡΉ bufio.Reader.Peek() Ͱઌ಄Λ೷͍ͯಡΈ෼͚ func parseEvents() (string, error) { reader := bufio.NewReader(os.Stdin) b, _ := reader.Peek(1) if string(b) == "[" { ev, err := ParseConsulEvents(reader) if err != nil { return "", err } return string(ev.Payload), nil } else { line, _, err := reader.ReadLine() return string(line), err } }

Consul event JSON Payloadʹ͸Base64͞Εͨ஋͕ೖ͍ͬͯΔ [ { "ID": "1d6731a8-833c-1aff-94e5-aa5e5a77da9f" "Name": "deploy", "Payload": "czM6Ly9leGFtcGxlLmNvbS9wYXRoL3RvL3Rhci5neg==", "NodeFilter": "", "ServiceFilter": "", "TagFilter": "", "Version": 1, "LTime": 2, }, ... ]

Consul event JSON ΛಡΉ type ConsulEvents []ConsulEvent type ConsulEvent struct { ID string `json:"ID"` Name string `json:"Name"` Payload []byte `json:"Payload"` LTime int `json:"LTime"` } []byte Ͱఆ͓ٛͯ͘͠ͱBase64Λdecodeͯ͘͠ΕΔ

S3, HTTP(S), File͔Βಉ͡Α͏ʹಡΉ Manifestͷ src ʹ͸4छྨͷURL scheme͕࢖༻Մೳ • s3:// • http:// • https:// • file:// io.ReadCloser interfaceΛѻ͏Α͏ʹઃܭͯ͠ந৅Խ

S3, HTTP(S), File͔Βಉ͡Α͏ʹಡΉ Manifestͷ src ʹ͸4छྨͷURL scheme͕࢖༻Մೳ • s3:// • http:// • https:// • file:// io.ReadCloser interfaceΛѻ͏Α͏ʹઃܭͯ͠ந৅Խ

getURL() - S3, HTTP(S), File͔ΒಡΉ stretcher.getURL() ͸ io.ReadCloser, error Λฦ͢ func getURL(urlStr string) (io.ReadCloser, error) { u, _ := url.Parse(urlStr) switch u.Scheme { case "s3": return getS3(u) case "http", "https": return getHTTP(u) case "file": return getFile(u) default: return nil, fmt.Errorf("manifest URL scheme must be s3 or http(s) or file: %s", urlStr) } }

getS3() - S3͔Βऔಘ͢Δ github.com/AdRoll/goamz func getS3(u *url.URL) (io.ReadCloser, error) { // credentialͷಡΈ͜Έ(ུ) client := s3.New(AWSAuth, AWSRegion) bucket := client.Bucket(u.Host) return bucket.GetReader(u.Path) } // goamz/s3 func (b *Bucket) GetReader(path string) (rc io.ReadCloser, err error) {

getHTTP() - HTTP(S)͔Βऔಘ͢Δ func getHTTP(u *url.URL) (io.ReadCloser, error) { req, _ := http.NewRequest("GET", u.String(), nil) resp, err := http.DefaultClient.Do(req) if err != nil { return nil, err } return resp.Body, nil }

getFile() - File͔Βऔಘ͢Δ func getFile(u *url.URL) (io.ReadCloser, error) { file, err := os.Open(u.Path) if err != nil { return nil, err } return file, nil }

srcΛಡΈࠐΜͰϑΝΠϧʹอଘ อଘ͠ͳ͕Β͍ͭͰʹchecksumܭࢉ func (m *Manifest) Deploy() error { src, _ := getURL(m.Src) defer src.Close() tmp, _ := ioutil.TempFile(os.TempDir(), "stretcher") defer os.Remove(tmp.Name()) written, sum, err := m.copyAndCalcHash(tmp, src)

ϑΝΠϧΛอଘ͠ͳ͕Βchecksumܭࢉ Goຊମͷ io.CopyBuffer() + ϋογϡ஋ܭࢉ func (m *Manifest) copyAndCalcHash(dst io.Writer, src io.Reader) (int64, string, error) { h, _ := m.newHash() // ͨͱ͑͹ sha256.New() buf := make([]byte, 32*1024) for { nr, er := src.Read(buf) // bufʹಡΈࠐΜͰ if nr > 0 { h.Write(buf[0:nr]) // ϋογϡ஋Λܭࢉ nw, ew := dst.Write(buf[0:nr]) // dstʹ΋ॻ͘ if nw > 0 { written += int64(nw) }

if ew != nil { err = ew break } if nr != nw { err = io.ErrShortWrite break } } if er == io.EOF { break } if er != nil { err = er break } } s := fmt.Sprintf("%x", h.Sum(nil)) return written, s, err }

औಘ଎౓Λ͍͍ײ͡ʹϩάʹग़͢ Wrote 218,761,853 bytes to /tmp/stretcher900495202 (in 4.187495 sec, 52MB/s) github.com/dustin/go-humanize begin := time.Now() // URLऔಘͱϑΝΠϧॻ͖ࠐΈ duration := float64(time.Now().Sub(begin).Nanoseconds()) / 1000000000 log.Printf("Wrote %s bytes to %s (in %s sec, %s/s)", humanize.Comma(written), tmp.Name(), humanize.Ftoa(duration), humanize.Bytes(uint64(float64(written)/duration)), )

tar & rsyncͷ࣮ߦ os/exec.Cmd.CombinedOutput() ֎෦ίϚϯυ࣮ߦɺඪ४ग़ྗɺඪ४Τϥʔग़ྗΛ·ͱΊͯฦ͢ out, err := exec.Command("tar", "xf", tmp.Name()).CombinedOutput()

ϢʔβఆٛίϚϯυ(pre,post)ͷ࣮ߦ sh -c ʹ౉࣮ͯ͠ߦ out, err := exec.Command("sh", "-c", c.String()).CombinedOutput() pipe, && || ͳͲ͕ॻ͚ΔͷͰศར ࣮ࡍͷྫ post: - /sbin/service zabbix-agent status || sleep 30 && sudo /sbin/service zabbix-agent start

ϢʔβఆٛίϚϯυ(success, failure)ͷ࣮ߦ stretcherͷ࣮ߦϩά͕ίϚϯυͷstdinʹ౉͞ΕΔ࢓༷ package stretcher var LogBuffer bytes.Buffer // άϩʔόϧ func Init() { log.SetOutput(io.MultiWriter(os.Stderr, &LogBuffer)) ඪ४logϞδϡʔϧͷग़ྗઌΛ os.Stderr ͱ LogBuffer ʹ ඪ४Τϥʔग़ྗ͠ͳ͕ΒίϚϯυʹ౉͢όοϑΝʹ΋ཷΊΔ

ϩάΛstdinʹ౉ͯ͠ίϚϯυ࣮ߦ buf := bytes.NewBuffer(src.Bytes()) c.InvokePipe(buf) func (c CommandLine) InvokePipe(src io.Reader) error { cmd := exec.Command("sh", "-c", c.String()) stdin, _ := cmd.StdinPipe() stdout, _ := cmd.StdoutPipe() stderr, _ := cmd.StderrPipe() cmd.Start()

֎෦ίϚϯυͷstdin΁ϩάΛྲྀ͠ࠐΉ stdinΛಡ·ͳ͍৔߹͸EPIPE͕ൃੜ͢ΔͷͰແࢹ var wg sync.WaitGroup wg.Add(3) // src => cmd.stdin go func() { _, err := io.Copy(stdin, src) if e, ok := err.(*os.PathError); ok && e.Err == syscall.EPIPE { // ignore EPIPE } else if err != nil { log.Println("failed to write to STDIN of", c.String(), ":", err) } stdin.Close() wg.Done() }()

֎෦ίϚϯυͷstdout,stderrΛಡΈࠐΉ sync.WaitGroupͰ଴ͪ߹Θͤͳ͍ͱ຤ඌ͕੾ΕΔ(͜ͱ͕͋Δ) // cmd.stdout => stretcher.stdout go func() { io.Copy(os.Stdout, stdout) stdout.Close() wg.Done() }() // cmd.stderr => stretcher.stderr go func() { io.Copy(os.Stderr, stderr) stderr.Close() wg.Done() }() wg.Wait() return cmd.Wait() }

·ͱΊ interfaceΛ͏·͘࢖͏ͱ͖ͬ͢Γͨ͠ίʔυʹ ඪ४Ϟδϡʔϧ͸ๅͷࢁ stretcherʹ͍ͭͯͷ࣭໰ɺ͝ཁ๬͸͓ؾܰʹ @fujiwara ·Ͱ