GolangでDockerベースのCIを作る

Golang を使って Dockerベース のCIを作るお話です。

Transcript

1. Golang Ͱ Docker ϕʔεͷ CI Λ࡞Δ ɹ GoConference 2019 Spring 2019.05.18 [SAT] Shunsuke Maeda (@duck8823)
2. ࣗݾ঺հ ✓ Shunsuke Maeda ✓ @duck8823 ✓ ॴଐ ✓גࣜձࣾΤεɾΤϜɾΤε ✓EarthCampusגࣜձࣾʢ෭ʣ 2/43
3. CI͠ΜͲ͍ 3/43
4. CI͠ΜͲ͍ ! ✓CIΛߏங͍ͨ͠ ci := ci.New() for ci.IsBroken { // CI͕յΕͯͨΒ fix(ci) // ϩʔΧϧͰ௚͚͢Ͳಈ࡞֬ೝͰ͖ͳ͍ͷͰ git.Commit() // ίϛοτ ͯ͠ git.Push() // ϓογϡ ͯ͠ ci.Run() // CI ಈ͔ͯ͠ΈΔ } 4/43
5. Golang Ͱ Docker ϕʔεͷ CI Λ࡞Δ duck8823/duci 5/43
6. duci ͷಛ௃ ✓DockerfileͰఆٛ ✓ ֶशίετ͕௿͍ ✓ ϩʔΧϧͰ΋࣮ߦ ✓GitHub ʹରԠ ✓ Push / PR্ͷίϝϯτ Ͱ࣮ߦ ✓ ίϛοτεςʔλεΛ࡞੒ 6/43
7. ֶशίετ ✓Dockerfile ͷΈ FROM golang:1.12.4-alpine # ϕʔεΠϝʔδ RUN apk --update add --no-cache ... # ඞཁͳύοέʔδͷΠϯετʔϧ WORKDIR /workdir COPY . . ENTRYPOINT ["make"] # λεΫϥϯφʔͷར༻Λਪ঑ CMD ["test"] # σϑΥϧτͷλεΫ 7/43
8. ֶशίετ ✓Dockerͷ஌ࣝͰΩϟογϡԽ FROM golang:1.12.4-alpine RUN apk --update add --no-cache ... WORKDIR /workdir COPY go.mod . COPY go.sum . RUN go mod download COPY . . ENTRYPOINT ["make"] CMD ["test"] 8/43
9. ϩʔΧϧͰ࠶ݱ ✓ίϚϯυΛ༻ҙ ✓ϩʔΧϧͰࢼ͔ͯ͠Β commit & push Ͱ͖Δ duci run INFO[14/May/2019 09:25:57.353] Step 1/9 : FROM openjdk:11 as Build INFO[14/May/2019 09:25:57.353] INFO[14/May/2019 09:25:57.353] ---> 0aa10063a184 INFO[14/May/2019 09:25:57.353] Step 2/9 : WORKDIR /workdir INFO[14/May/2019 09:25:57.353] INFO[14/May/2019 09:25:57.353] ---> Using cache INFO[14/May/2019 09:25:57.353] ---> 2dcc968d8994 ... 9/43
10. CIΛ࡞Δ 10/43
11. Continuous Integration 11/43
12. Continuous Integration ✓ఆظతʹ ✓Ϗϧυ΍ςετΛ࣮ߦ ͢Δ͜ͱͰ ✓ૣظʹ ϑΟʔυόοΫ Λಘͯ όά͕ຊ൪؀ڥʹࠞೖ͢ΔͷΛ๷͙ 12/43
13. CIͰߟ͑Δඞཁ͕͋Δ͜ͱ ✓ఆظతʹ ✓ ࣮ߦλΠϛϯά ✓Ϗϧυ΍ςετΛ࣮ߦ ✓ ࣮ߦ؀ڥ ✓ϑΟʔυόοΫ ✓ ࣮ߦ݁Ռ 13/43
14. duci ʹ͓͚Δߏ੒ ✓࣮ߦλΠϛϯά ✓ GitHub Webhooks ✓࣮ߦ؀ڥ ✓ Dockerίϯςφ ✓࣮ߦ݁Ռ ✓ ϩάग़ྗ / GitHub ίϛοτεςʔλε 14/43
15. CIͰߟ͑Δඞཁ͕͋Δ͜ͱ ✓ఆظతʹ ✓ ࣮ߦλΠϛϯά ✓Ϗϧυ΍ςετΛ࣮ߦ ✓ ࣮ߦ؀ڥ ✓ϑΟʔυόοΫ ✓ ࣮ߦ݁Ռ 15/43
16. ࣮ߦλΠϛϯά ✓GitHub Webhooks https://github.com/<owner>/<repo>/settings/hooks ✓ ೚ҙͷλΠϛϯάͰ Payload Λඈ͹͢ 16/43
17. PayloadΛड͚औΔ ✓GitHub ͔ΒඈΜͰ͘Δ Payload JSON Λड͚औΔඞཁ͕͋Δ import "net/http" func JobHandler(w http.ResponseWriter, r *http.Request) { // ֤ΠϕϯτʹରԠͨ͠PayloadͷύʔεॲཧͱCIδϣϒͷ࣮ߦ } func main() { http.HandleFunc("/", JobHandler) http.ListenAndServe(":8080", nil) } 17/43
18. GitHub ͷ Webhooks ✓Event ͱ Action ͷ૊Έ߹ΘͤͰλΠϛϯά੍ޚ ✓Event ✓ Webhooksͷछྨ (Header: X-GitHub-Event) ྫ. push, issue_comment, pull_request ✓ GitHubͰઃఆՄೳ ✓Action ✓ Eventຖͷࡉ͔͍छྨ (JSON payload಺) ྫ. created, opened, synchronize 18/43
19. Eventͷऔಘ ✓ϦΫΤετͷϔομʔ X-GitHub-Event ͔ΒΠϕϯτΛऔಘ ✓Πϕϯτຖʹ Payload ͕ҟͳΔ func JobHandler(w http.ResponseWriter, r *http.Request) { event := r.Header.Get("X-GitHub-Event") switch event { case "push": // https://developer.github.com/v3/activity/events/types/#pushevent case "issue_comment": // https://developer.github.com/v3/activity/events/types/#issuecommentevent case "pull_request": // https://developer.github.com/v3/activity/events/types/#pullrequestevent default: http.Error(w, fmt.Sprintf("Bad event type: %s", event), http.StatusBadRequest) } } 19/43
20. Payload JSON ͷऔΓѻ͍ ✓encoding/json ͰσίʔυͰ͖Δ ✓Payloadͷܕ github.com/google/go-github import ( "encoding/json" "github.com/google/go-github/github" ) func JobHandler(w http.ResponseWriter, r *http.Request) { switch r.Header.Get("X-GitHub-Event") { case "push": event := &github.PushEvent{} err := json.NewDecoder(r.Body).Decode(event) ... } } 20/43
21. Forkͨ͠ϦϙδτϦ͔Βͷ Pull Request ✓push Πϕϯτ͸ඈΜͰ͜ͳ͍ ✓ pull_request Πϕϯτͷ synchronize ΞΫγϣϯ func JobHandler(w http.ResponseWriter, r *http.Request) { switch r.Header.Get("X-GitHub-Event") { case "pull_request": event := &github.PullRequestEvent{} json.NewDecoder(r.Body).Decode(event) switch event.GetAction() { case "synchronize", "open": ... } } } 21/43
22. Pull Request ͷίϝϯτ import ( "github.com/google/go-github/github" "golang.org/x/oauth2" ) func JobHandler(w http.ResponseWriter, r *http.Request) { switch r.Header.Get("X-GitHub-Event") { case "issue_comment": event := &github.IssueComment{} json.NewDecoder(r.Body).Decode(event) ts := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: "access token"}) tc := oauth2.NewClient(context.Background(), ts) cli := github.NewClient(tc) pr := cli.PullRequests.Get( context.Background(), ownerName, repoName, event.GetIssue().GetNumber(), ) ... } } 22/43
23. ඇಉظͰϨεϙϯεΛฦ͢ ✓δϣϒ͸͕͔͔࣌ؒΔ => ඇಉظʹ࣮ߦͯ͠ϨεϙϯεΛฦ͢ func JobHandler(w http.ResponseWriter, r *http.Request) { if (invalidRequest(r)) { http.Error(w, r.Error(), http.StatusBadRequest) return } go func() { // CIδϣϒͷ࣮ߦ }() w.WriteHeader(http.StatusOK) } 23/43
24. ࣮ߦλΠϛϯά (·ͱΊ) ✓ϦΫΤετड෇ => net/http ✓Payload ͷܕ => ϥΠϒϥϦͷར༻ ྫ. GitHub Webhooks: github.com/google/go-github ✓ GitHub ͸ Event ͱ Action ͰλΠϛϯάΛ੍ޚ ✓ඇಉظͰϨεϙϯεΛฦ͢ => goroutine 24/43
25. CIͰߟ͑Δඞཁ͕͋Δ͜ͱ ✓ఆظతʹ ✓ ࣮ߦλΠϛϯά ✓Ϗϧυ΍ςετΛ࣮ߦ ✓ ࣮ߦ؀ڥ ✓ϑΟʔυόοΫ ✓ ࣮ߦ݁Ռ 25/43
26. ࣮ߦ؀ڥ ✓Dockerίϯςφ ✓ ϗετΛԚ͞ͳ͍ ✓ ϩʔΧϧ / CI Ͱͷಈ࡞ʹ࠶ݱੑΛͱΓ΍͍͢ ✓github.com/docker/docker Λར༻ 26/43
27. github.com/docker/docker ✓ϦϙδτϦ͸ moby/moby ✓Docker Daemon ͷରԠAPIόʔδϣϯ ɹ- Docker for Mac ͸ 1.39 ✓ Moby ͷ master ͸ 1.41 docker version Server: Docker Engine - Community Engine: Version: 18.09.2 API version: 1.39 (minimum version 1.12) ... 27/43
28. DockerΠϝʔδ/ίϯςφΛѻ͏ import "github.com/docker/docker/client" cli, _ := client.NewClientWithOpts(client.FromEnv) ✓௚ײతͳAPI ✓ cli#ImageBuild = docker build ✓ cli#ContainerCreate = docker create ✓ cli#ContainerStart = docker start 28/43
29. DockerΠϝʔδͷϏϧυ ✓ίϯςΩετʢσΟϨΫτϦʣΛ tarball ͷܗࣜʹ͢Δඞཁ͕͋Δ ✓cli#ImageBuild ✓ ඇಉظʹ࣮ߦ͞ΕΔ ✓ resp.Body Λ EOF ·ͰಡΜͰऴྃ଴ͪ opts := types.ImageBuildOptions{ Tags: []string{"tag"} } resp, _ := cli.ImageBuild(ctx, tarball("/workdir"), opts) ioutil.ReadAll(resp.Body) 29/43
30. ίϯςφͷ࣮ߦ ✓ίϯςφͷ࡞੒͸ cli#ContainerCreate ✓cli#ContainerStart ͸ඇಉظ ✓ cli#ContainerWait ΍ cli#ContainerLogs Ͱδϣϒͷ৘ใΛऔಘ sopts := types.ContainerStartOptions{} cli.ContainerStart(context.Background(), "containerId", sopts) code, _ := cli.ContainerWait(context.Background(), "containerId") if code != 0 { // exit code ͕ 0Ҏ֎ => δϣϒͷࣦഊ } lopts := types.ContainerLogsOptions{ ShowStdout: true, ShowStderr: true } log _ := cli.ContainerLogs(context.Background(), "containerId", lopts) 30/43
31. δϣϒͷฒྻ਺ͱλΠϜΞ΢τ ✓ϦΫΤετຖʹ goroutine Ͱ࣮ߦ ✓ δϣϒΛ͍ͭ͘΋࣮ߦͯ͠͠·͏ ✓ϗετͷϦιʔε͸༗ݶ => ฒྻ਺ͷ੍ޚ ✓ ͕͔͔࣌ؒΔॲཧ͕͋Δͱ࣍ͷδϣϒ͕࣮ߦ͞Εͳ͍ => λΠϜΞ΢τͷઃఆ 31/43
32. ฒྻ਺ͷ੍ޚ ✓channel Λར༻ var sem = make(chan struct{}, 2) // 2ͭ࿮Λ༻ҙ func execute() { sem <- struct{}{} // ࿮ΛҰͭ֬อ͢Δ(ۭ͖͕ͳ͚Ε͹͜͜ͰॲཧΛ଴ͭ worker.execute() // CIδϣϒͷ࣮ߦ <-sem // ࿮ΛҰͭղ์͢Δ } func main() { go execute() go execute() go execute() } 32/43
33. λΠϜΞ΢τͷઃఆ ✓context#WithTimeout ͰλΠϜΞ΢τΛઃఆͰ͖Δ func execute() { err := make(chan error) timeout, cancel := context.WithTimeout(context.Background(), 10 * time.Second) defer cancel() go func() { err <- worker.execute() // CIδϣϒͷ࣮ߦ }() select { case <-timeout.Done(): println("timeout") case e := <-err: println("done") } } 33/43
34. ࣮ߦ؀ڥ (·ͱΊ) ✓DockerΫϥΠΞϯτ ✓ github.com/docker/docker ✓ ௚ײతͳAPI ✓ ಉظॲཧ͸ image build ͱ container start ͰҟͳΔ ✓ฒྻ਺ͷ੍ޚ ✓ channel ✓λΠϜΞ΢τͷઃఆ ✓ context#WithTimeout 34/43
35. CIͰߟ͑Δඞཁ͕͋Δ͜ͱ ✓ఆظతʹ ✓ ࣮ߦλΠϛϯά ✓Ϗϧυ΍ςετΛ࣮ߦ ✓ ࣮ߦ؀ڥ ✓ϑΟʔυόοΫ ✓ ࣮ߦ݁Ռ 35/43
36. ࣮ߦ݁Ռ ✓GitHub ͷίϛοτεςʔλε 36/43
37. ίϛοτεςʔλε ts := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: "ΞΫηετʔΫϯ"}) tc := oauth2.NewClient(context.Background(), ts) cli := github.NewClient(tc) stat := &github.Status{ Context: github.String("ίϛοτεςʔλεͷ໊લ"), Description: github.String("ৄࡉઆ໌ͳͲ(จࣈ਺੍ݶ͋Γ)"), State: github.String("εςʔλεʢྫ. failureʣ"), TargetURL: github.String("֎෦ϖʔδ΁ͷϦϯΫ"), } cli.Repositories.CreateStatus( context.Background(), "ownerName", "repoName", "ref (commit hash)", stat, ) 37/43
38. ϩά / Ϩεϙϯεͷදࣔ ✓http.ResponseWriter#Write ͰϨεϙϯεʹॻ͖ࠐΈ func LogHandler(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) w.Write([]byte("log")) } 38/43
39. ࣮ߦ؀ڥͱ࣮ߦ݁ՌΛૄʹ͢Δ ✓ૄͳઃܭʹ͓ͯ͘͠ͱࠩ෼ͰػೳΛ࣮૷͠΍͍͢ ✓݁ՌΛग़ྗ͍ͨ͠λΠϛϯά͸δϣϒ࣮ߦத(લޙ)ʹ΋ ✓ δϣϒͷొ࿥࣌ ✓ δϣϒͷ࣮ߦ։࢝࣌ ✓ ϩά ✓ δϣϒͷऴྃ࣌ʢ੒ޭ/ࣦഊ/Τϥʔ) 39/43
40. ࣮ߦ؀ڥͱ࣮ߦ݁ՌΛૄʹ͢Δ ✓࣮ߦ؀ڥͰϑΟʔυόοΫ༻ؔ਺ΛઃఆͰ͖ΔΑ͏ʹ͢Δ type worker struct { startFun func() } func (w *worker) run() { err := w.startFun() ... } func localRun() { w := &worker{ startFun: printLog } w.run() } func serverRun() { w := &worker{ startFun: printLogAndCommitStatus } w.run() } 40/43
41. ࣮ߦ݁Ռ (·ͱΊ) ✓GitHubͷίϛοτεςʔλε ✓ github.com/google/go-github ✓Ϩεϙϯε ✓ net/http ͷ http.ResponseWriter#Write ✓࣮ߦ؀ڥͱ࣮ߦ݁ՌΛૄʹ͢Δ ✓ ϑΟʔυόοΫ༻ؔ਺ΛઃఆͰ͖ΔΑ͏ʹ͢Δ 41/43
42. ·ͱΊ ✓ඪ४ػೳ/ඪ४ϥΠϒϥϦ ✓ ඇಉظ΍λΠϜΞ΢τ ✓ HTTPαʔόʔ ✓GitHub ΍ Docker ͳͲͷΫϥΠΞϯτ͸ϥΠϒϥϦΛར༻ ✓ҰͭҰͭͷʢهड़͸ʣ؆ܿ ✓ ૊Έ߹ΘͤͯCIΛ࡞Δ͜ͱ͕Ͱ͖Δ 42/43
43. Let's make CI server! 43/43