Designing Go Services For Continuous Deployment (Am Laher 2019)

Transcript

1. Designing Go services for Continuous Designing Go services for Continuous Deployment Deployment Gophercon AU Gophercon AU 1 November 2019 1 November 2019 Am Laher Am Laher Lead Engineer, Vend Lead Engineer, Vend
2. Begin with a Premise Begin with a Premise Continuous Deployment can improve the design of your code in subtle, but remarkable Continuous Deployment can improve the design of your code in subtle, but remarkable ways. ways. 2 2
3. What is this, "Continuous Deployment"? What is this, "Continuous Deployment"? For the purposes of this talk ... For the purposes of this talk ... Continuous Deployment is a release strategy: any mainline change that passes testing, is Continuous Deployment is a release strategy: any mainline change that passes testing, is automatically released into production. automatically released into production. 3 3
4. Calm & Peaceful, right? Calm & Peaceful, right? OK, OK , it does sound a little bit super-risky. OK, OK , it does sound a little bit super-risky. Let's explore... Let's explore... 4 4
5. Let's make up a story ... Let's make up a story ... 5 5
6. Toasties Galore Toasties Galore Meet our heroine, Gita the Gopher. Meet our heroine, Gita the Gopher. Gita is an innovator, with a passion for technology and Gita is an innovator, with a passion for technology and melted cheese. melted cheese. 6 6
7. State of the art Techmology State of the art Techmology Gita builds her own machinery, it's software-driven. Gita builds her own machinery, it's software-driven.We We - - you and I you and I - - build the software. build the software. 7 7
8. Toasties Galore software Toasties Galore software An overview of the codebase: An overview of the codebase: A monorepo. A monorepo. 2 HTTP services, jafflr+chillybin, which interact. 2 HTTP services, jafflr+chillybin, which interact. Tests, Docker things, scripts and tooling. Tests, Docker things, scripts and tooling. A CI/CD pipeline. A CI/CD pipeline. See See github.com/laher/toasties-galore github.com/laher/toasties-galore (https://github.com/laher/toasties-galore) (https://github.com/laher/toasties-galore) 8 8
9. The Toasties Galore software architecture The Toasties Galore software architecture Invoking Jafflr: `POST /toastie` Invoking Jafflr: `POST /toastie` Gita Jafflr HTTP Request Toastie Chillybin Postgres (DB) This simple deployment powers a 24-hour, high-throughput toasting service, with some This simple deployment powers a 24-hour, high-throughput toasting service, with some agressive expectations. agressive expectations. Gophers are demanding. Gophers are demanding. 9 9
10. The CI/CD Pipeline The CI/CD Pipeline Any change Master only Changes Github Merge/Push Verify Test Build Publish DockerHub Travis Webhook Deploy Production 10 10
11. The pipeline The pipeline In this case, we're using In this case, we're using Travis Travis (https://travis-ci.org/laher/toasties-galore/branches) (https://travis-ci.org/laher/toasties-galore/branches) language: generic sudo: required services: - docker env: - GO111MODULES="on" script: - go test -mod=vendor ./chillybin ./jafflr ./tpi # <- run unit/functional tests - docker-compose up -d # <- prepare for integration tests - go test -mod=vendor -v ./integration-tests # <- run integration tests - bash ./scripts/publish.sh # <- scripted because monorepo deploy: provider: script script: bash ./scripts/deploy.sh prod # <- 'tag as prod' for now on: branch: master 11 11
12. Eek, auto-deploy is scary Eek, auto-deploy is scary 12 12
13. The Auto-Deploy Baseline The Auto-Deploy Baseline What kind of things should we have in place? What kind of things should we have in place? 13 13
14. The Baseline The Baseline Let's consider some Important Things in our code: Let's consider some Important Things in our code: Observability Observability Seamless Deploys Seamless Deploys Integration Tests Integration Tests 14 14
15. Jafflr schematic Jafflr schematic 15 15
16. Jafflr Jafflr func main() { var ( listenAddr = tpi.Getenv("ADDR", ":7000") chillybinAddr = tpi.Getenv("CHILLYBIN_ADDR", "http://127.0.0.1:7001") h = &handler{ client: chillybinClient{chillybinAddr}, // <- teh mechanical arm } server = &http.Server{ Addr: listenAddr, Handler: tpi.Middleware(newRouter(h)), // <- middleware for observability } ) log.Println("Server is about to listen for requests at", listenAddr) go func() { if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed { log.Fatalf("Could not listen on %s: %v\n", listenAddr, err) } }() tpi.GracefulShutdown(server) // <- for downtimeless deploy log.Println("Shutdown complete - stop") } Run 16 16
17. Observability: Instrumentation Observability: Instrumentation Use metrics to know what's happening in production. e.g. in a middleware ... Use metrics to know what's happening in production. e.g. in a middleware ... func TracingMiddleware(in http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t := time.Now().UTC() wsr := &responseStatusRecorder{ResponseWriter: w} in.ServeHTTP(wsr, r) // the wrapped handler is still invoked here labelsMap := map[string]string{ "method": r.Method, "status": strconv.Itoa(wsr.status), "version": version, // <- we can use this to compare error rates } someTracingObserve(time.Since(t), labelsMap) // maybe honeycomb/prometheus/statsd ... }) } 17 17
18. Seamless Deploys: The Rolling Update Seamless Deploys: The Rolling Update Stop listening, keep handling ... e.g. Stop listening, keep handling ... e.g. Continues processing in-flight requests, while making way for a newer deployment. Continues processing in-flight requests, while making way for a newer deployment. func GracefulShutdown(server *http.Server) { osInterrupt := make(chan os.Signal) signal.Notify(osInterrupt, os.Interrupt) <-osInterrupt // subscribe to Ctrl+C or kill/SIGTERM log.Println("Received OS interrupt - stop listening but avoid interrupting existing connections...") ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() server.SetKeepAlivesEnabled(false) if err := server.Shutdown(ctx); err != nil { // keeps existing connections open until ctx.Done() log.Fatalf("Server shutdown error: %v", err) } } 18 18
19. The Arm - An interaction The Arm - An interaction 19 19
20. Integration Tests Integration Tests Few tests - verify the interactions. Few tests - verify the interactions. func TestHappyPath(t *testing.T) { // reset environment: if resp, err := http.Post(fmt.Sprintf("%s/restock", chillybinAddr), "text/plain", nil); err != nil { t.Fatalf("error restocking chillybin: %v", err) } else if resp.StatusCode != http.StatusOK { body, _ := ioutil.ReadAll(resp.Body) t.Fatalf("error restocking chillybin (%s): %s", resp.Status, body) } if m := getChillybinStats(t); m["cheese"] != 10 { // assert initial state t.Fatalf("wrong amount of cheese after restocking: %v, %T", m, m["cheese"]) } // Method under test: if resp, err := http.Post(fmt.Sprintf("%s/toastie?i=cheese&i=vegemite", jafflrAddr), "text/plain", n t.Fatalf("error fetching toastie: %v", err) } else if resp.StatusCode != http.StatusOK { body, _ := ioutil.ReadAll(resp.Body) t.Fatalf("error fetching toastie (%s): %s", resp.Status, body) } if m := getChillybinStats(t); m["cheese"] != 9 { // assert resulting state t.Fatalf("wrong amount of cheese after grilling toastie: %v", m) } } 20 20
21. Success Success Toasties Galore is now a highly successful operation. Gophers Toasties Galore is now a highly successful operation. Gophers love love the recipes. Jafflr is busy the recipes. Jafflr is busy 24-7. 24-7. 21 21
22. But then ... But then ... Some gophers aren't happy ... Some gophers aren't happy ... We're super busy, and some orders are going to the wrong gophers. We're super busy, and some orders are going to the wrong gophers. Gwenda got pineapple instead of vegemite. Gwenda got pineapple instead of vegemite. Gary received cheese instead of Sheese (soy cheese). Gary received cheese instead of Sheese (soy cheese). Not good. Not good. 22 22
23. Design Problems - What to do? Design Problems - What to do? Decisions ... Decisions ... Gita: Gita: Track orders with customers' names. Track orders with customers' names. Us: Us: alter the Chillybin API to support richer request data. alter the Chillybin API to support richer request data. 23 23
24. Planned changes to /pick Planned changes to /pick Before Jafflr Chillybin GET DB UPDATE ingredients After Jafflr Chillybin POST DB INSERT orders UPDATE ingredients 24 24
25. Oh noes, Breaking Changes Oh noes, Breaking Changes API signature needs to change. API signature needs to change. DB structure changes. DB structure changes. 25 25
26. Us: OMG Us: OMG Gita Gita: s'all good. Let's dance. : s'all good. Let's dance. 26 26
27. The Foxtrot The Foxtrot So, let's dance. So, let's dance. A 3-step dance: A 3-step dance: Add new implementation. Add new implementation. Migrate comfortably. Migrate comfortably. Delete old implementation. Delete old implementation. 27 27
28. The 3-step release dance The 3-step release dance Three steps, or Three steps, or stages stages With CD, it's easy, you'll consider it more often. With CD, it's easy, you'll consider it more often. 28 28
29. Step 1 - Add new feature implementation Step 1 - Add new feature implementation Serve the new approach under a different URL, /v2/pick Serve the new approach under a different URL, /v2/pick Now there are 2 versions of Now there are 2 versions of pick pick, side-by-side. , side-by-side. v2 can be tested manually, but isn't invoked by any other service yet. v2 can be tested manually, but isn't invoked by any other service yet. No changes to calling service (jafflr). Just chillybin. No changes to calling service (jafflr). Just chillybin. See See github.com/laher/toasties-galore/pull/1/files github.com/laher/toasties-galore/pull/1/files (https://github.com/laher/toasties-galore/pull/1/files) (https://github.com/laher/toasties-galore/pull/1/files) 29 29
30. Step 2 - Invoke new implementation Step 2 - Invoke new implementation Use Feature Flags? Use Feature Flags? func HasFeature(customer string, feature string) bool { switch feature { case "pick.v2": if customer == "gita" { return true } return false default: panic("unknown feature") } } See See github.com/laher/toasties-galore/pull/2/files github.com/laher/toasties-galore/pull/2/files (https://github.com/laher/toasties-galore/pull/2/files) (https://github.com/laher/toasties-galore/pull/2/files) HasFeature could be backed by a service, a db table, or by configuration. HasFeature could be backed by a service, a db table, or by configuration. Alternatives: Canary Deploys, Shadowing, Blue-Green deploys Alternatives: Canary Deploys, Shadowing, Blue-Green deploys 30 30
31. Step 3 - Delete old implementation Step 3 - Delete old implementation Do it soon. Do it soon. Line it up in advance. Plan for it, otherwise you'll forget. Line it up in advance. Plan for it, otherwise you'll forget. See See github.com/laher/toasties-galore/pull/3/files github.com/laher/toasties-galore/pull/3/files (https://github.com/laher/toasties-galore/pull/3/files) (https://github.com/laher/toasties-galore/pull/3/files) It's the easy part, but it's also easy to forget it, or fear it. It's the easy part, but it's also easy to forget it, or fear it. 31 31
32. Foxtrot - Conclusions Foxtrot - Conclusions When deployment is a non-event: When deployment is a non-event: you can make your releases safer. you can make your releases safer. you code remains cleaner. you code remains cleaner. When you have more viable options within easy reach, you're empowered to make better When you have more viable options within easy reach, you're empowered to make better decisions. decisions. 32 32
33. Tips and Tricks Tips and Tricks 33 33
34. On Configuration On Configuration Deprecate extensive config files... Deprecate extensive config files... Configurable-everything is a popular pattern Configurable-everything is a popular pattern Makes it easier to reconfigure while an on-call emergency? Makes it easier to reconfigure while an on-call emergency? Knobs and twiddles are nice. Knobs and twiddles are nice. 34 34
35. But, with CD, we can simplify But, with CD, we can simplify Use $ENV variables for DSNs and such. [see 12-factor Apps] Use $ENV variables for DSNs and such. [see 12-factor Apps] Use constants more often. Use constants more often. Deployment is fast. Just edit code/artifact Deployment is fast. Just edit code/artifact 35 35
36. On Monorepos On Monorepos Monorepos offer some challenges and some advantages Monorepos offer some challenges and some advantages Benefit: easier to maintain CI/CD pipelines centrally. Benefit: easier to maintain CI/CD pipelines centrally. Benefit: in-house dependencies are all local. Benefit: in-house dependencies are all local. Benefit: testing component combinations is easier. Benefit: testing component combinations is easier. Challenge: identifying what to build, test, deploy on each commit. Challenge: identifying what to build, test, deploy on each commit. Challenge: speed. Challenge: speed. Challenge: 3p deps: _See toasties-galore for top-level go-mod w Docker_ Challenge: 3p deps: _See toasties-galore for top-level go-mod w Docker_ 36 36
37. Deploying services in a monorepo Deploying services in a monorepo Q: Do we really redeploy everything on every change? Q: Do we really redeploy everything on every change? A: Depends on language and tooling available. A: Depends on language and tooling available. 37 37
38. Publish / deploy monorepo service - Go Publish / deploy monorepo service - Go See See publish.sh publish.sh (./scripts/publish.sh) (./scripts/publish.sh) and and deploy.sh deploy.sh (./scripts/deploy.sh) (./scripts/deploy.sh) #!/bin/bash echo "BUILD/PUBLISH: begin" echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin function docker_tag_exists() { echo "CHECK: does $1 docker tag exist for commit $2 ?" curl --silent -f -lSL https://index.docker.io/v1/repositories/$1/tags/$2 > /dev/null } for SVC in jafflr chillybin do export SVC_COMMIT=$(./scripts/last_commit.sh "$SVC") if docker_tag_exists "$DOCKER_USERNAME/$SVC" "$SVC_COMMIT"; then echo "SKIP: Docker tag already exists for $DOCKER_USERNAME/$SVC: $SVC_COMMIT" else echo "PUBLISH: publishing image for $DOCKER_USERNAME/$SVC: $SVC_COMMIT" docker build -t $SVC -f ./$SVC/Dockerfile . # rebuild just in case docker tag $SVC $DOCKER_USERNAME/$SVC docker tag $SVC $DOCKER_USERNAME/$SVC:$SVC_COMMIT docker push $DOCKER_USERNAME/$SVC:$SVC_COMMIT fi done echo "BUILD/PUBLISH: done" 38 38
39. Last relevant commit - Go Last relevant commit - Go Calculate changes affecting a service [in a monorepo] Calculate changes affecting a service [in a monorepo] #!/bin/sh if [ -z $1 ]; then echo '"$1" should be service-name' >&2 exit 1 fi export PKG=$1 cd $PKG # Use go-list to dependencies of a package IMPORTS=$(go list -deps|grep 'github.com/laher/toasties-galore'|sed 's,github.com/laher/toasties-galore/ cd .. # Use git-rev-list to find last commit affecting dirs LAST_COMMIT=$(git rev-list -1 HEAD -- $IMPORTS $PKG ./go.mod ./go.sum) echo $LAST_COMMIT 39 39
40. Why last_commit.sh ? Why last_commit.sh ? Deploy only affected services. Deploy only affected services. Decide which integration tests to run, post-deploy Decide which integration tests to run, post-deploy 40 40
41. On DB Migrations On DB Migrations Migrations will always be challenging Migrations will always be challenging Altering big tables - time-consuming, performance concerns. Altering big tables - time-consuming, performance concerns. Rollbacks are awkward. Rollbacks are awkward. Where possible: Where possible: Use transactions (Postgres has transactional DDL) to Use transactions (Postgres has transactional DDL) to test test them in advance. them in advance. Make them idempotent (`if not exists`) to Make them idempotent (`if not exists`) to run run them in advance. them in advance. Split them up - 'create table' is cheap. Isolate the tricky stuff. Split them up - 'create table' is cheap. Isolate the tricky stuff. 41 41
42. Chillybin Chillybin 42 42
43. Chilly bin - migrations Chilly bin - migrations func runMigrationsSource(db *sql.DB) error { driver, err := postgres.WithInstance(db, &postgres.Config{}) if err != nil { return err } m, err := migrate.NewWithDatabaseInstance( "file://resources/migrations", // <- local files, bundled in container "postgres", driver) if err != nil { return err } if err := m.Up(); err != nil { if err == migrate.ErrNoChange { return nil } return err } return nil } 43 43
44. On DIY Tooling On DIY Tooling At Vend we have a slackbot, @overlord, which offers a number of CD-related utilities. e.g. At Vend we have a slackbot, @overlord, which offers a number of CD-related utilities. e.g. canary deploys canary deploys rollbacks rollbacks feature flagging feature flagging service status service status 44 44
45. On CI/CD Software On CI/CD Software They all work, and they're all different. I've tried heaps. They all work, and they're all different. I've tried heaps. My examples are only chosen to communicate some ideas. My examples are only chosen to communicate some ideas. Just choose one. Just choose one. 45 45
46. Conclusions Conclusions 46 46
47. How will it change my code? How will it change my code? More than anything else, CD affects More than anything else, CD affects how how you you change change your code. your code. CD forces you to deal with Important Things, up front. Most of these have their own CD forces you to deal with Important Things, up front. Most of these have their own payoff. payoff. CD encourages good practices. CD encourages good practices. CD enables safe and easy changes. CD enables safe and easy changes. Impact: Impact: Simpler: fewer contraptions in the code. Simpler: fewer contraptions in the code. Tidyier: fewer cobwebby corners. Tidyier: fewer cobwebby corners. More dynamic: change is simpler -> more, better options. More dynamic: change is simpler -> more, better options. 47 47
48. Advice Advice It can be hard to retrofit CD. Start early or add it incrementally. It can be hard to retrofit CD. Start early or add it incrementally. For research and business speak, see For research and business speak, see continuousdelivery.com continuousdelivery.com (https://continuousdelivery.com) (https://continuousdelivery.com) . . 48 48
49. Advice: Play with Toasties Galore Advice: Play with Toasties Galore See Toasties Galore on See Toasties Galore on github github (https://github.com/laher/toasties-galore) (https://github.com/laher/toasties-galore) or or travis travis (https://travis-ci.org/laher/toasties-galore) (https://travis-ci.org/laher/toasties-galore) to try this stuff to try this stuff out - maybe it'll help you. out - maybe it'll help you. 49 49
50. Thank you Thank you Am Laher Am Laher Lead Engineer, Vend Lead Engineer, Vend @amfastic @amfastic (http://twitter.com/amfastic) (http://twitter.com/amfastic)