Fully automated release processes for Go projects

Go projects need to rely on an automated process that ushers changes through all aspects of a delivery pipeline. A typical pipeline involves resolving packages, compilation, testing, generation of code coverage, static code analysis, binary creation, and the actual release of the binaries.

This session discusses automation aspects and tools helpful for streamlining Go projects. First, you’ll get to know individual tools and why they are so useful. Then you’ll tie it all together to form a simple delivery pipeline for Go projects powered by Travis CI or Jenkins. You will leave the session with a recipe for implementing a sensible, yet simple end-to-end automation approach for your own Go projects to ensure reliable and reproducible builds.

Transcript

1. Fully automated
   release processes
   for projects
   Benjamin Muschko
   

   View Slide

2. AUTOMATED
   ASCENT
   bmuschko
   bmuschko
   bmuschko.com
   About the speaker
   automatedascent.com
   

   View Slide

3. Who’s using the
   language Go?
   

   View Slide

4. Typical development tasks
   Package
   Management
   Source Code
   Analysis
   Testing
   Building
   Binaries
   Publishing
   Binaries
   CI/CD
   Pipeline
   

   View Slide

5. Do I need a build tool?
   

   View Slide

6. Go Modules
   The de facto
   packager
   manager
   

   View Slide

7. High-level usage
   Enabled via env. variable GO111MODULE
   Check in generated go.mod, go.sum
   Direct integra5on with go get
   Will try to download from central proxy first
   

   View Slide

8. SemanFc versioning
   2 . 13 . 5
   hBps:/
   /semver.org/
   
   Major Minor Patch
   (Breaking) (Feature) (Fix)
   

   View Slide

9. Dependency declaraFon
   module github.com/bmuschko/letsgopher
   require (
   github.com/Flaque/filet v0.0.0-20190209224823-fc4d33cfcf93
   github.com/blang/semver v3.5.1+incompatible
   github.com/ghodss/yaml v1.0.0
   github.com/gosuri/uitable v0.0.0-20160404203958-36ee7e946282
   github.com/inconshreveable/mousetrap v1.0.0 // indirect
   github.com/kr/text v0.1.0
   github.com/mattn/go-runewidth v0.0.4 // indirect
   github.com/mitchellh/go-homedir v1.0.0
   github.com/spf13/afero v1.2.2 // indirect
   github.com/spf13/cobra v0.0.3
   github.com/spf13/pflag v1.0.3 // indirect
   github.com/stretchr/testify v1.3.0
   gopkg.in/AlecAivazis/survey.v1 v1.8.2
   gopkg.in/yaml.v2 v2.2.2 // indirect
   )
   go 1.13
   

   View Slide

10. Building blocks
    go.mod go.sum vendor
    Optional
    ⚙
    Go runtime
    
    
    External
    packages
    

    View Slide

11. Testing is
    essential for
    fast feedback
    

    View Slide

12. Picking a test framework
    Standard tesFng package is built-in
    Eliminate asser5on logic with TesFfy
    Ginkgo supports wri5ng BDD-style tests
    hBps:/
    /bmuschko.com/blog/go-tes5ng-frameworks/
    
    

    View Slide

13. Capturing code coverage metrics
    $ go test ./... -coverprofile=coverage.txt -covermode=count
    ? github.com/bmuschko/letsgopher [no test files]
    ok github.com/bmuschko/letsgopher/cmd 0.047s↵
    coverage: 72.1% of statements
    ok github.com/bmuschko/letsgopher/template/archive 0.028s↵
    coverage: 69.6% of statements
    ok github.com/bmuschko/letsgopher/template/config 0.022s↵
    coverage: 94.1% of statements
    Vendor directory excluded with Go 1.9+
    Write metrics to coverage.txt for processing
    

    View Slide

14. Rendering HTML report
    $ go tool cover -html=coverage.txt
    

    View Slide

15. Codecov coverage visualizaFon
    $ bash <(curl -s https://codecov.io/bash) hBps:/
    /codecov.io
    
    

    View Slide

16. Demo
    

    View Slide

17. Code quality needs to be
    checked continuously
    

    View Slide

18. Typical concerns
    Enforcing coding style convenFons
    Detec5ng common mistakes
    Finding unused code
    Calcula5ng cyclomaFc complexiFes
    

    View Slide

19. Using GolangCI-Lint
    Aggregator for mul5ple tools
    Parallel execuFon of quality tools
    Free for Open Source projects
    hBps:/
    /github.com/golangci/golangci-lint
    
    

    View Slide

20. Example output
    $ golangci-lint run

    cmd/install.go:40:13: Error return value of `errors.New` is not↵
    checked (errcheck)

    errors.New("Currently templates can only be installed from↵
    a Git repository")

    ^

    templ/gen.go:22:15: Error return value of `utils.CopyDir` is not↵
    checked (errcheck)

    utils.CopyDir(templatePath, targetPath)

    ^

    cmd/install.go:38:11: composites: `github.com/bmuschko/↵
    lets-gopher-exercise/remote.GitRepo` composite literal uses↵
    unkeyed fields (govet)

    repo = &remote.GitRepo{repoUrl, templ.TemplateDir}

    ^
    

    View Slide

21. Demo
    

    View Slide

22. Cross-compile
    binaries for
    target
    platforms
    

    View Slide

23. Assembling binaries
    Executables for plaSorms & architectures
    Op5mized for performance & includes deps
    go build + GOOS and GOARCH env vars
    

    View Slide

24. OpFons for creaFng binaries
    Write a script to automate CLI invoca5ons
    Gox: Parallelize builds for mul5ple plaSorms
    GoReleaser: Binary crea5on and releasing
    

    View Slide

25. Using GoReleaser
    Defini5on in .goreleaser.yml
    Combina5on of OS and architecture
    Configurable binary crea5on and archives
    hBps:/
    /goreleaser.com
    
    

    View Slide

26. Building binaries
    $ goreleaser init
    • Generating .goreleaser.yml file
    • config created; please edit accordingly to your needs↵
    file=.goreleaser.yml
    $ goreleaser --snapshot --skip-publish --rm-dist
    • BUILDING BINARIES
    • building binary=dist/darwin_386
    • building binary=dist/linux_386
    • building binary=dist/linux_amd64
    • building binary=dist/darwin_amd64
    • ARCHIVES
    • creating archive=dist/_v0.0.0-next_Darwin_x86_64.tar.gz
    • creating archive=dist/_v0.0.0-next_Linux_i386.tar.gz
    • creating archive=dist/_v0.0.0-next_Linux_x86_64.tar.gz
    • creating archive=dist/_v0.0.0-next_Darwin_i386.tar.gz
    

    View Slide

27. InjecFng version into applicaFon
    builds:

    - ldflags: -s -w -X↵
    main.version={{.Version}}
    .goreleaser.yml
    package main

    

    import (

    "fmt"

    "github.com/bmuschko/letsgopher/cmd"

    "os"

    )

    

    var version = "undefined"

    

    func main() {

    cmd.SetVersion(version)

    }
    main.go
    

    View Slide

28. Demo
    

    View Slide

29. Releasing
    and hosting
    binaries
    

    View Slide

30. Release funcFonality
    Target plaRorm dependent on consumers
    Versioned binaries trackable to SCM tag
    Automa5c release notes genera5on
    

    View Slide

31. Publishing to GitHub/GitLab
    $ export GITHUB_TOKEN=`YOUR_GH_TOKEN`

    $ export GITLAB_TOKEN=`YOUR_GL_TOKEN`

    

    $ git tag -a v0.2.0 -m "First release"

    $ git push origin v0.2.0

    

    $ goreleaser
    

    View Slide

32. Publishing to ArFfactory
    release:

    disable: true

    artifactories:

    - name: production

    target: http://localhost:8081/artifactory/↵

    example-repo-local/{{ .ProjectName }}/↵
    {{ .Version }}/

    username: publisher
    .goreleaser.yml
    

    View Slide

33. CredenFals as env. variables
    ARTIFACTORY_
    PRODUCTION_
    USERNAME
    ARTIFACTORY_
    PRODUCTION_
    SECRET
    

    View Slide

34. Demo
    

    View Slide

35. Full process
    modeled as
    CI/CD pipeline
    

    View Slide

36. Different tooling opFons
    Travis CI: Hosted free and paid on-prem solu5on
    Jenkins: Free and commercial solu5on
    …and many others like Drone, Circle CI
    

    View Slide

37. Possible release strategies
    Manual release by pressing a buBon
    Automa5c release aka ConFnuous Deployment
    Release on tagging a SCM commit
    

    View Slide

38. Delivery pipeline steps
    Compile Test
    Code
    Analysis
    Release
    optional
    Commit
    

    View Slide

39. Different use cases
    Compile Release
    Non-tagged
    Commit
    Test
    Code
    Analysis
    ✗
    ✓
    ✓ ✓
    Compile Release
    Commit
    tagged v1.2.3
    Test
    Code
    Analysis
    ✓
    ✓ ✓ ✓
    

    View Slide

40. Pipeline as code
    .travis.yml
    Jenkinsfile
    Source code
    in SCM
    check in evaluate
    

    View Slide

41. language: go

    

    go:

    - 1.13.x

    

    env:

    - GO111MODULE=on

    

    install: true

    

    cache:

    directories:

    - $GOPATH/pkg/mod
    Basic setup
    

    View Slide

42. jobs:

    include:

    - stage: "Compile"

    name: "Compile Packages and Dependencies"

    script: go build

    - stage: "Tests"

    name: "Unit Tests"

    script: go test ./... -coverprofile=coverage.txt -covermode=count

    after_success:

    - bash <(curl -s https://codecov.io/bash)

    - stage: "Code Quality"

    name: "Code Quality Analysis"

    before_script:

    - curl -sfL https://install.goreleaser.com/github.com/golangci/↵
    golangci-lint.sh | bash -s -- -b $GOPATH/bin v1.18.0

    script: golangci-lint run
    Modeling stages
    

    View Slide

43. - stage: "Release"

    name: "Release Binaries"

    script: skip

    deploy:

    - provider: script

    skip_cleanup: true

    script: curl -sL https://git.io/goreleaser | bash

    on:

    tags: true

    condition: $TRAVIS_OS_NAME = linux
    Release binary for tagged commit
    

    View Slide

44. Pipeline visualizaFon
    jobs:

    include:

    - stage: ...

    - stage: ...

    - stage: ...
    

    View Slide

45. AutomaFc Go runFme installaFon
    Taken care of Jenkins Go plugin
    hBps:/
    /wiki.jenkins.io/display/JENKINS/Go+Plugin
    
    

    View Slide

46. pipeline {

    agent any

    tools {

    go 'go-1.13'

    }

    environment {

    GO111MODULE = 'on'

    }

    stages {

    ...

    }

    }
    Basic setup
    

    View Slide

47. stage('Compile') {

    steps {

    sh 'go build'

    }

    }

    stage('Test') {

    environment {

    CODECOV_TOKEN = credentials('codecov_token')

    }

    steps {

    sh 'go test ./... -coverprofile=coverage.txt'

    sh 'curl -s https://codecov.io/bash | bash -s -'

    }

    }

    stage('Code Analysis') {

    steps {

    sh 'curl -sfL https://install.goreleaser.com/github.com/golangci/↵
    golangci-lint.sh | bash -s -- -b $GOPATH/bin v1.18.0'

    sh 'golangci-lint run'

    }

    }
    Modeling stages
    

    View Slide

48. stage('Release') {

    when {

    buildingTag()

    }

    environment {

    GITHUB_TOKEN = credentials('github_token')

    }

    steps {

    sh 'curl -sL https://git.io/goreleaser | bash'

    }

    }
    Release binary for tagged commit
    

    View Slide

49. Pipeline visualizaFon
    Standard
    Blue Ocean
    

    View Slide

50. Demo
    

    View Slide

51. Thanks!
    Please ask quesFons.
    

    View Slide