Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fully automated release processes for Go projects

Fully automated release processes for Go projects

Go projects need to rely on an automated process that ushers changes through all aspects of a delivery pipeline. A typical pipeline involves resolving packages, compilation, testing, generation of code coverage, static code analysis, binary creation, and the actual release of the binaries.

This session discusses automation aspects and tools helpful for streamlining Go projects. First, you’ll get to know individual tools and why they are so useful. Then you’ll tie it all together to form a simple delivery pipeline for Go projects powered by Travis CI or Jenkins. You will leave the session with a recipe for implementing a sensible, yet simple end-to-end automation approach for your own Go projects to ensure reliable and reproducible builds.

Benjamin Muschko

September 16, 2019
Tweet

More Decks by Benjamin Muschko

Other Decks in Programming

Transcript

  1. Fully automated
    release processes
    for projects
    Benjamin Muschko

    View Slide

  2. AUTOMATED
    ASCENT
    bmuschko
    bmuschko
    bmuschko.com
    About the speaker
    automatedascent.com

    View Slide

  3. Who’s using the
    language Go?

    View Slide

  4. Typical development tasks
    Package
    Management
    Source Code
    Analysis
    Testing
    Building
    Binaries
    Publishing
    Binaries
    CI/CD
    Pipeline

    View Slide

  5. Do I need a build tool?

    View Slide

  6. Go Modules
    The de facto
    packager
    manager

    View Slide

  7. High-level usage
    Enabled via env. variable GO111MODULE
    Check in generated go.mod, go.sum
    Direct integra5on with go get
    Will try to download from central proxy first

    View Slide

  8. SemanFc versioning
    2 . 13 . 5
    hBps:/
    /semver.org/

    Major Minor Patch
    (Breaking) (Feature) (Fix)

    View Slide

  9. Dependency declaraFon
    module github.com/bmuschko/letsgopher
    require (
    github.com/Flaque/filet v0.0.0-20190209224823-fc4d33cfcf93
    github.com/blang/semver v3.5.1+incompatible
    github.com/ghodss/yaml v1.0.0
    github.com/gosuri/uitable v0.0.0-20160404203958-36ee7e946282
    github.com/inconshreveable/mousetrap v1.0.0 // indirect
    github.com/kr/text v0.1.0
    github.com/mattn/go-runewidth v0.0.4 // indirect
    github.com/mitchellh/go-homedir v1.0.0
    github.com/spf13/afero v1.2.2 // indirect
    github.com/spf13/cobra v0.0.3
    github.com/spf13/pflag v1.0.3 // indirect
    github.com/stretchr/testify v1.3.0
    gopkg.in/AlecAivazis/survey.v1 v1.8.2
    gopkg.in/yaml.v2 v2.2.2 // indirect
    )
    go 1.13

    View Slide

  10. Building blocks
    go.mod go.sum vendor
    Optional

    Go runtime


    External
    packages

    View Slide

  11. Testing is
    essential for
    fast feedback

    View Slide

  12. Picking a test framework
    Standard tesFng package is built-in
    Eliminate asser5on logic with TesFfy
    Ginkgo supports wri5ng BDD-style tests
    hBps:/
    /bmuschko.com/blog/go-tes5ng-frameworks/

    View Slide

  13. Capturing code coverage metrics
    $ go test ./... -coverprofile=coverage.txt -covermode=count
    ? github.com/bmuschko/letsgopher [no test files]
    ok github.com/bmuschko/letsgopher/cmd 0.047s↵
    coverage: 72.1% of statements
    ok github.com/bmuschko/letsgopher/template/archive 0.028s↵
    coverage: 69.6% of statements
    ok github.com/bmuschko/letsgopher/template/config 0.022s↵
    coverage: 94.1% of statements
    Vendor directory excluded with Go 1.9+
    Write metrics to coverage.txt for processing

    View Slide

  14. Rendering HTML report
    $ go tool cover -html=coverage.txt

    View Slide

  15. Codecov coverage visualizaFon
    $ bash <(curl -s https://codecov.io/bash) hBps:/
    /codecov.io

    View Slide

  16. Demo

    View Slide

  17. Code quality needs to be
    checked continuously

    View Slide

  18. Typical concerns
    Enforcing coding style convenFons
    Detec5ng common mistakes
    Finding unused code
    Calcula5ng cyclomaFc complexiFes

    View Slide

  19. Using GolangCI-Lint
    Aggregator for mul5ple tools
    Parallel execuFon of quality tools
    Free for Open Source projects
    hBps:/
    /github.com/golangci/golangci-lint

    View Slide

  20. Example output
    $ golangci-lint run

    cmd/install.go:40:13: Error return value of `errors.New` is not↵
    checked (errcheck)

    errors.New("Currently templates can only be installed from↵
    a Git repository")

    ^

    templ/gen.go:22:15: Error return value of `utils.CopyDir` is not↵
    checked (errcheck)

    utils.CopyDir(templatePath, targetPath)

    ^

    cmd/install.go:38:11: composites: `github.com/bmuschko/↵
    lets-gopher-exercise/remote.GitRepo` composite literal uses↵
    unkeyed fields (govet)

    repo = &remote.GitRepo{repoUrl, templ.TemplateDir}

    ^

    View Slide

  21. Demo

    View Slide

  22. Cross-compile
    binaries for
    target
    platforms

    View Slide

  23. Assembling binaries
    Executables for plaSorms & architectures
    Op5mized for performance & includes deps
    go build + GOOS and GOARCH env vars

    View Slide

  24. OpFons for creaFng binaries
    Write a script to automate CLI invoca5ons
    Gox: Parallelize builds for mul5ple plaSorms
    GoReleaser: Binary crea5on and releasing

    View Slide

  25. Using GoReleaser
    Defini5on in .goreleaser.yml
    Combina5on of OS and architecture
    Configurable binary crea5on and archives
    hBps:/
    /goreleaser.com

    View Slide

  26. Building binaries
    $ goreleaser init
    • Generating .goreleaser.yml file
    • config created; please edit accordingly to your needs↵
    file=.goreleaser.yml
    $ goreleaser --snapshot --skip-publish --rm-dist
    • BUILDING BINARIES
    • building binary=dist/darwin_386
    • building binary=dist/linux_386
    • building binary=dist/linux_amd64
    • building binary=dist/darwin_amd64
    • ARCHIVES
    • creating archive=dist/_v0.0.0-next_Darwin_x86_64.tar.gz
    • creating archive=dist/_v0.0.0-next_Linux_i386.tar.gz
    • creating archive=dist/_v0.0.0-next_Linux_x86_64.tar.gz
    • creating archive=dist/_v0.0.0-next_Darwin_i386.tar.gz

    View Slide

  27. InjecFng version into applicaFon
    builds:

    - ldflags: -s -w -X↵
    main.version={{.Version}}
    .goreleaser.yml
    package main


    import (

    "fmt"

    "github.com/bmuschko/letsgopher/cmd"

    "os"

    )


    var version = "undefined"


    func main() {

    cmd.SetVersion(version)

    }
    main.go

    View Slide

  28. Demo

    View Slide

  29. Releasing
    and hosting
    binaries

    View Slide

  30. Release funcFonality
    Target plaRorm dependent on consumers
    Versioned binaries trackable to SCM tag
    Automa5c release notes genera5on

    View Slide

  31. Publishing to GitHub/GitLab
    $ export GITHUB_TOKEN=`YOUR_GH_TOKEN`

    $ export GITLAB_TOKEN=`YOUR_GL_TOKEN`


    $ git tag -a v0.2.0 -m "First release"

    $ git push origin v0.2.0


    $ goreleaser

    View Slide

  32. Publishing to ArFfactory
    release:

    disable: true

    artifactories:

    - name: production

    target: http://localhost:8081/artifactory/↵

    example-repo-local/{{ .ProjectName }}/↵
    {{ .Version }}/

    username: publisher
    .goreleaser.yml

    View Slide

  33. CredenFals as env. variables
    ARTIFACTORY_
    PRODUCTION_
    USERNAME
    ARTIFACTORY_
    PRODUCTION_
    SECRET

    View Slide

  34. Demo

    View Slide

  35. Full process
    modeled as
    CI/CD pipeline

    View Slide

  36. Different tooling opFons
    Travis CI: Hosted free and paid on-prem solu5on
    Jenkins: Free and commercial solu5on
    …and many others like Drone, Circle CI

    View Slide

  37. Possible release strategies
    Manual release by pressing a buBon
    Automa5c release aka ConFnuous Deployment
    Release on tagging a SCM commit

    View Slide

  38. Delivery pipeline steps
    Compile Test
    Code
    Analysis
    Release
    optional
    Commit

    View Slide

  39. Different use cases
    Compile Release
    Non-tagged
    Commit
    Test
    Code
    Analysis


    ✓ ✓
    Compile Release
    Commit
    tagged v1.2.3
    Test
    Code
    Analysis

    ✓ ✓ ✓

    View Slide

  40. Pipeline as code
    .travis.yml
    Jenkinsfile
    Source code
    in SCM
    check in evaluate

    View Slide

  41. language: go


    go:

    - 1.13.x


    env:

    - GO111MODULE=on


    install: true


    cache:

    directories:

    - $GOPATH/pkg/mod
    Basic setup

    View Slide

  42. jobs:

    include:

    - stage: "Compile"

    name: "Compile Packages and Dependencies"

    script: go build

    - stage: "Tests"

    name: "Unit Tests"

    script: go test ./... -coverprofile=coverage.txt -covermode=count

    after_success:

    - bash <(curl -s https://codecov.io/bash)

    - stage: "Code Quality"

    name: "Code Quality Analysis"

    before_script:

    - curl -sfL https://install.goreleaser.com/github.com/golangci/↵
    golangci-lint.sh | bash -s -- -b $GOPATH/bin v1.18.0

    script: golangci-lint run
    Modeling stages

    View Slide

  43. - stage: "Release"

    name: "Release Binaries"

    script: skip

    deploy:

    - provider: script

    skip_cleanup: true

    script: curl -sL https://git.io/goreleaser | bash

    on:

    tags: true

    condition: $TRAVIS_OS_NAME = linux
    Release binary for tagged commit

    View Slide

  44. Pipeline visualizaFon
    jobs:

    include:

    - stage: ...

    - stage: ...

    - stage: ...

    View Slide

  45. AutomaFc Go runFme installaFon
    Taken care of Jenkins Go plugin
    hBps:/
    /wiki.jenkins.io/display/JENKINS/Go+Plugin

    View Slide

  46. pipeline {

    agent any

    tools {

    go 'go-1.13'

    }

    environment {

    GO111MODULE = 'on'

    }

    stages {

    ...

    }

    }
    Basic setup

    View Slide

  47. stage('Compile') {

    steps {

    sh 'go build'

    }

    }

    stage('Test') {

    environment {

    CODECOV_TOKEN = credentials('codecov_token')

    }

    steps {

    sh 'go test ./... -coverprofile=coverage.txt'

    sh 'curl -s https://codecov.io/bash | bash -s -'

    }

    }

    stage('Code Analysis') {

    steps {

    sh 'curl -sfL https://install.goreleaser.com/github.com/golangci/↵
    golangci-lint.sh | bash -s -- -b $GOPATH/bin v1.18.0'

    sh 'golangci-lint run'

    }

    }
    Modeling stages

    View Slide

  48. stage('Release') {

    when {

    buildingTag()

    }

    environment {

    GITHUB_TOKEN = credentials('github_token')

    }

    steps {

    sh 'curl -sL https://git.io/goreleaser | bash'

    }

    }
    Release binary for tagged commit

    View Slide

  49. Pipeline visualizaFon
    Standard
    Blue Ocean

    View Slide

  50. Demo

    View Slide

  51. Thanks!
    Please ask quesFons.

    View Slide