Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Golang Project Layout and Practice

Bo-Yi Wu
August 29, 2019

Golang Project Layout and Practice

This talk I will show you the golang project layout and some best practice like as the following:
1. RESTful api and GraphQL
2. Model testing (Postgres, SQLite, MySQL)
3. Software Quality
4. Data Metrics
5. Go Testing

Bo-Yi Wu

August 29, 2019
Tweet

More Decks by Bo-Yi Wu

Other Decks in Technology

Transcript

  1. Go Project Layout
    and Practice
    Bo-Yi Wu
    2019.08.29
    ModernWeb

    View Slide

  2. About me
    • Software Engineer in Mediatek
    • Member of Drone CI/CD Platform
    • Member of Gitea Platform
    • Member of Gin Golang Framework
    • Teacher of Udemy Platform: Golang + Drone
    https://blog.wu-boy.com

    View Slide

  3. Agenda
    • Go in Mediatek
    • Go Project Layout
    • Go Practices
    • RESTful api and GraphQL
    • Model testing (Postgres, SQLite, MySQL)
    • Software Quality
    • Data Metrics
    • Go Testing

    View Slide

  4. Tech Stack
    • Initial Project using Go in 2018/01
    • Golang
    • Easy to Learn
    • Performance
    • Deployment

    View Slide

  5. Repository folder
    • api
    • assets
    • cmd
    • configs
    • docker
    • pkg
    ├── api
    ├── assets
    │ └── dist
    ├── cmd
    │ └── ggz
    ├── configs
    ├── docker
    │ ├── server
    └── pkg
    ├── config
    ├── errors
    ├── fixtures
    ├── helper
    ├── middleware
    │ ├── auth
    │ └── header
    ├── model
    ├── module
    │ ├── mailer
    │ ├── metrics
    │ └── storage
    ├── router
    │ └── routes
    ├── schema
    └── version

    View Slide

  6. Root folder
    • .drone.yml (deploy config)
    • .revive.toml (golint config)
    • docker-compose.yml (DB, Redis and UI)
    • Makefile
    • go module config (go.mod and go.sum)
    • .env.example

    View Slide

  7. Go Module
    https://blog.golang.org/using-go-modules

    View Slide

  8. Improve Deployment
    Using Go Module Proxy
    https://github.com/gomods/athens

    View Slide

  9. save time
    with proxy
    97s -> 6s

    View Slide

  10. Makefile
    Build, Testing, Deploy

    View Slide

  11. GOFMT ?= gofmt "-s"
    GO ?= go
    TARGETS ?= linux darwin windows
    ARCHS ?= amd64 386
    BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
    GOFILES := $(shell find . -name "*.go" -type f)
    TAGS ?= sqlite sqlite_unlock_notify
    ifneq ($(shell uname), Darwin)
    EXTLDFLAGS = -extldflags "-static" $(null)
    else
    EXTLDFLAGS =
    endif
    ifneq ($(DRONE_TAG),)
    VERSION ?= $(subst v,,$(DRONE_TAG))
    else
    VERSION ?= $(shell git describe --tags --always')
    endif

    View Slide

  12. .env

    View Slide

  13. GGZ_DB_DRIVER=mysql
    GGZ_DB_USERNAME=root
    GGZ_DB_PASSWORD=123456
    GGZ_DB_NAME=ggz
    GGZ_DB_HOST=127.0.0.1:3307
    GGZ_SERVER_ADDR=:8080
    GGZ_DEBUG=true
    GGZ_SERVER_HOST=http://localhost:8080
    GGZ_STORAGE_DRIVER=disk
    GGZ_MINIO_ACCESS_ID=xxxxxxxx
    GGZ_MINIO_SECRET_KEY=xxxxxxxx
    GGZ_MINIO_ENDPOINT=s3.example.com
    GGZ_MINIO_BUCKET=example
    GGZ_MINIO_SSL=true
    GGZ_AUTH0_DEBUG=true

    View Slide

  14. docker-compose.yml

    View Slide

  15. db:
    image: mysql
    restart: always
    volumes:
    - mysql-data:/var/lib/mysql
    environment:
    MYSQL_USER: example
    MYSQL_PASSWORD: example
    MYSQL_DATABASE: example
    MYSQL_ROOT_PASSWORD: example
    minio:
    image: minio/minio
    restart: always
    ports:
    volumes:
    - minio-data:/data
    environment:
    MINIO_ACCESS_KEY: minio123456
    MINIO_SECRET_KEY: minio123456
    command: server /data
    Development

    View Slide

  16. Production
    api:
    image: foo/bar
    restart: always
    ports:
    - 8080:8080
    environment:
    - GGZ_METRICS_TOKEN=test-prometheus-token
    - GGZ_METRICS_ENABLED=true
    labels:
    - "traefik.enable=true"
    - "traefik.basic.frontend.rule=Host:${WEB_HOST}"
    - "traefik.basic.protocol=http"

    View Slide

  17. Version
    Compile version info into Go binary

    View Slide

  18. Version
    • -X github.com/go-ggz/ggz/pkg/
    version.Version=$(VERSION)
    • -X github.com/go-ggz/ggz/pkg/
    version.BuildDate=$(BUILD_DATE)
    go build -o bin/api -ldflags

    View Slide

  19. var (
    // Version number for git tag.
    Version string
    // BuildDate is the ISO 8601 day drone was built.
    BuildDate string
    )
    // PrintCLIVersion print server info
    func PrintCLIVersion() string {
    return fmt.Sprintf(
    "version %s, built on %s, %s",
    Version,
    BuildDate,
    runtime.Version(),
    )
    }

    View Slide

  20. BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
    ifneq ($(DRONE_TAG),)
    VERSION ?= $(subst v,,$(DRONE_TAG))
    else
    VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed
    's/^v//')
    endif

    View Slide

  21. Assets
    Embed files in Go
    https://github.com/UnnoTed/fileb0x

    View Slide

  22. func ReadSource(origPath string) (content []byte, err error) {
    content, err = ReadFile(origPath)
    if err != nil {
    log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath)
    }
    if config.Server.Assets != "" && file.IsDir(config.Server.Assets) {
    origPath = path.Join(config.Server.Assets, origPath)
    if file.IsFile(origPath) {
    content, err = ioutil.ReadFile(origPath)
    if err != nil {
    log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath)
    }
    }
    }
    return content, err
    }
    Debug Setting

    View Slide

  23. // ViewHandler support dist handler from UI
    func ViewHandler() gin.HandlerFunc {
    fileServer := http.FileServer(dist.HTTP)
    data := []byte(time.Now().String())
    etag := fmt.Sprintf("%x", md5.Sum(data))
    return func(c *gin.Context) {
    c.Header("Cache-Control", "public, max-age=31536000")
    c.Header("ETag", etag)
    if match := c.GetHeader("If-None-Match"); match != "" {
    if strings.Contains(match, etag) {
    c.Status(http.StatusNotModified)
    return
    }
    }
    fileServer.ServeHTTP(c.Writer, c.Request)
    }
    }
    File Server Handler

    View Slide

  24. ᅷยိݯɿIUUQTEFWFMPQFSTHPPHMFDPNXFCGVOEBNFOUBMTQFSGPSNBODF
    PQUJNJ[JOHDPOUFOUF⒏DJFODZIUUQDBDIJOH IM[IUX

    View Slide

  25. // Favicon represents the favicon.
    func Favicon(c *gin.Context) {
    file, _ := dist.ReadFile("favicon.ico")
    etag := fmt.Sprintf("%x", md5.Sum(file))
    c.Header("ETag", etag)
    c.Header("Cache-Control", "max-age=0")
    if match := c.GetHeader("If-None-Match"); match != "" {
    if strings.Contains(match, etag) {
    c.Status(http.StatusNotModified)
    return
    }
    }
    c.Data(
    http.StatusOK,
    "image/x-icon",
    file,
    )
    }
    NO Cache

    View Slide

  26. API

    View Slide

  27. /healthz
    • health check for load balancer
    func Heartbeat(c *gin.Context) {
    c.AbortWithStatus(http.StatusOK)
    c.String(http.StatusOK, "ok")
    }

    View Slide

  28. CMD
    Command line

    View Slide

  29. Command line package
    • Golang package: flag
    • urfave/cli
    • spf13/cobra

    View Slide

  30. ├── agent
    │ ├── config
    │ │ └── config.go
    │ └── main.go
    ├── notify
    │ └── main.go
    └── tcp-server
    ├── config
    │ └── config.go
    └── main.go

    View Slide

  31. Config
    Management
    github.com/spf13/viper

    View Slide

  32. Config management
    • Load config from File
    • .json
    • .ini
    • Load config from Environment Variables
    • .env

    View Slide

  33. var envfile string
    flag.StringVar(&envfile, "env-file", ".env", "Read in a file of environment
    variables")
    flag.Parse()
    godotenv.Load(envfile)
    _ "github.com/joho/godotenv/autoload"

    View Slide

  34. Logging struct {
    Debug bool `envconfig:"GGZ_LOGS_DEBUG"`
    Level string `envconfig:"GGZ_LOGS_LEVEL" default:"info"`
    Color bool `envconfig:"GGZ_LOGS_COLOR"`
    Pretty bool `envconfig:"GGZ_LOGS_PRETTY"`
    Text bool `envconfig:"GGZ_LOGS_TEXT"`
    }
    // Server provides the server configuration.
    Server struct {
    Addr string `envconfig:"GGZ_SERVER_ADDR"`
    Port string `envconfig:"GGZ_SERVER_PORT" default:"12000"`
    Path string `envconfig:”GGZ_SERVER_PATH" default:"data"`
    }
    github.com/kelseyhightower/envconfig

    View Slide

  35. config, err := config.Environ()
    if err != nil {
    log.Fatal().
    Err(err).
    Msg("invalid configuration")
    }
    initLogging(config)
    // check folder exist
    if !file.IsDir(config.Server.Path) {
    log.Fatal().
    Str("path", config.Server.Path).
    Msg("log folder not found")
    }
    Load env from structure

    View Slide

  36. /configs
    Configuration file templates or default config

    View Slide

  37. global:
    scrape_interval: 5s
    external_labels:
    monitor: 'my-monitor'
    scrape_configs:
    - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
    - job_name: 'ggz-server'
    static_configs:
    - targets: ['ggz-server:8080']
    bearer_token: 'test-prometheus-token'

    View Slide

  38. /docker
    Docker file template

    View Slide

  39. ├── ggz-redirect
    │ ├── Dockerfile.linux.amd64
    │ ├── Dockerfile.linux.arm
    │ ├── Dockerfile.linux.arm64
    │ ├── Dockerfile.windows.amd64
    │ └── manifest.tmpl
    └── ggz-server
    ├── Dockerfile.linux.amd64
    ├── Dockerfile.linux.arm
    ├── Dockerfile.linux.arm64
    ├── Dockerfile.windows.amd64
    └── manifest.tmpl

    View Slide

  40. /integrations

    View Slide

  41. ctx := context.Background()
    req := testcontainers.ContainerRequest{
    Image: "goggz/ggz-server",
    ExposedPorts: []string{"8080/tcp"},
    WaitingFor: wait.ForLog("Starting shorten server on :8080")
    }
    ggzServer, err := testcontainers.GenericContainer(
    ctx,
    testcontainers.GenericContainerRequest{
    ContainerRequest: req,
    Started: true,
    })
    if err != nil {
    t.Fatal(err)
    }
    github.com/testcontainers/testcontainers-go

    View Slide

  42. /pkg

    View Slide

  43. ├── config
    ├── errors
    ├── fixtures
    ├── helper
    ├── middleware
    │ ├── auth
    │ └── header
    ├── model
    ├── module
    │ ├── metrics
    │ └── storage
    │ ├── disk
    │ └── minio
    ├── router
    │ └── routes
    ├── schema
    └── version

    View Slide

  44. /pkg/errors

    View Slide

  45. // Type defines the type of an error
    type Type string
    const (
    // Internal error
    Internal Type = "internal"
    // NotFound error means that a specific item does not exis
    NotFound Type = "not_found"
    // BadRequest error
    BadRequest Type = "bad_request"
    // Validation error
    Validation Type = "validation"
    // AlreadyExists error
    AlreadyExists Type = "already_exists"
    // Unauthorized error
    Unauthorized Type = "unauthorized"
    )

    View Slide

  46. // ENotExists creates an error of type NotExist
    func ENotExists(msg string, err error, arg ...interface{}) error {
    return New(NotFound, fmt.Sprintf(msg, arg...), err)
    }
    // EBadRequest creates an error of type BadRequest
    func EBadRequest(msg string, err error, arg ...interface{}) error {
    return New(BadRequest, fmt.Sprintf(msg, arg...), err)
    }
    // EAlreadyExists creates an error of type AlreadyExists
    func EAlreadyExists(msg string, err error, arg ...interface{}) error {
    return New(AlreadyExists, fmt.Sprintf(msg, arg...), err)
    }

    View Slide

  47. /pkg/fixtures

    View Slide

  48. Rails-like test fixtures
    Write tests against a real database
    github.com/go-testfixtures/testfixtures

    View Slide

  49. fixtures/
    posts.yml
    comments.yml
    tags.yml
    posts_tags.yml
    users.yml

    View Slide

  50. -
    id: 1
    email: [email protected]
    full_name: test
    avatar: http://example.com
    avatar_email: [email protected]
    -
    id: 2
    email: [email protected]
    full_name: test1234
    avatar: http://example.com
    avatar_email: [email protected]

    View Slide

  51. Unit Testing with Database

    View Slide

  52. func TestMain(m *testing.M) {
    // test program to do extra
    setup or teardown before or after
    testing.
    os.Exit(m.Run())
    }
    https://golang.org/pkg/testing/#hdr-Main

    View Slide

  53. func MainTest(m *testing.M, pathToRoot string) {
    var err error
    fixturesDir := filepath.Join(pathToRoot, "pkg", "fixtures")
    if err = createTestEngine(fixturesDir); err != nil {
    fatalTestError("Error creating test engine: %v\n", err)
    }
    os.Exit(m.Run())
    }
    func createTestEngine(fixturesDir string) error {
    var err error
    x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared")
    if err != nil {
    return err
    }
    x.ShowSQL(config.Server.Debug)
    return InitFixtures(&testfixtures.SQLite{}, fixturesDir)
    }
    Testing with SQLite

    View Slide

  54. func TestIsUserExist(t *testing.T) {
    assert.NoError(t, PrepareTestDatabase())
    exists, err := IsUserExist(0, "[email protected]")
    assert.NoError(t, err)
    assert.True(t, exists)
    exists, err = IsUserExist(0, "[email protected]")
    assert.NoError(t, err)
    assert.False(t, exists)
    exists, err = IsUserExist(1, "[email protected]")
    assert.NoError(t, err)
    assert.True(t, exists)
    exists, err = IsUserExist(1, "[email protected]")
    assert.NoError(t, err)
    assert.False(t, exists)
    } go test -v -run=TestIsUserExist ./pkg/models/

    View Slide

  55. /pkg/helper

    View Slide

  56. Helper func
    • Encrypt and Decrypt
    • Regexp func
    • IsEmail, IsUsername
    • Zipfile

    View Slide

  57. /pkg/middleware

    View Slide

  58. func Secure(c *gin.Context) {
    c.Header("Access-Control-Allow-Origin", "*")
    c.Header("X-Frame-Options", "DENY")
    c.Header("X-Content-Type-Options", "nosniff")
    c.Header("X-XSS-Protection", "1; mode=block")
    if c.Request.TLS != nil {
    c.Header("Strict-Transport-Security", "max-age=31536000")
    }
    }

    View Slide

  59. /pkg/model

    View Slide

  60. Use gorm or xorm

    View Slide

  61. Build in
    SQLite3

    View Slide

  62. // +build sqlite
    package model
    import (
    _ "github.com/mattn/go-sqlite3"
    )
    func init() {
    EnableSQLite3 = true
    }
    go build -v -tags 'sqlite sqlite_unlock_notify'

    View Slide

  63. /pkg/module

    View Slide

  64. ├── cron
    ├── download
    ├── jwt
    ├── ldap
    ├── mailer
    │ ├── ses
    │ └── smtp
    ├── queue
    ├── metrics
    ├── redis
    └── storage
    ├── disk
    └── minio

    View Slide

  65. Integration with
    Prometheus + Grafana

    View Slide

  66. func NewCollector() Collector {
    return Collector{
    Users: prometheus.NewDesc(
    namespace+"users",
    "Number of Users",
    nil, nil,
    ),
    }
    // Collect returns the metrics with values
    func (c Collector) Collect(ch chanstats := model.GetStatistic()
    ch c.Users,
    prometheus.GaugeValue,
    float64(stats.Counter.User),
    )
    }

    View Slide

  67. Prometheus Handler

    View Slide

  68. func Metrics(token string) gin.HandlerFunc {
    h := promhttp.Handler()
    return func(c *gin.Context) {
    if token == "" {
    h.ServeHTTP(c.Writer, c.Request)
    return
    }
    header := c.Request.Header.Get("Authorization")
    if header == "" {
    c.String(http.StatusUnauthorized, errInvalidToken.Error())
    return
    }
    bearer := fmt.Sprintf("Bearer %s", token)
    if header != bearer {
    c.String(http.StatusUnauthorized, errInvalidToken.Error())
    return
    }
    h.ServeHTTP(c.Writer, c.Request)
    }
    }

    View Slide

  69. c := metrics.NewCollector()
    prometheus.MustRegister(c)
    if config.Metrics.Enabled {
    root.GET("/metrics", router.Metrics(config.Metrics.Token))
    }
    Your prometheus token

    View Slide

  70. /pkg/schema

    View Slide

  71. RESTful vs GraphQL
    See the Slide: GraphQL in Go

    View Slide

  72. var rootQuery = graphql.NewObject(
    graphql.ObjectConfig{
    Name: "RootQuery",
    Description: "Root Query",
    Fields: graphql.Fields{
    "queryShortenURL": &queryShortenURL,
    "queryMe": &queryMe,
    },
    })
    var rootMutation = graphql.NewObject(
    graphql.ObjectConfig{
    Name: "RootMutation",
    Description: "Root Mutation",
    Fields: graphql.Fields{
    "createUser": &createUser,
    },
    })
    // Schema is the GraphQL schema served by the server.
    var Schema, _ = graphql.NewSchema(
    graphql.SchemaConfig{
    Query: rootQuery,
    Mutation: rootMutation,
    })

    View Slide

  73. Write the GraphQL Testing

    View Slide

  74. assert.NoError(t, model.PrepareTestDatabase())
    t.Run("user not login", func(t *testing.T) {
    test := T{
    Query: `{
    queryMe {
    email
    }
    }`,
    Schema: Schema,
    Expected: &graphql.Result{
    Data: map[string]interface{}{
    "queryMe": nil,
    },
    Errors: []gqlerrors.FormattedError{
    {
    Message: errorYouAreNotLogin,
    },
    },
    },
    }
    })
    }

    View Slide

  75. Best Practice
    Testing your Go code

    View Slide

  76. View Slide

  77. Testable Code
    • Code Quality
    • Readability
    • Maintainability
    • Testability

    View Slide

  78. #1. Testing in Go
    func TestFooBar(t *testing.T) {}
    func ExampleFooBar(t *testing.T) {}
    func BenchmarkFooBar(b *testing.B) {}
    go test package_name

    View Slide

  79. #2. Benchmark Testing
    Profiling: CPU, Memory, Goroutine Block

    View Slide

  80. func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) {
    for i := 0; i < b.N; i++ {
    context := map[string]interface{}{}
    variables := map[string]interface{}{}
    playlyfeExecutor.Execute(context, "{hello}", variables, "")
    }
    }
    func BenchmarkGophersGraphQLMaster(b *testing.B) {
    for i := 0; i < b.N; i++ {
    ctx := context.Background()
    variables := map[string]interface{}{}
    gopherSchema.Exec(ctx, "{hello}", "", variables)
    }
    } http://bit.ly/2L0CG3Q

    View Slide

  81. View Slide

  82. #3. Example Testing
    Examples on how to use your code

    View Slide

  83. func ExampleFooBar() {
    fmt.Println(strings.Compare("a", "b"))
    fmt.Println(strings.Compare("a", "a"))
    fmt.Println(strings.Compare("b", "a"))
    // Output:
    // -1
    // 0
    // 1
    }

    View Slide

  84. $ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/...
    === RUN ExampleFooBar
    --- PASS: ExampleFooBar (0.00s)
    PASS
    ok github.com/go-ggz/ggz/pkg/model 0.022s

    View Slide

  85. #4. Subtests in Testing Package
    func (t *T) Run(name string, f func(t *T)) bool {}
    func (b *B) Run(name string, f func(b *B)) bool {}

    View Slide

  86. tests := []struct {
    name string
    fields fields
    args args
    }{}
    for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
    c := Collector{
    Shortens: tt.fields.Shortens,
    Users: tt.fields.Users,
    }
    c.Describe(tt.args.ch)
    })
    }
    }

    View Slide

  87. #5. Skipping Testing
    t.Skip()

    View Slide

  88. package metrics
    import (
    "os"
    "testing"
    )
    func TestSkip(t *testing.T) {
    if os.Getenv("DEBUG_MODE") == "true" {
    t.Skipf("test skipped")
    }
    }

    View Slide

  89. #6. Running Tests in Parallel
    Speedup your CI/CD Flow
    t.Parallel()

    View Slide

  90. func TestFooBar01(t *testing.T) {
    t.Parallel()
    time.Sleep(time.Second)
    }
    func TestFooBar02(t *testing.T) {
    t.Parallel()
    time.Sleep(time.Second * 2)
    }
    func TestFooBar03(t *testing.T) {
    t.Parallel()
    time.Sleep(time.Second * 3)
    }

    View Slide

  91. Just only use
    one package
    github.com/stretchr/testify

    View Slide

  92. https://www.udemy.com/course/golang-fight/?couponCode=GOLANG2019

    View Slide

  93. https://www.udemy.com/course/devops-oneday/?couponCode=DRONE2019

    View Slide

  94. END

    View Slide