Golang Project Layout and Practice

This talk I will show you the golang project layout and some best practice like as the following:
1. RESTful api and GraphQL
2. Model testing (Postgres, SQLite, MySQL)
3. Software Quality
4. Data Metrics
5. Go Testing

Transcript

1. Go Project Layout
   and Practice
   Bo-Yi Wu
   2019.08.29
   ModernWeb
   

   View Slide

2. About me
   • Software Engineer in Mediatek
   • Member of Drone CI/CD Platform
   • Member of Gitea Platform
   • Member of Gin Golang Framework
   • Teacher of Udemy Platform: Golang + Drone
   https://blog.wu-boy.com
   

   View Slide

3. Agenda
   • Go in Mediatek
   • Go Project Layout
   • Go Practices
   • RESTful api and GraphQL
   • Model testing (Postgres, SQLite, MySQL)
   • Software Quality
   • Data Metrics
   • Go Testing
   

   View Slide

4. Tech Stack
   • Initial Project using Go in 2018/01
   • Golang
   • Easy to Learn
   • Performance
   • Deployment
   

   View Slide

5. Repository folder
   • api
   • assets
   • cmd
   • configs
   • docker
   • pkg
   ├── api
   ├── assets
   │ └── dist
   ├── cmd
   │ └── ggz
   ├── configs
   ├── docker
   │ ├── server
   └── pkg
   ├── config
   ├── errors
   ├── fixtures
   ├── helper
   ├── middleware
   │ ├── auth
   │ └── header
   ├── model
   ├── module
   │ ├── mailer
   │ ├── metrics
   │ └── storage
   ├── router
   │ └── routes
   ├── schema
   └── version
   

   View Slide

6. Root folder
   • .drone.yml (deploy config)
   • .revive.toml (golint config)
   • docker-compose.yml (DB, Redis and UI)
   • Makefile
   • go module config (go.mod and go.sum)
   • .env.example
   

   View Slide

7. Go Module
   https://blog.golang.org/using-go-modules
   

   View Slide

8. Improve Deployment
   Using Go Module Proxy
   https://github.com/gomods/athens
   

   View Slide

9. save time
   with proxy
   97s -> 6s
   

   View Slide

10. Makefile
    Build, Testing, Deploy
    

    View Slide

11. GOFMT ?= gofmt "-s"
    GO ?= go
    TARGETS ?= linux darwin windows
    ARCHS ?= amd64 386
    BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
    GOFILES := $(shell find . -name "*.go" -type f)
    TAGS ?= sqlite sqlite_unlock_notify
    ifneq ($(shell uname), Darwin)
    EXTLDFLAGS = -extldflags "-static" $(null)
    else
    EXTLDFLAGS =
    endif
    ifneq ($(DRONE_TAG),)
    VERSION ?= $(subst v,,$(DRONE_TAG))
    else
    VERSION ?= $(shell git describe --tags --always')
    endif
    

    View Slide

12. .env
    

    View Slide

13. GGZ_DB_DRIVER=mysql
    GGZ_DB_USERNAME=root
    GGZ_DB_PASSWORD=123456
    GGZ_DB_NAME=ggz
    GGZ_DB_HOST=127.0.0.1:3307
    GGZ_SERVER_ADDR=:8080
    GGZ_DEBUG=true
    GGZ_SERVER_HOST=http://localhost:8080
    GGZ_STORAGE_DRIVER=disk
    GGZ_MINIO_ACCESS_ID=xxxxxxxx
    GGZ_MINIO_SECRET_KEY=xxxxxxxx
    GGZ_MINIO_ENDPOINT=s3.example.com
    GGZ_MINIO_BUCKET=example
    GGZ_MINIO_SSL=true
    GGZ_AUTH0_DEBUG=true
    

    View Slide

14. docker-compose.yml
    

    View Slide

15. db:
    image: mysql
    restart: always
    volumes:
    - mysql-data:/var/lib/mysql
    environment:
    MYSQL_USER: example
    MYSQL_PASSWORD: example
    MYSQL_DATABASE: example
    MYSQL_ROOT_PASSWORD: example
    minio:
    image: minio/minio
    restart: always
    ports:
    volumes:
    - minio-data:/data
    environment:
    MINIO_ACCESS_KEY: minio123456
    MINIO_SECRET_KEY: minio123456
    command: server /data
    Development
    

    View Slide

16. Production
    api:
    image: foo/bar
    restart: always
    ports:
    - 8080:8080
    environment:
    - GGZ_METRICS_TOKEN=test-prometheus-token
    - GGZ_METRICS_ENABLED=true
    labels:
    - "traefik.enable=true"
    - "traefik.basic.frontend.rule=Host:${WEB_HOST}"
    - "traefik.basic.protocol=http"
    

    View Slide

17. Version
    Compile version info into Go binary
    

    View Slide

18. Version
    • -X github.com/go-ggz/ggz/pkg/
    version.Version=$(VERSION)
    • -X github.com/go-ggz/ggz/pkg/
    version.BuildDate=$(BUILD_DATE)
    go build -o bin/api -ldflags
    

    View Slide

19. var (
    // Version number for git tag.
    Version string
    // BuildDate is the ISO 8601 day drone was built.
    BuildDate string
    )
    // PrintCLIVersion print server info
    func PrintCLIVersion() string {
    return fmt.Sprintf(
    "version %s, built on %s, %s",
    Version,
    BuildDate,
    runtime.Version(),
    )
    }
    

    View Slide

20. BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
    ifneq ($(DRONE_TAG),)
    VERSION ?= $(subst v,,$(DRONE_TAG))
    else
    VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed
    's/^v//')
    endif
    

    View Slide

21. Assets
    Embed files in Go
    https://github.com/UnnoTed/fileb0x
    

    View Slide

22. func ReadSource(origPath string) (content []byte, err error) {
    content, err = ReadFile(origPath)
    if err != nil {
    log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath)
    }
    if config.Server.Assets != "" && file.IsDir(config.Server.Assets) {
    origPath = path.Join(config.Server.Assets, origPath)
    if file.IsFile(origPath) {
    content, err = ioutil.ReadFile(origPath)
    if err != nil {
    log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath)
    }
    }
    }
    return content, err
    }
    Debug Setting
    

    View Slide

23. // ViewHandler support dist handler from UI
    func ViewHandler() gin.HandlerFunc {
    fileServer := http.FileServer(dist.HTTP)
    data := []byte(time.Now().String())
    etag := fmt.Sprintf("%x", md5.Sum(data))
    return func(c *gin.Context) {
    c.Header("Cache-Control", "public, max-age=31536000")
    c.Header("ETag", etag)
    if match := c.GetHeader("If-None-Match"); match != "" {
    if strings.Contains(match, etag) {
    c.Status(http.StatusNotModified)
    return
    }
    }
    fileServer.ServeHTTP(c.Writer, c.Request)
    }
    }
    File Server Handler
    

    View Slide

24. ᅷยိݯɿIUUQTEFWFMPQFSTHPPHMFDPNXFCGVOEBNFOUBMTQFSGPSNBODF
    PQUJNJ[JOHDPOUFOUF⒏DJFODZIUUQDBDIJOH IM[IUX
    

    View Slide

25. // Favicon represents the favicon.
    func Favicon(c *gin.Context) {
    file, _ := dist.ReadFile("favicon.ico")
    etag := fmt.Sprintf("%x", md5.Sum(file))
    c.Header("ETag", etag)
    c.Header("Cache-Control", "max-age=0")
    if match := c.GetHeader("If-None-Match"); match != "" {
    if strings.Contains(match, etag) {
    c.Status(http.StatusNotModified)
    return
    }
    }
    c.Data(
    http.StatusOK,
    "image/x-icon",
    file,
    )
    }
    NO Cache
    

    View Slide

26. API
    

    View Slide

27. /healthz
    • health check for load balancer
    func Heartbeat(c *gin.Context) {
    c.AbortWithStatus(http.StatusOK)
    c.String(http.StatusOK, "ok")
    }
    

    View Slide

28. CMD
    Command line
    

    View Slide

29. Command line package
    • Golang package: flag
    • urfave/cli
    • spf13/cobra
    

    View Slide

30. ├── agent
    │ ├── config
    │ │ └── config.go
    │ └── main.go
    ├── notify
    │ └── main.go
    └── tcp-server
    ├── config
    │ └── config.go
    └── main.go
    

    View Slide

31. Config
    Management
    github.com/spf13/viper
    

    View Slide

32. Config management
    • Load config from File
    • .json
    • .ini
    • Load config from Environment Variables
    • .env
    

    View Slide

33. var envfile string
    flag.StringVar(&envfile, "env-file", ".env", "Read in a file of environment
    variables")
    flag.Parse()
    godotenv.Load(envfile)
    _ "github.com/joho/godotenv/autoload"
    

    View Slide

34. Logging struct {
    Debug bool `envconfig:"GGZ_LOGS_DEBUG"`
    Level string `envconfig:"GGZ_LOGS_LEVEL" default:"info"`
    Color bool `envconfig:"GGZ_LOGS_COLOR"`
    Pretty bool `envconfig:"GGZ_LOGS_PRETTY"`
    Text bool `envconfig:"GGZ_LOGS_TEXT"`
    }
    // Server provides the server configuration.
    Server struct {
    Addr string `envconfig:"GGZ_SERVER_ADDR"`
    Port string `envconfig:"GGZ_SERVER_PORT" default:"12000"`
    Path string `envconfig:”GGZ_SERVER_PATH" default:"data"`
    }
    github.com/kelseyhightower/envconfig
    

    View Slide

35. config, err := config.Environ()
    if err != nil {
    log.Fatal().
    Err(err).
    Msg("invalid configuration")
    }
    initLogging(config)
    // check folder exist
    if !file.IsDir(config.Server.Path) {
    log.Fatal().
    Str("path", config.Server.Path).
    Msg("log folder not found")
    }
    Load env from structure
    

    View Slide

36. /configs
    Configuration file templates or default config
    

    View Slide

37. global:
    scrape_interval: 5s
    external_labels:
    monitor: 'my-monitor'
    scrape_configs:
    - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
    - job_name: 'ggz-server'
    static_configs:
    - targets: ['ggz-server:8080']
    bearer_token: 'test-prometheus-token'
    

    View Slide

38. /docker
    Docker file template
    

    View Slide

39. ├── ggz-redirect
    │ ├── Dockerfile.linux.amd64
    │ ├── Dockerfile.linux.arm
    │ ├── Dockerfile.linux.arm64
    │ ├── Dockerfile.windows.amd64
    │ └── manifest.tmpl
    └── ggz-server
    ├── Dockerfile.linux.amd64
    ├── Dockerfile.linux.arm
    ├── Dockerfile.linux.arm64
    ├── Dockerfile.windows.amd64
    └── manifest.tmpl
    

    View Slide

40. /integrations
    

    View Slide

41. ctx := context.Background()
    req := testcontainers.ContainerRequest{
    Image: "goggz/ggz-server",
    ExposedPorts: []string{"8080/tcp"},
    WaitingFor: wait.ForLog("Starting shorten server on :8080")
    }
    ggzServer, err := testcontainers.GenericContainer(
    ctx,
    testcontainers.GenericContainerRequest{
    ContainerRequest: req,
    Started: true,
    })
    if err != nil {
    t.Fatal(err)
    }
    github.com/testcontainers/testcontainers-go
    

    View Slide

42. /pkg
    

    View Slide

43. ├── config
    ├── errors
    ├── fixtures
    ├── helper
    ├── middleware
    │ ├── auth
    │ └── header
    ├── model
    ├── module
    │ ├── metrics
    │ └── storage
    │ ├── disk
    │ └── minio
    ├── router
    │ └── routes
    ├── schema
    └── version
    

    View Slide

44. /pkg/errors
    

    View Slide

45. // Type defines the type of an error
    type Type string
    const (
    // Internal error
    Internal Type = "internal"
    // NotFound error means that a specific item does not exis
    NotFound Type = "not_found"
    // BadRequest error
    BadRequest Type = "bad_request"
    // Validation error
    Validation Type = "validation"
    // AlreadyExists error
    AlreadyExists Type = "already_exists"
    // Unauthorized error
    Unauthorized Type = "unauthorized"
    )
    

    View Slide

46. // ENotExists creates an error of type NotExist
    func ENotExists(msg string, err error, arg ...interface{}) error {
    return New(NotFound, fmt.Sprintf(msg, arg...), err)
    }
    // EBadRequest creates an error of type BadRequest
    func EBadRequest(msg string, err error, arg ...interface{}) error {
    return New(BadRequest, fmt.Sprintf(msg, arg...), err)
    }
    // EAlreadyExists creates an error of type AlreadyExists
    func EAlreadyExists(msg string, err error, arg ...interface{}) error {
    return New(AlreadyExists, fmt.Sprintf(msg, arg...), err)
    }
    

    View Slide

47. /pkg/fixtures
    

    View Slide

48. Rails-like test fixtures
    Write tests against a real database
    github.com/go-testfixtures/testfixtures
    

    View Slide

49. fixtures/
    posts.yml
    comments.yml
    tags.yml
    posts_tags.yml
    users.yml
    

    View Slide

50. -
    id: 1
    email: test@gmail.com
    full_name: test
    avatar: http://example.com
    avatar_email: test@gmail.com
    -
    id: 2
    email: test1234@gmail.com
    full_name: test1234
    avatar: http://example.com
    avatar_email: test1234@gmail.com
    

    View Slide

51. Unit Testing with Database
    

    View Slide

52. func TestMain(m *testing.M) {
    // test program to do extra
    setup or teardown before or after
    testing.
    os.Exit(m.Run())
    }
    https://golang.org/pkg/testing/#hdr-Main
    

    View Slide

53. func MainTest(m *testing.M, pathToRoot string) {
    var err error
    fixturesDir := filepath.Join(pathToRoot, "pkg", "fixtures")
    if err = createTestEngine(fixturesDir); err != nil {
    fatalTestError("Error creating test engine: %v\n", err)
    }
    os.Exit(m.Run())
    }
    func createTestEngine(fixturesDir string) error {
    var err error
    x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared")
    if err != nil {
    return err
    }
    x.ShowSQL(config.Server.Debug)
    return InitFixtures(&testfixtures.SQLite{}, fixturesDir)
    }
    Testing with SQLite
    

    View Slide

54. func TestIsUserExist(t *testing.T) {
    assert.NoError(t, PrepareTestDatabase())
    exists, err := IsUserExist(0, "test@gmail.com")
    assert.NoError(t, err)
    assert.True(t, exists)
    exists, err = IsUserExist(0, "test123456@gmail.com")
    assert.NoError(t, err)
    assert.False(t, exists)
    exists, err = IsUserExist(1, "test1234@gmail.com")
    assert.NoError(t, err)
    assert.True(t, exists)
    exists, err = IsUserExist(1, "test123456@gmail.com")
    assert.NoError(t, err)
    assert.False(t, exists)
    } go test -v -run=TestIsUserExist ./pkg/models/
    

    View Slide

55. /pkg/helper
    

    View Slide

56. Helper func
    • Encrypt and Decrypt
    • Regexp func
    • IsEmail, IsUsername
    • Zipfile
    

    View Slide

57. /pkg/middleware
    

    View Slide

58. func Secure(c *gin.Context) {
    c.Header("Access-Control-Allow-Origin", "*")
    c.Header("X-Frame-Options", "DENY")
    c.Header("X-Content-Type-Options", "nosniff")
    c.Header("X-XSS-Protection", "1; mode=block")
    if c.Request.TLS != nil {
    c.Header("Strict-Transport-Security", "max-age=31536000")
    }
    }
    

    View Slide

59. /pkg/model
    

    View Slide

60. Use gorm or xorm
    

    View Slide

61. Build in
    SQLite3
    

    View Slide

62. // +build sqlite
    package model
    import (
    _ "github.com/mattn/go-sqlite3"
    )
    func init() {
    EnableSQLite3 = true
    }
    go build -v -tags 'sqlite sqlite_unlock_notify'
    

    View Slide

63. /pkg/module
    

    View Slide

64. ├── cron
    ├── download
    ├── jwt
    ├── ldap
    ├── mailer
    │ ├── ses
    │ └── smtp
    ├── queue
    ├── metrics
    ├── redis
    └── storage
    ├── disk
    └── minio
    

    View Slide

65. Integration with
    Prometheus + Grafana
    

    View Slide

66. func NewCollector() Collector {
    return Collector{
    Users: prometheus.NewDesc(
    namespace+"users",
    "Number of Users",
    nil, nil,
    ),
    }
    // Collect returns the metrics with values
    func (c Collector) Collect(ch chanstats := model.GetStatistic()
    ch c.Users,
    prometheus.GaugeValue,
    float64(stats.Counter.User),
    )
    }
    

    View Slide

67. Prometheus Handler
    

    View Slide

68. func Metrics(token string) gin.HandlerFunc {
    h := promhttp.Handler()
    return func(c *gin.Context) {
    if token == "" {
    h.ServeHTTP(c.Writer, c.Request)
    return
    }
    header := c.Request.Header.Get("Authorization")
    if header == "" {
    c.String(http.StatusUnauthorized, errInvalidToken.Error())
    return
    }
    bearer := fmt.Sprintf("Bearer %s", token)
    if header != bearer {
    c.String(http.StatusUnauthorized, errInvalidToken.Error())
    return
    }
    h.ServeHTTP(c.Writer, c.Request)
    }
    }
    

    View Slide

69. c := metrics.NewCollector()
    prometheus.MustRegister(c)
    if config.Metrics.Enabled {
    root.GET("/metrics", router.Metrics(config.Metrics.Token))
    }
    Your prometheus token
    

    View Slide

70. /pkg/schema
    

    View Slide

71. RESTful vs GraphQL
    See the Slide: GraphQL in Go
    

    View Slide

72. var rootQuery = graphql.NewObject(
    graphql.ObjectConfig{
    Name: "RootQuery",
    Description: "Root Query",
    Fields: graphql.Fields{
    "queryShortenURL": &queryShortenURL,
    "queryMe": &queryMe,
    },
    })
    var rootMutation = graphql.NewObject(
    graphql.ObjectConfig{
    Name: "RootMutation",
    Description: "Root Mutation",
    Fields: graphql.Fields{
    "createUser": &createUser,
    },
    })
    // Schema is the GraphQL schema served by the server.
    var Schema, _ = graphql.NewSchema(
    graphql.SchemaConfig{
    Query: rootQuery,
    Mutation: rootMutation,
    })
    

    View Slide

73. Write the GraphQL Testing
    

    View Slide

74. assert.NoError(t, model.PrepareTestDatabase())
    t.Run("user not login", func(t *testing.T) {
    test := T{
    Query: `{
    queryMe {
    email
    }
    }`,
    Schema: Schema,
    Expected: &graphql.Result{
    Data: map[string]interface{}{
    "queryMe": nil,
    },
    Errors: []gqlerrors.FormattedError{
    {
    Message: errorYouAreNotLogin,
    },
    },
    },
    }
    })
    }
    

    View Slide

75. Best Practice
    Testing your Go code
    

    View Slide

76. View Slide

77. Testable Code
    • Code Quality
    • Readability
    • Maintainability
    • Testability
    

    View Slide

78. #1. Testing in Go
    func TestFooBar(t *testing.T) {}
    func ExampleFooBar(t *testing.T) {}
    func BenchmarkFooBar(b *testing.B) {}
    go test package_name
    

    View Slide

79. #2. Benchmark Testing
    Profiling: CPU, Memory, Goroutine Block
    

    View Slide

80. func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) {
    for i := 0; i < b.N; i++ {
    context := map[string]interface{}{}
    variables := map[string]interface{}{}
    playlyfeExecutor.Execute(context, "{hello}", variables, "")
    }
    }
    func BenchmarkGophersGraphQLMaster(b *testing.B) {
    for i := 0; i < b.N; i++ {
    ctx := context.Background()
    variables := map[string]interface{}{}
    gopherSchema.Exec(ctx, "{hello}", "", variables)
    }
    } http://bit.ly/2L0CG3Q
    

    View Slide

81. View Slide

82. #3. Example Testing
    Examples on how to use your code
    

    View Slide

83. func ExampleFooBar() {
    fmt.Println(strings.Compare("a", "b"))
    fmt.Println(strings.Compare("a", "a"))
    fmt.Println(strings.Compare("b", "a"))
    // Output:
    // -1
    // 0
    // 1
    }
    

    View Slide

84. $ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/...
    === RUN ExampleFooBar
    --- PASS: ExampleFooBar (0.00s)
    PASS
    ok github.com/go-ggz/ggz/pkg/model 0.022s
    

    View Slide

85. #4. Subtests in Testing Package
    func (t *T) Run(name string, f func(t *T)) bool {}
    func (b *B) Run(name string, f func(b *B)) bool {}
    

    View Slide

86. tests := []struct {
    name string
    fields fields
    args args
    }{}
    for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
    c := Collector{
    Shortens: tt.fields.Shortens,
    Users: tt.fields.Users,
    }
    c.Describe(tt.args.ch)
    })
    }
    }
    

    View Slide

87. #5. Skipping Testing
    t.Skip()
    

    View Slide

88. package metrics
    import (
    "os"
    "testing"
    )
    func TestSkip(t *testing.T) {
    if os.Getenv("DEBUG_MODE") == "true" {
    t.Skipf("test skipped")
    }
    }
    

    View Slide

89. #6. Running Tests in Parallel
    Speedup your CI/CD Flow
    t.Parallel()
    

    View Slide

90. func TestFooBar01(t *testing.T) {
    t.Parallel()
    time.Sleep(time.Second)
    }
    func TestFooBar02(t *testing.T) {
    t.Parallel()
    time.Sleep(time.Second * 2)
    }
    func TestFooBar03(t *testing.T) {
    t.Parallel()
    time.Sleep(time.Second * 3)
    }
    

    View Slide

91. Just only use
    one package
    github.com/stretchr/testify
    

    View Slide

92. https://www.udemy.com/course/golang-fight/?couponCode=GOLANG2019
    

    View Slide

93. https://www.udemy.com/course/devops-oneday/?couponCode=DRONE2019
    

    View Slide

94. END
    

    View Slide