Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Golang Project Layout and Practice

265bcbb56e831266de7a9f9281aab57a?s=47 Bo-Yi Wu
August 29, 2019

Golang Project Layout and Practice

This talk I will show you the golang project layout and some best practice like as the following:
1. RESTful api and GraphQL
2. Model testing (Postgres, SQLite, MySQL)
3. Software Quality
4. Data Metrics
5. Go Testing

265bcbb56e831266de7a9f9281aab57a?s=128

Bo-Yi Wu

August 29, 2019
Tweet

Transcript

  1. Go Project Layout and Practice Bo-Yi Wu 2019.08.29 ModernWeb

  2. About me • Software Engineer in Mediatek • Member of

    Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Teacher of Udemy Platform: Golang + Drone https://blog.wu-boy.com
  3. Agenda • Go in Mediatek • Go Project Layout •

    Go Practices • RESTful api and GraphQL • Model testing (Postgres, SQLite, MySQL) • Software Quality • Data Metrics • Go Testing
  4. Tech Stack • Initial Project using Go in 2018/01 •

    Golang • Easy to Learn • Performance • Deployment
  5. Repository folder • api • assets • cmd • configs

    • docker • pkg ├── api ├── assets │ └── dist ├── cmd │ └── ggz ├── configs ├── docker │ ├── server └── pkg ├── config ├── errors ├── fixtures ├── helper ├── middleware │ ├── auth │ └── header ├── model ├── module │ ├── mailer │ ├── metrics │ └── storage ├── router │ └── routes ├── schema └── version
  6. Root folder • .drone.yml (deploy config) • .revive.toml (golint config)

    • docker-compose.yml (DB, Redis and UI) • Makefile • go module config (go.mod and go.sum) • .env.example
  7. Go Module https://blog.golang.org/using-go-modules

  8. Improve Deployment Using Go Module Proxy https://github.com/gomods/athens

  9. save time with proxy 97s -> 6s

  10. Makefile Build, Testing, Deploy

  11. GOFMT ?= gofmt "-s" GO ?= go TARGETS ?= linux

    darwin windows ARCHS ?= amd64 386 BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") GOFILES := $(shell find . -name "*.go" -type f) TAGS ?= sqlite sqlite_unlock_notify ifneq ($(shell uname), Darwin) EXTLDFLAGS = -extldflags "-static" $(null) else EXTLDFLAGS = endif ifneq ($(DRONE_TAG),) VERSION ?= $(subst v,,$(DRONE_TAG)) else VERSION ?= $(shell git describe --tags --always') endif
  12. .env

  13. GGZ_DB_DRIVER=mysql GGZ_DB_USERNAME=root GGZ_DB_PASSWORD=123456 GGZ_DB_NAME=ggz GGZ_DB_HOST=127.0.0.1:3307 GGZ_SERVER_ADDR=:8080 GGZ_DEBUG=true GGZ_SERVER_HOST=http://localhost:8080 GGZ_STORAGE_DRIVER=disk GGZ_MINIO_ACCESS_ID=xxxxxxxx

    GGZ_MINIO_SECRET_KEY=xxxxxxxx GGZ_MINIO_ENDPOINT=s3.example.com GGZ_MINIO_BUCKET=example GGZ_MINIO_SSL=true GGZ_AUTH0_DEBUG=true
  14. docker-compose.yml

  15. db: image: mysql restart: always volumes: - mysql-data:/var/lib/mysql environment: MYSQL_USER:

    example MYSQL_PASSWORD: example MYSQL_DATABASE: example MYSQL_ROOT_PASSWORD: example minio: image: minio/minio restart: always ports: volumes: - minio-data:/data environment: MINIO_ACCESS_KEY: minio123456 MINIO_SECRET_KEY: minio123456 command: server /data Development
  16. Production api: image: foo/bar restart: always ports: - 8080:8080 environment:

    - GGZ_METRICS_TOKEN=test-prometheus-token - GGZ_METRICS_ENABLED=true labels: - "traefik.enable=true" - "traefik.basic.frontend.rule=Host:${WEB_HOST}" - "traefik.basic.protocol=http"
  17. Version Compile version info into Go binary

  18. Version • -X github.com/go-ggz/ggz/pkg/ version.Version=$(VERSION) • -X github.com/go-ggz/ggz/pkg/ version.BuildDate=$(BUILD_DATE) go

    build -o bin/api -ldflags
  19. var ( // Version number for git tag. Version string

    // BuildDate is the ISO 8601 day drone was built. BuildDate string ) // PrintCLIVersion print server info func PrintCLIVersion() string { return fmt.Sprintf( "version %s, built on %s, %s", Version, BuildDate, runtime.Version(), ) }
  20. BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") ifneq ($(DRONE_TAG),) VERSION ?=

    $(subst v,,$(DRONE_TAG)) else VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed 's/^v//') endif
  21. Assets Embed files in Go https://github.com/UnnoTed/fileb0x

  22. func ReadSource(origPath string) (content []byte, err error) { content, err

    = ReadFile(origPath) if err != nil { log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath) } if config.Server.Assets != "" && file.IsDir(config.Server.Assets) { origPath = path.Join(config.Server.Assets, origPath) if file.IsFile(origPath) { content, err = ioutil.ReadFile(origPath) if err != nil { log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath) } } } return content, err } Debug Setting
  23. // ViewHandler support dist handler from UI func ViewHandler() gin.HandlerFunc

    { fileServer := http.FileServer(dist.HTTP) data := []byte(time.Now().String()) etag := fmt.Sprintf("%x", md5.Sum(data)) return func(c *gin.Context) { c.Header("Cache-Control", "public, max-age=31536000") c.Header("ETag", etag) if match := c.GetHeader("If-None-Match"); match != "" { if strings.Contains(match, etag) { c.Status(http.StatusNotModified) return } } fileServer.ServeHTTP(c.Writer, c.Request) } } File Server Handler
  24. ᅷยိݯɿIUUQTEFWFMPQFSTHPPHMFDPNXFCGVOEBNFOUBMTQFSGPSNBODF PQUJNJ[JOHDPOUFOUF⒏DJFODZIUUQDBDIJOH IM[IUX

  25. // Favicon represents the favicon. func Favicon(c *gin.Context) { file,

    _ := dist.ReadFile("favicon.ico") etag := fmt.Sprintf("%x", md5.Sum(file)) c.Header("ETag", etag) c.Header("Cache-Control", "max-age=0") if match := c.GetHeader("If-None-Match"); match != "" { if strings.Contains(match, etag) { c.Status(http.StatusNotModified) return } } c.Data( http.StatusOK, "image/x-icon", file, ) } NO Cache
  26. API

  27. /healthz • health check for load balancer func Heartbeat(c *gin.Context)

    { c.AbortWithStatus(http.StatusOK) c.String(http.StatusOK, "ok") }
  28. CMD Command line

  29. Command line package • Golang package: flag • urfave/cli •

    spf13/cobra
  30. ├── agent │ ├── config │ │ └── config.go │

    └── main.go ├── notify │ └── main.go └── tcp-server ├── config │ └── config.go └── main.go
  31. Config Management github.com/spf13/viper

  32. Config management • Load config from File • .json •

    .ini • Load config from Environment Variables • .env
  33. var envfile string flag.StringVar(&envfile, "env-file", ".env", "Read in a file

    of environment variables") flag.Parse() godotenv.Load(envfile) _ "github.com/joho/godotenv/autoload"
  34. Logging struct { Debug bool `envconfig:"GGZ_LOGS_DEBUG"` Level string `envconfig:"GGZ_LOGS_LEVEL" default:"info"`

    Color bool `envconfig:"GGZ_LOGS_COLOR"` Pretty bool `envconfig:"GGZ_LOGS_PRETTY"` Text bool `envconfig:"GGZ_LOGS_TEXT"` } // Server provides the server configuration. Server struct { Addr string `envconfig:"GGZ_SERVER_ADDR"` Port string `envconfig:"GGZ_SERVER_PORT" default:"12000"` Path string `envconfig:”GGZ_SERVER_PATH" default:"data"` } github.com/kelseyhightower/envconfig
  35. config, err := config.Environ() if err != nil { log.Fatal().

    Err(err). Msg("invalid configuration") } initLogging(config) // check folder exist if !file.IsDir(config.Server.Path) { log.Fatal(). Str("path", config.Server.Path). Msg("log folder not found") } Load env from structure
  36. /configs Configuration file templates or default config

  37. global: scrape_interval: 5s external_labels: monitor: 'my-monitor' scrape_configs: - job_name: 'prometheus'

    static_configs: - targets: ['localhost:9090'] - job_name: 'ggz-server' static_configs: - targets: ['ggz-server:8080'] bearer_token: 'test-prometheus-token'
  38. /docker Docker file template

  39. ├── ggz-redirect │ ├── Dockerfile.linux.amd64 │ ├── Dockerfile.linux.arm │ ├──

    Dockerfile.linux.arm64 │ ├── Dockerfile.windows.amd64 │ └── manifest.tmpl └── ggz-server ├── Dockerfile.linux.amd64 ├── Dockerfile.linux.arm ├── Dockerfile.linux.arm64 ├── Dockerfile.windows.amd64 └── manifest.tmpl
  40. /integrations

  41. ctx := context.Background() req := testcontainers.ContainerRequest{ Image: "goggz/ggz-server", ExposedPorts: []string{"8080/tcp"},

    WaitingFor: wait.ForLog("Starting shorten server on :8080") } ggzServer, err := testcontainers.GenericContainer( ctx, testcontainers.GenericContainerRequest{ ContainerRequest: req, Started: true, }) if err != nil { t.Fatal(err) } github.com/testcontainers/testcontainers-go
  42. /pkg

  43. ├── config ├── errors ├── fixtures ├── helper ├── middleware

    │ ├── auth │ └── header ├── model ├── module │ ├── metrics │ └── storage │ ├── disk │ └── minio ├── router │ └── routes ├── schema └── version
  44. /pkg/errors

  45. // Type defines the type of an error type Type

    string const ( // Internal error Internal Type = "internal" // NotFound error means that a specific item does not exis NotFound Type = "not_found" // BadRequest error BadRequest Type = "bad_request" // Validation error Validation Type = "validation" // AlreadyExists error AlreadyExists Type = "already_exists" // Unauthorized error Unauthorized Type = "unauthorized" )
  46. // ENotExists creates an error of type NotExist func ENotExists(msg

    string, err error, arg ...interface{}) error { return New(NotFound, fmt.Sprintf(msg, arg...), err) } // EBadRequest creates an error of type BadRequest func EBadRequest(msg string, err error, arg ...interface{}) error { return New(BadRequest, fmt.Sprintf(msg, arg...), err) } // EAlreadyExists creates an error of type AlreadyExists func EAlreadyExists(msg string, err error, arg ...interface{}) error { return New(AlreadyExists, fmt.Sprintf(msg, arg...), err) }
  47. /pkg/fixtures

  48. Rails-like test fixtures Write tests against a real database github.com/go-testfixtures/testfixtures

  49. fixtures/ posts.yml comments.yml tags.yml posts_tags.yml users.yml

  50. - id: 1 email: test@gmail.com full_name: test avatar: http://example.com avatar_email:

    test@gmail.com - id: 2 email: test1234@gmail.com full_name: test1234 avatar: http://example.com avatar_email: test1234@gmail.com
  51. Unit Testing with Database

  52. func TestMain(m *testing.M) { // test program to do extra

    setup or teardown before or after testing. os.Exit(m.Run()) } https://golang.org/pkg/testing/#hdr-Main
  53. func MainTest(m *testing.M, pathToRoot string) { var err error fixturesDir

    := filepath.Join(pathToRoot, "pkg", "fixtures") if err = createTestEngine(fixturesDir); err != nil { fatalTestError("Error creating test engine: %v\n", err) } os.Exit(m.Run()) } func createTestEngine(fixturesDir string) error { var err error x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared") if err != nil { return err } x.ShowSQL(config.Server.Debug) return InitFixtures(&testfixtures.SQLite{}, fixturesDir) } Testing with SQLite
  54. func TestIsUserExist(t *testing.T) { assert.NoError(t, PrepareTestDatabase()) exists, err := IsUserExist(0,

    "test@gmail.com") assert.NoError(t, err) assert.True(t, exists) exists, err = IsUserExist(0, "test123456@gmail.com") assert.NoError(t, err) assert.False(t, exists) exists, err = IsUserExist(1, "test1234@gmail.com") assert.NoError(t, err) assert.True(t, exists) exists, err = IsUserExist(1, "test123456@gmail.com") assert.NoError(t, err) assert.False(t, exists) } go test -v -run=TestIsUserExist ./pkg/models/
  55. /pkg/helper

  56. Helper func • Encrypt and Decrypt • Regexp func •

    IsEmail, IsUsername • Zipfile
  57. /pkg/middleware

  58. func Secure(c *gin.Context) { c.Header("Access-Control-Allow-Origin", "*") c.Header("X-Frame-Options", "DENY") c.Header("X-Content-Type-Options", "nosniff")

    c.Header("X-XSS-Protection", "1; mode=block") if c.Request.TLS != nil { c.Header("Strict-Transport-Security", "max-age=31536000") } }
  59. /pkg/model

  60. Use gorm or xorm

  61. Build in SQLite3

  62. // +build sqlite package model import ( _ "github.com/mattn/go-sqlite3" )

    func init() { EnableSQLite3 = true } go build -v -tags 'sqlite sqlite_unlock_notify'
  63. /pkg/module

  64. ├── cron ├── download ├── jwt ├── ldap ├── mailer

    │ ├── ses │ └── smtp ├── queue ├── metrics ├── redis └── storage ├── disk └── minio
  65. Integration with Prometheus + Grafana

  66. func NewCollector() Collector { return Collector{ Users: prometheus.NewDesc( namespace+"users", "Number

    of Users", nil, nil, ), } // Collect returns the metrics with values func (c Collector) Collect(ch chan<- prometheus.Metric) { stats := model.GetStatistic() ch <- prometheus.MustNewConstMetric( c.Users, prometheus.GaugeValue, float64(stats.Counter.User), ) }
  67. Prometheus Handler

  68. func Metrics(token string) gin.HandlerFunc { h := promhttp.Handler() return func(c

    *gin.Context) { if token == "" { h.ServeHTTP(c.Writer, c.Request) return } header := c.Request.Header.Get("Authorization") if header == "" { c.String(http.StatusUnauthorized, errInvalidToken.Error()) return } bearer := fmt.Sprintf("Bearer %s", token) if header != bearer { c.String(http.StatusUnauthorized, errInvalidToken.Error()) return } h.ServeHTTP(c.Writer, c.Request) } }
  69. c := metrics.NewCollector() prometheus.MustRegister(c) if config.Metrics.Enabled { root.GET("/metrics", router.Metrics(config.Metrics.Token)) }

    Your prometheus token
  70. /pkg/schema

  71. RESTful vs GraphQL See the Slide: GraphQL in Go

  72. var rootQuery = graphql.NewObject( graphql.ObjectConfig{ Name: "RootQuery", Description: "Root Query",

    Fields: graphql.Fields{ "queryShortenURL": &queryShortenURL, "queryMe": &queryMe, }, }) var rootMutation = graphql.NewObject( graphql.ObjectConfig{ Name: "RootMutation", Description: "Root Mutation", Fields: graphql.Fields{ "createUser": &createUser, }, }) // Schema is the GraphQL schema served by the server. var Schema, _ = graphql.NewSchema( graphql.SchemaConfig{ Query: rootQuery, Mutation: rootMutation, })
  73. Write the GraphQL Testing

  74. assert.NoError(t, model.PrepareTestDatabase()) t.Run("user not login", func(t *testing.T) { test :=

    T{ Query: `{ queryMe { email } }`, Schema: Schema, Expected: &graphql.Result{ Data: map[string]interface{}{ "queryMe": nil, }, Errors: []gqlerrors.FormattedError{ { Message: errorYouAreNotLogin, }, }, }, } }) }
  75. Best Practice Testing your Go code

  76. None
  77. Testable Code • Code Quality • Readability • Maintainability •

    Testability
  78. #1. Testing in Go func TestFooBar(t *testing.T) {} func ExampleFooBar(t

    *testing.T) {} func BenchmarkFooBar(b *testing.B) {} go test package_name
  79. #2. Benchmark Testing Profiling: CPU, Memory, Goroutine Block

  80. func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) { for i := 0; i <

    b.N; i++ { context := map[string]interface{}{} variables := map[string]interface{}{} playlyfeExecutor.Execute(context, "{hello}", variables, "") } } func BenchmarkGophersGraphQLMaster(b *testing.B) { for i := 0; i < b.N; i++ { ctx := context.Background() variables := map[string]interface{}{} gopherSchema.Exec(ctx, "{hello}", "", variables) } } http://bit.ly/2L0CG3Q
  81. None
  82. #3. Example Testing Examples on how to use your code

  83. func ExampleFooBar() { fmt.Println(strings.Compare("a", "b")) fmt.Println(strings.Compare("a", "a")) fmt.Println(strings.Compare("b", "a")) //

    Output: // -1 // 0 // 1 }
  84. $ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/... === RUN ExampleFooBar

    --- PASS: ExampleFooBar (0.00s) PASS ok github.com/go-ggz/ggz/pkg/model 0.022s
  85. #4. Subtests in Testing Package func (t *T) Run(name string,

    f func(t *T)) bool {} func (b *B) Run(name string, f func(b *B)) bool {}
  86. tests := []struct { name string fields fields args args

    }{} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := Collector{ Shortens: tt.fields.Shortens, Users: tt.fields.Users, } c.Describe(tt.args.ch) }) } }
  87. #5. Skipping Testing t.Skip()

  88. package metrics import ( "os" "testing" ) func TestSkip(t *testing.T)

    { if os.Getenv("DEBUG_MODE") == "true" { t.Skipf("test skipped") } }
  89. #6. Running Tests in Parallel Speedup your CI/CD Flow t.Parallel()

  90. func TestFooBar01(t *testing.T) { t.Parallel() time.Sleep(time.Second) } func TestFooBar02(t *testing.T)

    { t.Parallel() time.Sleep(time.Second * 2) } func TestFooBar03(t *testing.T) { t.Parallel() time.Sleep(time.Second * 3) }
  91. Just only use one package github.com/stretchr/testify

  92. https://www.udemy.com/course/golang-fight/?couponCode=GOLANG2019

  93. https://www.udemy.com/course/devops-oneday/?couponCode=DRONE2019

  94. END