Golang Project Layout and Practice

Slide 1

Slide 1 text

Go Project Layout and Practice Bo-Yi Wu 2019.08.29 ModernWeb

Slide 2

Slide 2 text

About me • Software Engineer in Mediatek • Member of Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Teacher of Udemy Platform: Golang + Drone https://blog.wu-boy.com

Slide 3

Slide 3 text

Agenda • Go in Mediatek • Go Project Layout • Go Practices • RESTful api and GraphQL • Model testing (Postgres, SQLite, MySQL) • Software Quality • Data Metrics • Go Testing

Slide 4

Slide 4 text

Tech Stack • Initial Project using Go in 2018/01 • Golang • Easy to Learn • Performance • Deployment

Slide 5

Slide 5 text

Repository folder • api • assets • cmd • conﬁgs • docker • pkg ├── api ├── assets │ └── dist ├── cmd │ └── ggz ├── configs ├── docker │ ├── server └── pkg ├── config ├── errors ├── fixtures ├── helper ├── middleware │ ├── auth │ └── header ├── model ├── module │ ├── mailer │ ├── metrics │ └── storage ├── router │ └── routes ├── schema └── version

Slide 6

Slide 6 text

Root folder • .drone.yml (deploy config) • .revive.toml (golint config) • docker-compose.yml (DB, Redis and UI) • Makefile • go module config (go.mod and go.sum) • .env.example

Slide 7

Slide 7 text

Go Module https://blog.golang.org/using-go-modules

Slide 8

Slide 8 text

Improve Deployment Using Go Module Proxy https://github.com/gomods/athens

Slide 9

Slide 9 text

save time with proxy 97s -> 6s

Slide 10

Slide 10 text

Makeﬁle Build, Testing, Deploy

Slide 11

Slide 11 text

GOFMT ?= gofmt "-s" GO ?= go TARGETS ?= linux darwin windows ARCHS ?= amd64 386 BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") GOFILES := $(shell find . -name "*.go" -type f) TAGS ?= sqlite sqlite_unlock_notify ifneq ($(shell uname), Darwin) EXTLDFLAGS = -extldflags "-static" $(null) else EXTLDFLAGS = endif ifneq ($(DRONE_TAG),) VERSION ?= $(subst v,,$(DRONE_TAG)) else VERSION ?= $(shell git describe --tags --always') endif

Slide 12

Slide 12 text

.env

Slide 13

Slide 13 text

GGZ_DB_DRIVER=mysql GGZ_DB_USERNAME=root GGZ_DB_PASSWORD=123456 GGZ_DB_NAME=ggz GGZ_DB_HOST=127.0.0.1:3307 GGZ_SERVER_ADDR=:8080 GGZ_DEBUG=true GGZ_SERVER_HOST=http://localhost:8080 GGZ_STORAGE_DRIVER=disk GGZ_MINIO_ACCESS_ID=xxxxxxxx GGZ_MINIO_SECRET_KEY=xxxxxxxx GGZ_MINIO_ENDPOINT=s3.example.com GGZ_MINIO_BUCKET=example GGZ_MINIO_SSL=true GGZ_AUTH0_DEBUG=true

Slide 14

Slide 14 text

docker-compose.yml

Slide 15

Slide 15 text

db: image: mysql restart: always volumes: - mysql-data:/var/lib/mysql environment: MYSQL_USER: example MYSQL_PASSWORD: example MYSQL_DATABASE: example MYSQL_ROOT_PASSWORD: example minio: image: minio/minio restart: always ports: volumes: - minio-data:/data environment: MINIO_ACCESS_KEY: minio123456 MINIO_SECRET_KEY: minio123456 command: server /data Development

Slide 16

Slide 16 text

Production api: image: foo/bar restart: always ports: - 8080:8080 environment: - GGZ_METRICS_TOKEN=test-prometheus-token - GGZ_METRICS_ENABLED=true labels: - "traefik.enable=true" - "traefik.basic.frontend.rule=Host:${WEB_HOST}" - "traefik.basic.protocol=http"

Slide 17

Slide 17 text

Version Compile version info into Go binary

Slide 18

Slide 18 text

Version • -X github.com/go-ggz/ggz/pkg/ version.Version=$(VERSION) • -X github.com/go-ggz/ggz/pkg/ version.BuildDate=$(BUILD_DATE) go build -o bin/api -ldﬂags

Slide 19

Slide 19 text

var ( // Version number for git tag. Version string // BuildDate is the ISO 8601 day drone was built. BuildDate string ) // PrintCLIVersion print server info func PrintCLIVersion() string { return fmt.Sprintf( "version %s, built on %s, %s", Version, BuildDate, runtime.Version(), ) }

Slide 20

Slide 20 text

BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ") ifneq ($(DRONE_TAG),) VERSION ?= $(subst v,,$(DRONE_TAG)) else VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed 's/^v//') endif

Slide 21

Slide 21 text

Assets Embed ﬁles in Go https://github.com/UnnoTed/ﬁleb0x

Slide 22

Slide 22 text

func ReadSource(origPath string) (content []byte, err error) { content, err = ReadFile(origPath) if err != nil { log.Warn().Err(err).Msgf("Failed to read builtin %s file.", origPath) } if config.Server.Assets != "" && file.IsDir(config.Server.Assets) { origPath = path.Join(config.Server.Assets, origPath) if file.IsFile(origPath) { content, err = ioutil.ReadFile(origPath) if err != nil { log.Warn().Err(err).Msgf("Failed to read custom %s file", origPath) } } } return content, err } Debug Setting

Slide 23

Slide 23 text

// ViewHandler support dist handler from UI func ViewHandler() gin.HandlerFunc { fileServer := http.FileServer(dist.HTTP) data := []byte(time.Now().String()) etag := fmt.Sprintf("%x", md5.Sum(data)) return func(c *gin.Context) { c.Header("Cache-Control", "public, max-age=31536000") c.Header("ETag", etag) if match := c.GetHeader("If-None-Match"); match != "" { if strings.Contains(match, etag) { c.Status(http.StatusNotModified) return } } fileServer.ServeHTTP(c.Writer, c.Request) } } File Server Handler

Slide 24

Slide 24 text

ᅷยိݯɿIUUQTEFWFMPQFSTHPPHMFDPNXFCGVOEBNFOUBMTQFSGPSNBODF PQUJNJ[JOHDPOUFOUF⒏DJFODZIUUQDBDIJOH IM[IUX

Slide 25

Slide 25 text

// Favicon represents the favicon. func Favicon(c *gin.Context) { file, _ := dist.ReadFile("favicon.ico") etag := fmt.Sprintf("%x", md5.Sum(file)) c.Header("ETag", etag) c.Header("Cache-Control", "max-age=0") if match := c.GetHeader("If-None-Match"); match != "" { if strings.Contains(match, etag) { c.Status(http.StatusNotModified) return } } c.Data( http.StatusOK, "image/x-icon", file, ) } NO Cache

Slide 26

Slide 26 text

API

Slide 27

Slide 27 text

/healthz • health check for load balancer func Heartbeat(c *gin.Context) { c.AbortWithStatus(http.StatusOK) c.String(http.StatusOK, "ok") }

Slide 28

Slide 28 text

CMD Command line

Slide 29

Slide 29 text

Command line package • Golang package: ﬂag • urfave/cli • spf13/cobra

Slide 30

Slide 30 text

├── agent │ ├── config │ │ └── config.go │ └── main.go ├── notify │ └── main.go └── tcp-server ├── config │ └── config.go └── main.go

Slide 31

Slide 31 text

Conﬁg Management github.com/spf13/viper

Slide 32

Slide 32 text

Config management • Load config from File • .json • .ini • Load config from Environment Variables • .env

Slide 33

Slide 33 text

var envfile string flag.StringVar(&envfile, "env-file", ".env", "Read in a file of environment variables") flag.Parse() godotenv.Load(envfile) _ "github.com/joho/godotenv/autoload"

Slide 34

Slide 34 text

Logging struct { Debug bool ènvconfig:"GGZ_LOGS_DEBUG"` Level string ènvconfig:"GGZ_LOGS_LEVEL" default:"info"` Color bool ènvconfig:"GGZ_LOGS_COLOR"` Pretty bool ènvconfig:"GGZ_LOGS_PRETTY"` Text bool ènvconfig:"GGZ_LOGS_TEXT"` } // Server provides the server configuration. Server struct { Addr string ènvconfig:"GGZ_SERVER_ADDR"` Port string ènvconfig:"GGZ_SERVER_PORT" default:"12000"` Path string ènvconfig:”GGZ_SERVER_PATH" default:"data"` } github.com/kelseyhightower/envconfig

Slide 35

Slide 35 text

config, err := config.Environ() if err != nil { log.Fatal(). Err(err). Msg("invalid configuration") } initLogging(config) // check folder exist if !file.IsDir(config.Server.Path) { log.Fatal(). Str("path", config.Server.Path). Msg("log folder not found") } Load env from structure

Slide 36

Slide 36 text

/configs Configuration file templates or default config

Slide 37

Slide 37 text

global: scrape_interval: 5s external_labels: monitor: 'my-monitor' scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'ggz-server' static_configs: - targets: ['ggz-server:8080'] bearer_token: 'test-prometheus-token'

Slide 38

Slide 38 text

/docker Docker ﬁle template

Slide 39

Slide 39 text

├── ggz-redirect │ ├── Dockerfile.linux.amd64 │ ├── Dockerfile.linux.arm │ ├── Dockerfile.linux.arm64 │ ├── Dockerfile.windows.amd64 │ └── manifest.tmpl └── ggz-server ├── Dockerfile.linux.amd64 ├── Dockerfile.linux.arm ├── Dockerfile.linux.arm64 ├── Dockerfile.windows.amd64 └── manifest.tmpl

Slide 40

Slide 40 text

/integrations

Slide 41

Slide 41 text

ctx := context.Background() req := testcontainers.ContainerRequest{ Image: "goggz/ggz-server", ExposedPorts: []string{"8080/tcp"}, WaitingFor: wait.ForLog("Starting shorten server on :8080") } ggzServer, err := testcontainers.GenericContainer( ctx, testcontainers.GenericContainerRequest{ ContainerRequest: req, Started: true, }) if err != nil { t.Fatal(err) } github.com/testcontainers/testcontainers-go

Slide 42

Slide 42 text

/pkg

Slide 43

Slide 43 text

├── conﬁg ├── errors ├── ﬁxtures ├── helper ├── middleware │ ├── auth │ └── header ├── model ├── module │ ├── metrics │ └── storage │ ├── disk │ └── minio ├── router │ └── routes ├── schema └── version

Slide 44

Slide 44 text

/pkg/errors

Slide 45

Slide 45 text

// Type deﬁnes the type of an error type Type string const ( // Internal error Internal Type = "internal" // NotFound error means that a speciﬁc item does not exis NotFound Type = "not_found" // BadRequest error BadRequest Type = "bad_request" // Validation error Validation Type = "validation" // AlreadyExists error AlreadyExists Type = "already_exists" // Unauthorized error Unauthorized Type = "unauthorized" )

Slide 46

Slide 46 text

// ENotExists creates an error of type NotExist func ENotExists(msg string, err error, arg ...interface{}) error { return New(NotFound, fmt.Sprintf(msg, arg...), err) } // EBadRequest creates an error of type BadRequest func EBadRequest(msg string, err error, arg ...interface{}) error { return New(BadRequest, fmt.Sprintf(msg, arg...), err) } // EAlreadyExists creates an error of type AlreadyExists func EAlreadyExists(msg string, err error, arg ...interface{}) error { return New(AlreadyExists, fmt.Sprintf(msg, arg...), err) }

Slide 47

Slide 47 text

/pkg/ﬁxtures

Slide 48

Slide 48 text

Rails-like test fixtures Write tests against a real database github.com/go-testfixtures/testfixtures

Slide 49

Slide 49 text

ﬁxtures/ posts.yml comments.yml tags.yml posts_tags.yml users.yml

Slide 50

Slide 50 text

- id: 1 email: [email protected] full_name: test avatar: http://example.com avatar_email: [email protected] - id: 2 email: [email protected] full_name: test1234 avatar: http://example.com avatar_email: [email protected]

Slide 51

Slide 51 text

Unit Testing with Database

Slide 52

Slide 52 text

func TestMain(m *testing.M) { // test program to do extra setup or teardown before or after testing. os.Exit(m.Run()) } https://golang.org/pkg/testing/#hdr-Main

Slide 53

Slide 53 text

func MainTest(m *testing.M, pathToRoot string) { var err error fixturesDir := filepath.Join(pathToRoot, "pkg", "fixtures") if err = createTestEngine(fixturesDir); err != nil { fatalTestError("Error creating test engine: %v\n", err) } os.Exit(m.Run()) } func createTestEngine(fixturesDir string) error { var err error x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared") if err != nil { return err } x.ShowSQL(config.Server.Debug) return InitFixtures(&testfixtures.SQLite{}, fixturesDir) } Testing with SQLite

Slide 54

Slide 54 text

func TestIsUserExist(t *testing.T) { assert.NoError(t, PrepareTestDatabase()) exists, err := IsUserExist(0, "[email protected]") assert.NoError(t, err) assert.True(t, exists) exists, err = IsUserExist(0, "[email protected]") assert.NoError(t, err) assert.False(t, exists) exists, err = IsUserExist(1, "[email protected]") assert.NoError(t, err) assert.True(t, exists) exists, err = IsUserExist(1, "[email protected]") assert.NoError(t, err) assert.False(t, exists) } go test -v -run=TestIsUserExist ./pkg/models/

Slide 55

Slide 55 text

/pkg/helper

Slide 56

Slide 56 text

Helper func • Encrypt and Decrypt • Regexp func • IsEmail, IsUsername • Zipﬁle

Slide 57

Slide 57 text

/pkg/middleware

Slide 58

Slide 58 text

func Secure(c *gin.Context) { c.Header("Access-Control-Allow-Origin", "*") c.Header("X-Frame-Options", "DENY") c.Header("X-Content-Type-Options", "nosniff") c.Header("X-XSS-Protection", "1; mode=block") if c.Request.TLS != nil { c.Header("Strict-Transport-Security", "max-age=31536000") } }

Slide 59

Slide 59 text

/pkg/model

Slide 60

Slide 60 text

Use gorm or xorm

Slide 61

Slide 61 text

Build in SQLite3

Slide 62

Slide 62 text

// +build sqlite package model import ( _ "github.com/mattn/go-sqlite3" ) func init() { EnableSQLite3 = true } go build -v -tags 'sqlite sqlite_unlock_notify'

Slide 63

Slide 63 text

/pkg/module

Slide 64

Slide 64 text

├── cron ├── download ├── jwt ├── ldap ├── mailer │ ├── ses │ └── smtp ├── queue ├── metrics ├── redis └── storage ├── disk └── minio

Slide 65

Slide 65 text

Integration with Prometheus + Grafana

Slide 66

Slide 66 text

func NewCollector() Collector { return Collector{ Users: prometheus.NewDesc( namespace+"users", "Number of Users", nil, nil, ), } // Collect returns the metrics with values func (c Collector) Collect(ch chan<- prometheus.Metric) { stats := model.GetStatistic() ch <- prometheus.MustNewConstMetric( c.Users, prometheus.GaugeValue, ﬂoat64(stats.Counter.User), ) }

Slide 67

Slide 67 text

Prometheus Handler

Slide 68

Slide 68 text

func Metrics(token string) gin.HandlerFunc { h := promhttp.Handler() return func(c *gin.Context) { if token == "" { h.ServeHTTP(c.Writer, c.Request) return } header := c.Request.Header.Get("Authorization") if header == "" { c.String(http.StatusUnauthorized, errInvalidToken.Error()) return } bearer := fmt.Sprintf("Bearer %s", token) if header != bearer { c.String(http.StatusUnauthorized, errInvalidToken.Error()) return } h.ServeHTTP(c.Writer, c.Request) } }

Slide 69

Slide 69 text

c := metrics.NewCollector() prometheus.MustRegister(c) if conﬁg.Metrics.Enabled { root.GET("/metrics", router.Metrics(conﬁg.Metrics.Token)) } Your prometheus token

Slide 70

Slide 70 text

/pkg/schema

Slide 71

Slide 71 text

RESTful vs GraphQL See the Slide: GraphQL in Go

Slide 72

Slide 72 text

var rootQuery = graphql.NewObject( graphql.ObjectConfig{ Name: "RootQuery", Description: "Root Query", Fields: graphql.Fields{ "queryShortenURL": &queryShortenURL, "queryMe": &queryMe, }, }) var rootMutation = graphql.NewObject( graphql.ObjectConfig{ Name: "RootMutation", Description: "Root Mutation", Fields: graphql.Fields{ "createUser": &createUser, }, }) // Schema is the GraphQL schema served by the server. var Schema, _ = graphql.NewSchema( graphql.SchemaConfig{ Query: rootQuery, Mutation: rootMutation, })

Slide 73

Slide 73 text

Write the GraphQL Testing

Slide 74

Slide 74 text

assert.NoError(t, model.PrepareTestDatabase()) t.Run("user not login", func(t *testing.T) { test := T{ Query: `{ queryMe { email } }`, Schema: Schema, Expected: &graphql.Result{ Data: map[string]interface{}{ "queryMe": nil, }, Errors: []gqlerrors.FormattedError{ { Message: errorYouAreNotLogin, }, }, }, } }) }

Slide 75

Slide 75 text

Best Practice Testing your Go code

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

Testable Code • Code Quality • Readability • Maintainability • Testability

Slide 78

Slide 78 text

#1. Testing in Go func TestFooBar(t *testing.T) {} func ExampleFooBar(t *testing.T) {} func BenchmarkFooBar(b *testing.B) {} go test package_name

Slide 79

Slide 79 text

#2. Benchmark Testing Proﬁling: CPU, Memory, Goroutine Block

Slide 80

Slide 80 text

func BenchmarkPlaylyfeGraphQLMaster(b *testing.B) { for i := 0; i < b.N; i++ { context := map[string]interface{}{} variables := map[string]interface{}{} playlyfeExecutor.Execute(context, "{hello}", variables, "") } } func BenchmarkGophersGraphQLMaster(b *testing.B) { for i := 0; i < b.N; i++ { ctx := context.Background() variables := map[string]interface{}{} gopherSchema.Exec(ctx, "{hello}", "", variables) } } http://bit.ly/2L0CG3Q

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

#3. Example Testing Examples on how to use your code

Slide 83

Slide 83 text

func ExampleFooBar() { fmt.Println(strings.Compare("a", "b")) fmt.Println(strings.Compare("a", "a")) fmt.Println(strings.Compare("b", "a")) // Output: // -1 // 0 // 1 }

Slide 84

Slide 84 text

$ go test -v -tags=sqlite -run=ExampleFooBar ./pkg/model/... === RUN ExampleFooBar --- PASS: ExampleFooBar (0.00s) PASS ok github.com/go-ggz/ggz/pkg/model 0.022s

Slide 85

Slide 85 text

#4. Subtests in Testing Package func (t *T) Run(name string, f func(t *T)) bool {} func (b *B) Run(name string, f func(b *B)) bool {}

Slide 86

Slide 86 text

tests := []struct { name string fields fields args args }{} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := Collector{ Shortens: tt.fields.Shortens, Users: tt.fields.Users, } c.Describe(tt.args.ch) }) } }

Slide 87

Slide 87 text

#5. Skipping Testing t.Skip()

Slide 88

Slide 88 text

package metrics import ( "os" "testing" ) func TestSkip(t *testing.T) { if os.Getenv("DEBUG_MODE") == "true" { t.Skipf("test skipped") } }

Slide 89

Slide 89 text

#6. Running Tests in Parallel Speedup your CI/CD Flow t.Parallel()

Slide 90

Slide 90 text

func TestFooBar01(t *testing.T) { t.Parallel() time.Sleep(time.Second) } func TestFooBar02(t *testing.T) { t.Parallel() time.Sleep(time.Second * 2) } func TestFooBar03(t *testing.T) { t.Parallel() time.Sleep(time.Second * 3) }