τϧςϦϦʔε·Ͱͷ Go Tips 16 த઒ ෢ݑ גࣜձࣾτϧς / αʔόʔαΠυΤϯδχΞ CA.go #2 2017.09.06

No content

IUUQTTXFFUUPSUFDPN ࡀҎ্ݶఆ

த઒ ෢ݑ TAKENORI NAKAGAWA גࣜձࣾτϧς / αʔόʔαΠυΤϯδχΞ 2016೥౓ೖࣾ ࣗݾ঺հ https://github.com/ww24

• Go on Docker 1. timezone 2. static build • Upload Images 3. Exif আڈ 4. Orientation ໰୊ 5. ύϑΥʔϚϯε • Terminate 6. Signal ϋϯυϦϯά 7. context ͱ cancel ͱ goroutine GO TIPS 16 • JSON 8. json.Marshaler 9. struct embed 10. omitempty • Credentials 11. ؀ڥม਺ 12. IAMೝূ • Logging 13. zap 14. access log • Test & Error handling 15. gomock 16. errors

Golang 1.8ܥ Docker (alpine linux image) Amazon EC2 Container Service (ECS) 1. લఏ

Go on Docker

• alpine linux
 → ca_certificates, tzdataΛผ్install • single binary
 → static build
 → go-bindata Go on Docker

• ca-certificates
 ৴པ͢Δೝূہ(CA)ͷূ໌ॻϦετ
 TLS Ͱ௨৴͢Δ࣌ʹඞཁɻ
 ※ Go ಛ༗ͷ໰୊Ͱ͸ͳ͍ Go on Docker

• timezone
 zoneinfo ΋
 $GOROOT/lib/time/zoneinfo.zip ΋
 ͲͪΒ΋ଘࡏ͠ͳ͍ͱΤϥʔ Go on Docker

• time.LoadLocation Go on Docker // LoadLocation returns the Location with the given name. // // If the name is "" or "UTC", LoadLocation returns UTC. // If the name is "Local", LoadLocation returns Local. // // Otherwise, the name is taken to be a location name corresponding to a file // in the IANA Time Zone database, such as "America/New_York". // // The time zone database needed by LoadLocation may not be // present on all systems, especially non-Unix systems. // LoadLocation looks in the directory or uncompressed zip file // named by the ZONEINFO environment variable, if any, then looks in // known installation locations on Unix systems, // and finally looks in $GOROOT/lib/time/zoneinfo.zip. func LoadLocation(name string) (*Location, error) {

• alpine Go on Docker FROM alpine:3.6 RUN apk --no-cache add ca-certificates tzdata

• static build
 Go on Docker CGO_ENABLED=0 go build -a -tags netgo \
 -installsuffix netgo -ldflags '-extldflags "-static"' ढจԽʜ

Upload Images

• Exif ফ͞ͳ͍ͱ… • ػछ৘ใ • Ґஔ৘ใ • ճస൓స৘ใ • etc… ͕࿙ΕΔ Upload Images

• JPEG decode → encode
 ͢Δ͚ͩͰ Exif ͸औΓআ͚Δ͕… • Orientation (࣮ࡍͷը૾Λදࣔ͢Δํ޲)
 ͜Ε͕ܽଛ͢Δͱճసͨ͠Γ൓సͨ͠ঢ়ଶ Ͱදࣔ͞Εͯ͠·͏ Upload Images

• Go Ͱ Exif Λ parse ͢Δํ๏ • https://github.com/golang/go/issues/4341
 ͔ͳΓલ͔Βٞ࿦͞Ε͍ͯΔ͕ਐḿ͕ͳ͍ • ౰෼͸αʔυύʔςΟ੡ͷϥΠϒϥϦΛ
 ࢖͏͔͠ແ͍ (or ࣗ࡞) Upload Images

Upload Images IUUQXXXDJQBKQTUEEPDVNFOUTK%$+QEG ■ ը૾ํ޲ Orientation ߦͱྻͷ؍఺͔Βݟͨɺը૾ͷํ޲ɻ Tag = 274 (112.H) Type = SHORT Count = 1 Default = 1 1 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷ্(visual top)ɺ0 ൪໨ͷྻ͕ࠨଆ(visual left-hand side)ͱͳΔɻ 2 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷ্ɺ0 ൪໨ͷྻ͕ӈଆ(visual right-hand side)ͱͳΔɻ 3 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷԼ(visual bottom)ɺ0 ൪໨ͷྻ͕ӈଆͱͳΔɻ 4 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷԼɺ0 ൪໨ͷྻ͕ࠨଆͱͳΔɻ 5 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷࠨଆɺ0 ൪໨ͷྻ্͕ͱͳΔɻ 6 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷӈଆɺ0 ൪໨ͷྻ্͕ͱͳΔɻ 7 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷӈଆɺ0 ൪໨ͷྻ͕ԼͱͳΔɻ 8 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷࠨଆɺ0 ൪໨ͷྻ͕ԼͱͳΔɻ ͦͷଞ = ༧໿ 1ΑΓҾ༻ • Exif ͷن֨ॻΛಡ΋͏

• Exif ͷن֨ॻΛಡ΋͏ ■ ը૾ํ޲ Orientation ߦͱྻͷ؍఺͔Βݟͨɺը૾ͷํ޲ɻ Tag = 274 (112.H) Type = SHORT Count = 1 Default = 1 1 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷ্(visual top)ɺ0 ൪໨ͷྻ͕ࠨଆ(visual left-hand side)ͱͳΔɻ 2 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷ্ɺ0 ൪໨ͷྻ͕ӈଆ(visual right-hand side)ͱͳΔɻ 3 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷԼ(visual bottom)ɺ0 ൪໨ͷྻ͕ӈଆͱͳΔɻ 4 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷԼɺ0 ൪໨ͷྻ͕ࠨଆͱͳΔɻ 5 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷࠨଆɺ0 ൪໨ͷྻ্͕ͱͳΔɻ 6 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷӈଆɺ0 ൪໨ͷྻ্͕ͱͳΔɻ 7 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷӈଆɺ0 ൪໨ͷྻ͕ԼͱͳΔɻ 8 = 0 ൪໨ͷߦ͕໨Ͱݟͨͱ͖ͷը૾ͷࠨଆɺ0 ൪໨ͷྻ͕ԼͱͳΔɻ ͦͷଞ = ༧໿ Upload Images IUUQXXXDJQBKQTUEEPDVNFOUTK%$+QEG 1ΑΓҾ༻ ೔ຊޠͳͷͰಡΈ΍͍͢ʂʂ
 ࣮૷͍ͨ͠ͱ͸ݴͬͯͳ͍

• αʔυύʔςΟ੡ϥΠϒϥϦɺπʔϧ • ImageMagick • GraphicsMagick • GD Graphics Library (libgd) • libvips • Pure Go • github.com/BurntSushi/graphics-go • github.com/anthonynsimon/bild • github.com/disintegration/imaging • github.com/disintegration/gift Upload Images

• ϕϯνϚʔΫ݁Ռ Upload Images BenchmarkGraphicsgo_LargeImage-8 1 4686575195 ns/op BenchmarkBild_LargeImage-8 3 460527153 ns/op BenchmarkImaging_LargeImage-8 10 115713079 ns/op BenchmarkGift_LargeImage-8 5 237302274 ns/op BenchmarkGraphicsgo-8 20 88962194 ns/op BenchmarkBild-8 200 7481979 ns/op BenchmarkImaging-8 500 2507617 ns/op BenchmarkGift-8 300 4335670 ns/op ࣌ܭճΓʹ౓ճసͯ͠ࠨӈ൓స͢Δͱ͍͏ॲཧΛɺ
 Ұ؟ͰࡱӨ͞Εͨେ͖͍ࣸਅ .#௒͑ ͱɺ
 খ͍͞ը૾ ,#ఔ౓ ͷ௨ΓͰࢼߦɻ

• ϕϯνϚʔΫ݁Ռ Upload Images BenchmarkGraphicsgo_LargeImage-8 1 4686575195 ns/op BenchmarkBild_LargeImage-8 3 460527153 ns/op BenchmarkImaging_LargeImage-8 10 115713079 ns/op BenchmarkGift_LargeImage-8 5 237302274 ns/op BenchmarkGraphicsgo-8 20 88962194 ns/op BenchmarkBild-8 200 7481979 ns/op BenchmarkImaging-8 500 2507617 ns/op BenchmarkGift-8 300 4335670 ns/op ࣌ܭճΓʹ౓ճసͯ͠ࠨӈ൓స͢Δͱ͍͏ॲཧΛɺ
 Ұ؟ͰࡱӨ͞Εͨେ͖͍ࣸਅ .#௒͑ ͱɺ
 খ͍͞ը૾ ,#ఔ౓ ͷ௨ΓͰࢼߦɻ github.com/disintegration/imaging
 ࠾༻

Upload Images func applyOrientation(img image.Image, meta *exif.Exif) (image.Image, error) { tag, err := meta.Get(exif.Orientation) if err != nil { return nil, errors.WithStack(err) } orientation, err := tag.Int(0) if err != nil { return nil, errors.WithStack(err) } switch orientation { case 1: return img, nil case 2: img = imaging.FlipH(img) case 3: img = imaging.FlipH(img) img = imaging.FlipV(img) case 4: img = imaging.FlipV(img) case 5: img = imaging.Rotate270(img) img = imaging.FlipH(img) case 6: img = imaging.Rotate270(img) case 7: img = imaging.Rotate90(img) img = imaging.FlipH(img) case 8: img = imaging.Rotate90(img) default: return nil, errors.New("invalid orientation") } }

Terminate

• ϓϩηε͸͍ͭऴྃ͞ΕΔ͔Θ͔Βͳ͍ • API ͸جຊతʹ͍ͭࢮΜͰ΋େৎ෉ͳΑ͏ ʹઃܭ͞Ε͍ͯΔ
 (جຊతʹεςʔτϨεͰτϥϯβΫγϣϯॲཧ͕͋Δ) • Batch ͸్தͰࢮ͵ͱࠔΔ Terminate 4JHOBM)BOEMJOH

• Signal ͕ඈΜͰ͖ͨ࣌ʹऴྃॲཧΛߦ͏ Terminate func terminateHandler(cancel func()) { sigCh := make(chan os.Signal) signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGHUP, syscall.SIGINT, syscall.SIGQUIT, ) go func() { sig := <-sigCh log.Printf("Terminated by SIGNAL: %v\n", sig) cancel() }() } ˞syscall.SIGKILL͸ϋϯυϦϯάͰ͖ͳ͍

Terminate func main() { ctx, cancel := context.WithCancel(context.Background()) terminateHandler(cancel) } func exec(ctx context.Context) { ctx, cancel := context.WithTimeout(ctx, execTimeout) defer cancel() ch := make(chan *Record, channelBufferSize) go fetchRecords(ctx, ch) for { select { case <-ctx.Done(): return case record := <-ch: // 1Ϩίʔυͣͭߦ͏ԿΒ͔ͷॲཧ } } }

Terminate func fetchRecords(ctx context.Context, ch chan<- *Record) (err error) { defer func() { if cause := recover(); cause != nil { if e, ok := cause.(error); ok { err = e } else { // Τϥʔϩάͱ͔ } } }() // ࣮ࡍʹϨίʔυΛऔͬͯ͘Δॲཧ }

JSON

• json.Marshaler Λ࣮૷ • MarshalJSON ͷதͰ json.Marshal ΛݺͿ ͱ stack overflow ͢Δ࣌ͷରॲ๏ JSON IUUQTQMBZHPMBOHPSHQ6X"&-R

JSON IUUQTQMBZHPMBOHPSHQ6X"&-R • json.Marshaler Λ࣮૷ͨ͠৔߹ʹ
 stack overflow ͢Δ࣌ͷରॲ๏ package main import ( "encoding/json" "fmt" ) type Obj struct { Text string } func (o *Obj) MarshalJSON() ([]byte, error) { // ԿΒ͔ͷॲཧ type alias Obj return json.Marshal((*alias)(o)) } func main() { obj := &Obj{ Text: "json.Marshaler", } d, _ := json.Marshal(obj) fmt.Println(string(d)) }

JSON IUUQTQMBZHPMBOHPSHQ6X"&-R • json.Marshaler Λ࣮૷ͨ͠৔߹ʹ
 stack overflow ͢Δ࣌ͷରॲ๏ package main import ( "encoding/json" "fmt" ) type Obj struct { Text string } func (o *Obj) MarshalJSON() ([]byte, error) { // ԿΒ͔ͷॲཧ type alias Obj return json.Marshal((*alias)(o)) } func main() { obj := &Obj{ Text: "json.Marshaler", } d, _ := json.Marshal(obj) fmt.Println(string(d)) }

• struct ͷຒΊࠐΈ • struct in struct ͸ flat • interface in struct ͸ nest IUUQTQMBZHPMBOHPSHQ[Q'(*XS(X' JSON

• struct ͷຒΊࠐΈ • struct in struct ͸ flat • interface in struct ͸ nest package main import ( "encoding/json" "fmt" ) type Interface interface{} type OuterA struct { Interface Property string } type InnerA struct { InnerProperty string } func main() { objA := &OuterA{ Interface: &InnerA{InnerProperty: "inner"}, Property: "outer", } d, _ := json.Marshal(objA) fmt.Println(string(d)) } IUUQTQMBZHPMBOHPSHQ[Q'(*XS(X' JSON

• struct in struct ͸ flatten ͞ΕΔ • interface in struct ͸ nest ͞ΕΔ • ͑͑ͬʂʁ package main type Interface interface{} type OuterA struct { Interface Property string } type InnerA struct { InnerProperty string } func main() { objA := &OuterA{ Interface: &InnerA{InnerProperty: "inner"}, Property: "outer", } d, _ := json.Marshal(objA) fmt.Println(string(d)) } IUUQTQMBZHPMBOHPSHQ[Q'(*XS(X' { "Interface": { "InnerProperty": "inner" }, "Property": "outer" } JSON ͑͑ͬʂʁ

• time.Time ͸ omitempty Ͱ͖ͳ͍ • https://github.com/golang/go/issues/4357 • https://github.com/golang/go/issues/11939 • ͭΒ͍ • ྑ͘ͳ͍͚Ͳ
 *time.Time ࢖͏͔͠ແͦ͞͏ JSON

Credentials

• ೝূ৘ใΛͲ͜ʹ͔࣋ͭ • ౰ॳ: ؀ڥม਺
 → TaskDefinition, Terraform ʹ΂ͨॻ͖
 → ΋ͪΖΜฏจͰ Credentials

• EC2 System Manager Parameter Store
 → AWS ͳΒ͜ΕҰ୒ʁ
 → KMS ͷ伴Ͱ҉߸Խͯ͘͠ΕΔ • ଞͳΒ etcd, consul ͱ͍͏બ୒ࢶ΋ • github.com/vrischmann/envconfig Λ wrap ͯ͠ೝূ৘ใ͚ͩ Parameter Store ࢀর Credentials

• ೝূϦΫΤετ • Amazon Elasticsearch Service
 ͳͲΛ࢖͏࣌ʹ Signing AWS API Request ʹ ै͏ඞཁ͕͋Δ • ରԠ͍ͯ͠ͳ͍αʔυύʔςΟͷΫϥΠΞϯτ Λͦͷ··࢖͏ͳΒ http.RoundTripper Λ
 ࣮૷͢Δ Credentials

Credentials // Transport implement http.RoundTripper. type Transport struct { Base http.RoundTripper } func (t *Transport) base() http.RoundTripper { if t.Base != nil { return t.Base } return http.DefaultTransport } // RoundTrip wraps http.DefaultTransport.RoundTrip for AWS signed request. func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error) { // clone request body var body io.ReadSeeker buff := &bytes.Buffer{} if req.Body != nil { buff.ReadFrom(req.Body) req.Body = ioutil.NopCloser(buff) body = bytes.NewReader(buff.Bytes()) } signer := v4.NewSigner(credentials.AWSSession.Config.Credentials) _, err := signer.Sign(req, body, awsServiceName, awsRegion, time.Now()) if err != nil { return nil, err } return t.base().RoundTrip(req) }

ͪͳΈʹ…
 དྷि Elasticsearch ͷษڧձͰొஃ͠·͢ʂ Credentials IUUQTDZCFSBHFOUDPOOQBTTDPNFWFOU

Credentials ൃදλΠτϧ
 ʰτϧς͕࣮ફͨ͠Ϛονͨ͠ϢʔβʔΛআͭ͘ͷํ๏ʱ

Logging

• zap • JSON ܗࣜͷϩά • ͍Ζ͍ΖͭΒ͔ͬͨ json package Λ࢖͏ ඞཁ͕ͳ͍ • ύϑΥʔϚϯεʹ༏Ε͍ͯΔ Logging HJUIVCDPNVCFSHP[BQ

Logging logger, _ := zap.NewProduction() defer logger.Sync() logger.Info("failed to fetch URL", zap.String("url", url), zap.Int("attempt", 3), zap.Duration("backoff", time.Second), )

• access/app log • RequestID Λੜ੒ͯ͠෇༩͢Δ • ϨεϙϯεͰ X-Request-ID ϔομΛฦ͢ • 1ͭͷϦΫΤετͷྲྀΕ͕௥͑ΔͷͰɺ
 σόοάָ͕ʹʂ Logging

Test & Error handling

• gomock • interface Λݩʹ mock Λੜ੒ • ಛʹ࣍ͷΑ͏ͳ֎෦ͷঢ়ଶʹґଘ͢Δ΋ͷΛ
 mock Խ͢Δͱศར • σʔλετΞपΓ • ֎෦ API पΓ • ཚ਺ͳͲ • ςετͰ༗༻ Test & Error handling HJUIVCDPNHPMBOHNPDL

• gomock • interface Λݩʹ mock Λੜ੒ • ಛʹ࣍ͷΑ͏ͳ֎෦ͷঢ়ଶʹґଘ͢Δ΋ͷΛ
 mock Խ͢Δͱศར • σʔλετΞपΓ • ֎෦ API पΓ • ཚ਺ͳͲ • ςετͰ༗༻ Test & Error handling HJUIVCDPNHPMBOHNPDL package s3 import "io" type Client interface { Upload(bucket string, key string, file io.ReadSeeker) error } type implement struct{} func (s *implement) Upload(bucket string, key string, file io.ReadSeeker) error { // implement return nil } func New() Client { return &implement{} }

• gomock • interface Λݩʹ mock Λੜ੒ • ಛʹ࣍ͷΑ͏ͳ֎෦ͷঢ়ଶʹґଘ͢Δ΋ͷΛ
 mock Խ͢Δͱศར • σʔλετΞपΓ • ֎෦ API पΓ • ཚ਺ͳͲ • ςετͰ༗༻ Test & Error handling HJUIVCDPNHPMBOHNPDL package main import // தུ const bucketName = "{bucket_name}" var s3Client = s3.New() func UploadImage(fileName string, file io.ReadSeeker) error { fileHeader := make([]byte, 512) file.Read(fileHeader) mimeType := http.DetectContentType(fileHeader) _, err := file.Seek(0, io.SeekStart) if err != nil { return err } var ext string switch mimeType { case "image/png": ext = ".png" case "image/jpeg": ext = ".jpg" default: return errors.New("unsupported file type") } return s3Client.Upload(bucketName, fileName+ext, file) }

• gomock • interface Λݩʹ mock Λੜ੒ • ಛʹ࣍ͷΑ͏ͳ֎෦ͷঢ়ଶʹґଘ͢Δ΋ͷΛ
 mock Խ͢Δͱศར • σʔλετΞपΓ • ֎෦ API पΓ • ཚ਺ͳͲ • ςετͰ༗༻ Test & Error handling HJUIVCDPNHPMBOHNPDL package main import ( "os" "path/filepath" "testing" // தུ "github.com/golang/mock/gomock" ) func TestUploadImage(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() mockS3Client := s3.NewMockClient(ctrl) mockS3Client.EXPECT().Upload( bucketName, "test.jpg", gomock.Any(), ).Return(nil) s3Client = mockS3Client filePath, _ := filepath.Abs("testdata/test.jpg") file, _ := os.Open(filePath) err := UploadImage("test", file) if err != nil { t.Error(err) } }

• Error handling • ඪ४ͷ errors package ͷ୅ΘΓʹ࢖͏ • ಛʹ WithStack ͱ͍͏ϝιου͕ศར • ඪ४ͷ error Λड͚औΔՕॴͰ WithStack Ͱ stack trace Λ෇༩͢Δ͜ͱͰϩά͔ΒΤϥʔ ൃੜՕॴ͕෼͔ΓσόοάḿΔ Test & Error handling HJUIVCDPNQLHFSSPST

• Error handling • ඪ४ͷ errors package ͷ୅ΘΓʹ࢖͏ • ಛʹ WithStack ͱ͍͏ϝιου͕ศར • ඪ४ͷ error Λड͚औΔՕॴͰ WithStack Ͱ stack trace Λ෇༩͢Δ͜ͱͰϩά͔ΒΤϥʔ ൃੜՕॴ͕෼͔ΓσόοάḿΔ Test & Error handling HJUIVCDPNQLHFSSPST package main import ( "strconv" "github.com/pkg/errors" ) func main() { } func errorMethod() error { i, err := strconv.Atoi("") if err != nil { return errors.WithStack(err) } return nil }

• Error handling • ඪ४ͷ errors package ͷ୅ΘΓʹ࢖͏ • ಛʹ WithStack ͱ͍͏ϝιου͕ศར • ඪ४ͷ error Λड͚औΔՕॴͰ WithStack Ͱ stack trace Λ෇༩͢Δ͜ͱͰϩά͔ΒΤϥʔ ൃੜՕॴ͕෼͔ΓσόοάḿΔ Test & Error handling HJUIVCDPNQLHFSSPST package main import ( "strconv" "github.com/pkg/errors" ) func main() { } func errorMethod() error { i, err := strconv.Atoi("") if err != nil { return errors.WithStack(err) } return nil } TUSDPOW"UPJQBSTJOHJOWBMJETZOUBY NBJOFSSPS.FUIPE 6TFSTXXHPTSDFSSPSTNBJOHP NBJONBJO 6TFSTXXHPTSDFSSPSTNBJOHP SVOUJNFNBJO VTSMPDBM$FMMBSHPMJCFYFDTSDSVOUJNFQSPDHP SVOUJNFHPFYJU VTSMPDBM$FMMBSHPMJCFYFDTSDSVOUJNFBTN@BNET

·ͱΊ

͜͜·Ͱొ৔͠·ͤΜͰ͕ͨ͠ɺ • ϑϨʔϜϫʔΫ • echo (github.com/labstack/echo) • package ؅ཧ • glide (github.com/Masterminds/glide) • ORM • xorm (github.com/go-xorm/xorm) ࢖ͬͯ·͢ɻ ·ͱΊ

• εϥΠυதͰ঺հͨ͠ package Ұཡ • github.com/disintegration/imaging • github.com/vrischmann/envconfig • github.com/uber-go/zap • github.com/golang/mock • github.com/pkg/errors ·ͱΊ

THANK YOU !