Errors: to log, or not to log?

Slide 1

Slide 1 text

Errors to log, or not to log? Konrad Reiche By Renée French (CC)

Slide 2

Slide 2 text

if err != nil { return nil, err }

Slide 3

Slide 3 text

if err != nil { return nil, err }

Slide 4

Slide 4 text

cmd, err := rdb.Get(ctx, "key") if err != nil { return nil, err }

Slide 5

Slide 5 text

resp, err := handler(ctx, req) middleware.go

Slide 6

Slide 6 text

resp, err := handler(ctx, req) if err != nil { log.Error(err) } middleware.go

Slide 7

Slide 7 text

[ERROR] client update: read tcp: i/o timeout [ERROR] dial tcp: lookup redis: i/o timeout [ERROR] write tcp 10.101.244.225:46470->192.168.22.191: i/o timeout [ERROR] redis: read tcp 10.101.244.225:46470->192.168.22.191:6379: i/o timeout [ERROR] fetch data: invalid parameter type: int [ERROR] context canceled [ERROR] unexpected end of JSON input [ERROR] read tcp 10.108.131.55:46078->192.168.10.169:9090: read: connection reset by peer [ERROR] internal error [ERROR] rpc error: code = Unknown desc = no actions were found to be valid [ERROR] rpc error: code = Canceled desc = context canceled [ERROR] context canceled [ERROR] context canceled [ERROR] HTTP 500: write tcp 10.108.251.133:9090->10.111.167.92:33674: i/o timeout [ERROR] rpc error: code = Unknown desc = no actions were found to be valid

Slide 8

Slide 8 text

Noisy Error Monitoring System - Time consuming to ﬁnd critical errors - Error log is ignored altogether - Logging errors in the ﬁrst place becomes pointless

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

resp, err := handler(ctx, req) if err != nil { log.Error(err) } middleware.go

Slide 12

Slide 12 text

resp, err := handler(ctx, req) if err != nil { log.Error(err) } middleware.go

Slide 13

Slide 13 text

resp, err := handler(ctx, req) if err != nil { var e net.Error if errors.As(err, &e) && e.Timeout() { timeouts.Inc() return } log.Error(err) } middleware.go

Slide 14

Slide 14 text

resp, err := handler(ctx, req) if err != nil { var e net.Error if errors.As(err, &e) && e.Timeout() { timeouts.Inc() return } if errors.Is(err, context.Canceled) { contextsCanceled.Inc() return } log.Error(err) } middleware.go

Slide 15

Slide 15 text

cmd, err := rdb.Get(ctx, "key") if err != nil { return nil, err } redis.go middleware.go ... ...

Slide 16

Slide 16 text

cmd, err := rdb.Get(ctx, "key") if err != nil { return nil, err } redis.go middleware.go ... ...

Slide 17

Slide 17 text

cmd, err := rdb.Get(ctx, "key") if err != nil { var e net.Error if errors.As(err, &e) && e.Timeout() { timeouts.Inc() // ??? } return nil, err } redis.go middleware.go ... ...

Slide 18

Slide 18 text

type loggable interface { error Log() bool }

Slide 19

Slide 19 text

type ObservedError struct { cause error } func (e ObservedError) Log() bool { return false } func (e ObservedError) Unwrap() error { return e.cause }

Slide 20

Slide 20 text

cmd, err := rdb.Get(ctx, "key") if err != nil { var e net.Error if errors.As(err, &e) && e.Timeout() { timeouts.Inc() // ??? } return nil, err } redis.go middleware.go ... ...

Slide 21

Slide 21 text

cmd, err := rdb.Get(ctx, "key") if err != nil { var e net.Error if errors.As(err, &e) && e.Timeout() { timeouts.Inc() err = metrics.NewObservedError(err) return nil, err } return nil, err } redis.go middleware.go ... ...

Slide 22

Slide 22 text

resp, err := handler(ctx, req) if err != nil { var e loggable if errors.As(err, &e) && !e.Log() { return } log.Error(err) } middleware.go

Slide 23

Slide 23 text

[ERROR] client update: read tcp: i/o timeout [ERROR] dial tcp: lookup redis: i/o timeout [ERROR] fatal error: concurrent map read and map write [ERROR] redis: read tcp 10.101.244.225:46470->192.168.22.191:6379: i/o timeout [ERROR] fetch data: invalid parameter type: int [ERROR] context canceled [ERROR] unexpected end of JSON input [ERROR] read tcp 10.108.131.55:46078->192.168.10.169:9090: read: connection reset by peer [ERROR] internal error [ERROR] rpc error: code = Unknown desc = no actions were found to be valid [ERROR] rpc error: code = Canceled desc = context canceled [ERROR] context canceled [ERROR] context canceled [ERROR] HTTP 500: write tcp 10.108.251.133:9090->10.111.167.92:33674: i/o timeout [ERROR] rpc error: code = Unknown desc = no actions were found to be valid

Slide 24

Slide 24 text

[ERROR] fatal error: concurrent map read and map write [ERROR] fetch data: invalid parameter type: int

Slide 25

Slide 25 text

Thank you [@|u/]konradreiche[.com]