Go at Cybozu

Go Conference 2018 Spring, 2018-04-15 at Tokyo, Japan
https://gocon.connpass.com/event/82515/

Transcript

1. Go at Cybozu @ymmt2005 Go Conference 2018 Spring
2. Me ▌@ymmt2005 ▌Love computers  and Go ☺ ▌Currently working as  Architect, and  Project manager
3. Cybozu ▌No.1 Groupware vendor in Japan  cybozu.com for Japan  kintone.com for world-wide
4. New projects at Cybozu ▌Neco  Rearchitecting cybozu.com with  Kubernetes and  CLOS + Pure L3 network ▌Yakumo  Migrate kintone.com to AWS Here I am
5. History of Go usage ▌2013-08  Start using Go to replace Python services ▌2014-02  First Go OSS: kintone Go SDK https://github.com/kintone/go-kintone ▌2016-02  Org. for Go OSS: https://github.com/cybozu-go
6. Go usage right now ▌In-house  30+ programs  40K LoC ▌Open Source  12 repositories ▌People  4 teams  20+ developers
7. Go at Yakumo ▌Yakumo consists of 50%+ Go! 50.5% ⇒
8. Why do we prefer Go? ▌Statically typed, yet compiles fast ▌Small and container-friendly executables ▌Concurrent programming made easy ▌Less to learn than C++, Java, …
9. Managing Go Code
10. Problem ▌We have a lot of Go products  using a lot of third-party libraries  to run in a production environment. ▌We should be able to:  update third-party libraries easily,  reproduce the executable reliably, and  keep enough quality for production use.
11. Our solutions 1.Mono-repository 2.Mirroring 3.Frameworks
12. Mono-repository ▌We maintain a single Git repository  to share library packages  between all programs used in-house ▌No package-local vendoring ▌Updating a library is just easy!
13. Mirroring ▌We use `git subtree` to mirror packages on Internet  including ours. ▌By mirroring, we can:  tolerate github.com failures,  tolerate repository deletions, and  reproduce the same executable.
14. Frameworks ▌No, it doesn’t mean Web frameworks. ▌We think production-grade programs should:  leak no resources,  output enough logs to shoot troubles,  have proper timeouts, and  exit / restart gracefully.
15. Frameworks (contd) ▌To help creating production-grade Go products, we have two frameworks:  github.com/cybozu-go/log standardize log fields and formats.  github.com/cybozu-go/cmd manage goroutines, logging, signals, etc. ▌Both are open source.
16. Our Go Products
17. github.com/cybozu-go/log ▌Structured logging framework (≠ library) ▌Support three formats:  plain, logfmt, JSON Lines ▌Simple and very fast  449K log/s in JSON  Plain is a bit slow because it sorts fields.
18. github.com/cybozu-go/cmd ▌ This framework imposes the use of context in virtually all goroutines. ▌ Features  Standardized signal handling  Graceful restart of servers  Logging using cybozu-go/log  etc. https://ymmt2005.hatenablog.com/entry/2016/09/03/Making_well-behaved_programs_in_Go_with_cybozu-go/cmd
19. transocks and usocksd ▌github.com/cybozu-go/transocks  Redirect outgoing TCP connections to a SOCKS or HTTP proxy transparently  Use iptables instead of LD_PRELOAD  works for programs independent of libc. ▌github.com/cybozu-go/usocksd  SOCKS4/5 server (and library)  Dynamic IP deselection using DNSBL https://ymmt2005.hatenablog.com/entry/2016/03/13/Transparent_SOCKS_proxy_in_Go_to_replace_NAT
20. github.com/cybozu-go/aptutil ▌Provide alternatives to  apt-cacher-ng  apt-mirror ▌Features  No inconsistent cache/mirror! https://ymmt2005.hatenablog.com/entry/2016/07/19/Introducing_go-apt-cacher_and_go-apt-mirror
21. github.com/cybozu-go/placemat ▌Virtual data center construction tool  A product from Neco project to simulate CLOS & BGP network  Under active development ▌Features  Stateless (unlike libvirt)  cloud-init & Ignition  UEFI boot  …
22. Vermeer ▌Not an OSS yet ▌Thumbnail generation service  supports JPEG, GIF, TIFF, PNG, BMP ▌Pure Go implementation  using github.com/disintegration/imaging  to minimize vulnerabilities
23. Logshipper ▌Not an OSS yet ▌Export logs to Kafka  at least once. ▌That’s all! https://www.slideshare.net/ShinyaUeoka/ss-78228751
24. We were so successful using Go, but
25. Journald Journald Journarld …
26. We are running Go programs as systemd services.
27. systemd unit file to run transocks [Unit] Description=transocks [Service] Type=simple Restart=on-failure ExecStart=/path/to/transocks
28. One day, transocks exited silently. systemd did not restart it.
29. This led us to HUGE service breakage.
30. What happened 1. Journald died. 2. Go got EPIPE and sent SIGPIPE to itself. 3. transocks died with SIGPIPE. 4. systemd did not restart transocks because it figured that exit with SIGPIPE is not a failure!!!
31. Journald dies ▌Journald is not PID 1. ▌So, the process may be killed, for example, by OOM killer. ▌Journald had bugs that killed it.  One bug was fixed by us:
32. Go and SIGPIPE ▌Go masks SIGPIPE to get EPIPE errors from broken sockets or pipes. ▌For stdout & stderr, Go raises SIGPIPE manually.
33. systemd and SIGPIPE ▌SuccessExitStatus directive defines what should be considered successful exit. ▌The default is exit code 0, SIGHUP, SIGINT, SIGTERM, and SIGPIPE!
34. So, Restart=on-failure did not help.
35. Our recommendations ▌Output logs to files rather than journald. ▌Adjust OOM score of journald: ▌Add this line to your service unit files: $ cat /etc/systemd/system/systemd-journald.service.d/oom_score_adj.conf [Service] OOMScoreAdjust=-1000 RestartForceExitStatus=SIGPIPE
36. Thank you for listening! Meet us at:
37. https://cybozu.connpass.com/