Go Conference 2018 Spring, 2018-04-15 at Tokyo, Japan
Go at Cybozu
Go Conference 2018 Spring
and Go ☺
▌Currently working as
▌No.1 Groupware vendor in Japan
cybozu.com for Japan
kintone.com for world-wide
New projects at Cybozu
Rearchitecting cybozu.com with
CLOS + Pure L3 network
Migrate kintone.com to AWS
Here I am
History of Go usage
Start using Go to replace Python services
First Go OSS: kintone Go SDK
Org. for Go OSS: https://github.com/cybozu-go
Go usage right now
Go at Yakumo
▌Yakumo consists of 50%+ Go!
Why do we prefer Go?
▌Statically typed, yet compiles fast
▌Small and container-friendly executables
▌Concurrent programming made easy
▌Less to learn than C++, Java, …
Managing Go Code
▌We have a lot of Go products
using a lot of third-party libraries
to run in a production environment.
▌We should be able to:
update third-party libraries easily,
reproduce the executable reliably, and
keep enough quality for production use.
▌We maintain a single Git repository
to share library packages
between all programs used in-house
▌No package-local vendoring
▌Updating a library is just easy!
▌We use `git subtree` to mirror packages on Internet
▌By mirroring, we can:
tolerate github.com failures,
tolerate repository deletions, and
reproduce the same executable.
▌No, it doesn’t mean Web frameworks.
▌We think production-grade programs should:
leak no resources,
output enough logs to shoot troubles,
have proper timeouts, and
exit / restart gracefully.
▌To help creating production-grade Go products,
we have two frameworks:
standardize log fields and formats.
manage goroutines, logging, signals, etc.
▌Both are open source.
Our Go Products
▌Structured logging framework (≠ library)
▌Support three formats:
plain, logfmt, JSON Lines
▌Simple and very fast
449K log/s in JSON
Plain is a bit slow
because it sorts fields.
▌ This framework imposes the use of
context in virtually all goroutines.
Standardized signal handling
Graceful restart of servers
Logging using cybozu-go/log
transocks and usocksd
Redirect outgoing TCP connections
to a SOCKS or HTTP proxy transparently
Use iptables instead of LD_PRELOAD
works for programs independent of libc.
SOCKS4/5 server (and library)
Dynamic IP deselection using DNSBL
▌Provide alternatives to
No inconsistent cache/mirror!
▌Virtual data center construction tool
A product from Neco project to
simulate CLOS & BGP network
Under active development
Stateless (unlike libvirt)
cloud-init & Ignition
▌Not an OSS yet
▌Thumbnail generation service
supports JPEG, GIF, TIFF, PNG, BMP
▌Pure Go implementation
to minimize vulnerabilities
▌Not an OSS yet
▌Export logs to Kafka
at least once.
We were so successful
using Go, but
We are running Go programs
as systemd services.
systemd unit file to run transocks
One day, transocks exited silently.
systemd did not restart it.
This led us to HUGE service breakage.
1. Journald died.
2. Go got EPIPE and sent SIGPIPE to itself.
3. transocks died with SIGPIPE.
4. systemd did not restart transocks
because it figured that exit with SIGPIPE
is not a failure!!!
▌Journald is not PID 1.
▌So, the process may be killed,
for example, by OOM killer.
▌Journald had bugs that killed it.
One bug was fixed by us:
Go and SIGPIPE
▌Go masks SIGPIPE to get EPIPE errors from broken
sockets or pipes.
▌For stdout & stderr, Go raises SIGPIPE manually.
systemd and SIGPIPE
▌SuccessExitStatus directive defines what should be
considered successful exit.
▌The default is exit code 0, SIGHUP, SIGINT, SIGTERM, and
So, Restart=on-failure did not help.
▌Output logs to files rather than journald.
▌Adjust OOM score of journald:
▌Add this line to your service unit files:
$ cat /etc/systemd/system/systemd-journald.service.d/oom_score_adj.conf
Thank you for listening!
Meet us at: