Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Go at Cybozu

April 15, 2018

Go at Cybozu

Go Conference 2018 Spring, 2018-04-15 at Tokyo, Japan


April 15, 2018

More Decks by ymmt2005

Other Decks in Programming


  1. New projects at Cybozu ▌Neco  Rearchitecting cybozu.com with 

    Kubernetes and  CLOS + Pure L3 network ▌Yakumo  Migrate kintone.com to AWS Here I am
  2. History of Go usage ▌2013-08  Start using Go to

    replace Python services ▌2014-02  First Go OSS: kintone Go SDK https://github.com/kintone/go-kintone ▌2016-02  Org. for Go OSS: https://github.com/cybozu-go
  3. Go usage right now ▌In-house  30+ programs  40K

    LoC ▌Open Source  12 repositories ▌People  4 teams  20+ developers
  4. Why do we prefer Go? ▌Statically typed, yet compiles fast

    ▌Small and container-friendly executables ▌Concurrent programming made easy ▌Less to learn than C++, Java, …
  5. Problem ▌We have a lot of Go products  using

    a lot of third-party libraries  to run in a production environment. ▌We should be able to:  update third-party libraries easily,  reproduce the executable reliably, and  keep enough quality for production use.
  6. Mono-repository ▌We maintain a single Git repository  to share

    library packages  between all programs used in-house ▌No package-local vendoring ▌Updating a library is just easy!
  7. Mirroring ▌We use `git subtree` to mirror packages on Internet

     including ours. ▌By mirroring, we can:  tolerate github.com failures,  tolerate repository deletions, and  reproduce the same executable.
  8. Frameworks ▌No, it doesn’t mean Web frameworks. ▌We think production-grade

    programs should:  leak no resources,  output enough logs to shoot troubles,  have proper timeouts, and  exit / restart gracefully.
  9. Frameworks (contd) ▌To help creating production-grade Go products, we have

    two frameworks:  github.com/cybozu-go/log standardize log fields and formats.  github.com/cybozu-go/cmd manage goroutines, logging, signals, etc. ▌Both are open source.
  10. github.com/cybozu-go/log ▌Structured logging framework (≠ library) ▌Support three formats: 

    plain, logfmt, JSON Lines ▌Simple and very fast  449K log/s in JSON  Plain is a bit slow because it sorts fields.
  11. github.com/cybozu-go/cmd ▌ This framework imposes the use of context in

    virtually all goroutines. ▌ Features  Standardized signal handling  Graceful restart of servers  Logging using cybozu-go/log  etc. https://ymmt2005.hatenablog.com/entry/2016/09/03/Making_well-behaved_programs_in_Go_with_cybozu-go/cmd
  12. transocks and usocksd ▌github.com/cybozu-go/transocks  Redirect outgoing TCP connections to

    a SOCKS or HTTP proxy transparently  Use iptables instead of LD_PRELOAD  works for programs independent of libc. ▌github.com/cybozu-go/usocksd  SOCKS4/5 server (and library)  Dynamic IP deselection using DNSBL https://ymmt2005.hatenablog.com/entry/2016/03/13/Transparent_SOCKS_proxy_in_Go_to_replace_NAT
  13. github.com/cybozu-go/aptutil ▌Provide alternatives to  apt-cacher-ng  apt-mirror ▌Features 

    No inconsistent cache/mirror! https://ymmt2005.hatenablog.com/entry/2016/07/19/Introducing_go-apt-cacher_and_go-apt-mirror
  14. github.com/cybozu-go/placemat ▌Virtual data center construction tool  A product from

    Neco project to simulate CLOS & BGP network  Under active development ▌Features  Stateless (unlike libvirt)  cloud-init & Ignition  UEFI boot  …
  15. Vermeer ▌Not an OSS yet ▌Thumbnail generation service  supports

    JPEG, GIF, TIFF, PNG, BMP ▌Pure Go implementation  using github.com/disintegration/imaging  to minimize vulnerabilities
  16. Logshipper ▌Not an OSS yet ▌Export logs to Kafka 

    at least once. ▌That’s all! https://www.slideshare.net/ShinyaUeoka/ss-78228751
  17. What happened 1. Journald died. 2. Go got EPIPE and

    sent SIGPIPE to itself. 3. transocks died with SIGPIPE. 4. systemd did not restart transocks because it figured that exit with SIGPIPE is not a failure!!!
  18. Journald dies ▌Journald is not PID 1. ▌So, the process

    may be killed, for example, by OOM killer. ▌Journald had bugs that killed it.  One bug was fixed by us:
  19. Go and SIGPIPE ▌Go masks SIGPIPE to get EPIPE errors

    from broken sockets or pipes. ▌For stdout & stderr, Go raises SIGPIPE manually.
  20. systemd and SIGPIPE ▌SuccessExitStatus directive defines what should be considered

    successful exit. ▌The default is exit code 0, SIGHUP, SIGINT, SIGTERM, and SIGPIPE!
  21. Our recommendations ▌Output logs to files rather than journald. ▌Adjust

    OOM score of journald: ▌Add this line to your service unit files: $ cat /etc/systemd/system/systemd-journald.service.d/oom_score_adj.conf [Service] OOMScoreAdjust=-1000 RestartForceExitStatus=SIGPIPE