Go at Cybozu

Ae03eb47606f4e0125268d39f8aebfad?s=47 ymmt2005
April 15, 2018

Go at Cybozu

Go Conference 2018 Spring, 2018-04-15 at Tokyo, Japan
https://gocon.connpass.com/event/82515/

Ae03eb47606f4e0125268d39f8aebfad?s=128

ymmt2005

April 15, 2018
Tweet

Transcript

  1. Go at Cybozu @ymmt2005 Go Conference 2018 Spring

  2. Me ▌@ymmt2005 ▌Love computers  and Go ☺ ▌Currently working

    as  Architect, and  Project manager
  3. Cybozu ▌No.1 Groupware vendor in Japan  cybozu.com for Japan

     kintone.com for world-wide
  4. New projects at Cybozu ▌Neco  Rearchitecting cybozu.com with 

    Kubernetes and  CLOS + Pure L3 network ▌Yakumo  Migrate kintone.com to AWS Here I am
  5. History of Go usage ▌2013-08  Start using Go to

    replace Python services ▌2014-02  First Go OSS: kintone Go SDK https://github.com/kintone/go-kintone ▌2016-02  Org. for Go OSS: https://github.com/cybozu-go
  6. Go usage right now ▌In-house  30+ programs  40K

    LoC ▌Open Source  12 repositories ▌People  4 teams  20+ developers
  7. Go at Yakumo ▌Yakumo consists of 50%+ Go! 50.5% ⇒

  8. Why do we prefer Go? ▌Statically typed, yet compiles fast

    ▌Small and container-friendly executables ▌Concurrent programming made easy ▌Less to learn than C++, Java, …
  9. Managing Go Code

  10. Problem ▌We have a lot of Go products  using

    a lot of third-party libraries  to run in a production environment. ▌We should be able to:  update third-party libraries easily,  reproduce the executable reliably, and  keep enough quality for production use.
  11. Our solutions 1.Mono-repository 2.Mirroring 3.Frameworks

  12. Mono-repository ▌We maintain a single Git repository  to share

    library packages  between all programs used in-house ▌No package-local vendoring ▌Updating a library is just easy!
  13. Mirroring ▌We use `git subtree` to mirror packages on Internet

     including ours. ▌By mirroring, we can:  tolerate github.com failures,  tolerate repository deletions, and  reproduce the same executable.
  14. Frameworks ▌No, it doesn’t mean Web frameworks. ▌We think production-grade

    programs should:  leak no resources,  output enough logs to shoot troubles,  have proper timeouts, and  exit / restart gracefully.
  15. Frameworks (contd) ▌To help creating production-grade Go products, we have

    two frameworks:  github.com/cybozu-go/log standardize log fields and formats.  github.com/cybozu-go/cmd manage goroutines, logging, signals, etc. ▌Both are open source.
  16. Our Go Products

  17. github.com/cybozu-go/log ▌Structured logging framework (≠ library) ▌Support three formats: 

    plain, logfmt, JSON Lines ▌Simple and very fast  449K log/s in JSON  Plain is a bit slow because it sorts fields.
  18. github.com/cybozu-go/cmd ▌ This framework imposes the use of context in

    virtually all goroutines. ▌ Features  Standardized signal handling  Graceful restart of servers  Logging using cybozu-go/log  etc. https://ymmt2005.hatenablog.com/entry/2016/09/03/Making_well-behaved_programs_in_Go_with_cybozu-go/cmd
  19. transocks and usocksd ▌github.com/cybozu-go/transocks  Redirect outgoing TCP connections to

    a SOCKS or HTTP proxy transparently  Use iptables instead of LD_PRELOAD  works for programs independent of libc. ▌github.com/cybozu-go/usocksd  SOCKS4/5 server (and library)  Dynamic IP deselection using DNSBL https://ymmt2005.hatenablog.com/entry/2016/03/13/Transparent_SOCKS_proxy_in_Go_to_replace_NAT
  20. github.com/cybozu-go/aptutil ▌Provide alternatives to  apt-cacher-ng  apt-mirror ▌Features 

    No inconsistent cache/mirror! https://ymmt2005.hatenablog.com/entry/2016/07/19/Introducing_go-apt-cacher_and_go-apt-mirror
  21. github.com/cybozu-go/placemat ▌Virtual data center construction tool  A product from

    Neco project to simulate CLOS & BGP network  Under active development ▌Features  Stateless (unlike libvirt)  cloud-init & Ignition  UEFI boot  …
  22. Vermeer ▌Not an OSS yet ▌Thumbnail generation service  supports

    JPEG, GIF, TIFF, PNG, BMP ▌Pure Go implementation  using github.com/disintegration/imaging  to minimize vulnerabilities
  23. Logshipper ▌Not an OSS yet ▌Export logs to Kafka 

    at least once. ▌That’s all! https://www.slideshare.net/ShinyaUeoka/ss-78228751
  24. We were so successful using Go, but

  25. Journald Journald Journarld …

  26. We are running Go programs as systemd services.

  27. systemd unit file to run transocks [Unit] Description=transocks [Service] Type=simple

    Restart=on-failure ExecStart=/path/to/transocks
  28. One day, transocks exited silently. systemd did not restart it.

  29. This led us to HUGE service breakage.

  30. What happened 1. Journald died. 2. Go got EPIPE and

    sent SIGPIPE to itself. 3. transocks died with SIGPIPE. 4. systemd did not restart transocks because it figured that exit with SIGPIPE is not a failure!!!
  31. Journald dies ▌Journald is not PID 1. ▌So, the process

    may be killed, for example, by OOM killer. ▌Journald had bugs that killed it.  One bug was fixed by us:
  32. Go and SIGPIPE ▌Go masks SIGPIPE to get EPIPE errors

    from broken sockets or pipes. ▌For stdout & stderr, Go raises SIGPIPE manually.
  33. systemd and SIGPIPE ▌SuccessExitStatus directive defines what should be considered

    successful exit. ▌The default is exit code 0, SIGHUP, SIGINT, SIGTERM, and SIGPIPE!
  34. So, Restart=on-failure did not help.

  35. Our recommendations ▌Output logs to files rather than journald. ▌Adjust

    OOM score of journald: ▌Add this line to your service unit files: $ cat /etc/systemd/system/systemd-journald.service.d/oom_score_adj.conf [Service] OOMScoreAdjust=-1000 RestartForceExitStatus=SIGPIPE
  36. Thank you for listening! Meet us at:

  37. https://cybozu.connpass.com/