Go at Cybozu

Go Conference 2018 Spring, 2018-04-15 at Tokyo, Japan
https://gocon.connpass.com/event/82515/

Transcript

1. Go at Cybozu
   @ymmt2005
   Go Conference 2018 Spring
   

   View Slide

2. Me
   ▌@ymmt2005
   ▌Love computers
    and Go ☺
   ▌Currently working as
    Architect, and
    Project manager
   

   View Slide

3. Cybozu
   ▌No.1 Groupware vendor in Japan
    cybozu.com for Japan
    kintone.com for world-wide
   

   View Slide

4. New projects at Cybozu
   ▌Neco
    Rearchitecting cybozu.com with
    Kubernetes and
    CLOS + Pure L3 network
   ▌Yakumo
    Migrate kintone.com to AWS
   Here I am
   

   View Slide

5. History of Go usage
   ▌2013-08
    Start using Go to replace Python services
   ▌2014-02
    First Go OSS: kintone Go SDK
   https://github.com/kintone/go-kintone
   ▌2016-02
    Org. for Go OSS: https://github.com/cybozu-go
   

   View Slide

6. Go usage right now
   ▌In-house
    30+ programs
    40K LoC
   ▌Open Source
    12 repositories
   ▌People
    4 teams
    20+ developers
   

   View Slide

7. Go at Yakumo
   ▌Yakumo consists of 50%+ Go!
   50.5% ⇒
   

   View Slide

8. Why do we prefer Go?
   ▌Statically typed, yet compiles fast
   ▌Small and container-friendly executables
   ▌Concurrent programming made easy
   ▌Less to learn than C++, Java, …
   

   View Slide

9. Managing Go Code
   

   View Slide

10. Problem
    ▌We have a lot of Go products
     using a lot of third-party libraries
     to run in a production environment.
    ▌We should be able to:
     update third-party libraries easily,
     reproduce the executable reliably, and
     keep enough quality for production use.
    

    View Slide

11. Our solutions
    1.Mono-repository
    2.Mirroring
    3.Frameworks
    

    View Slide

12. Mono-repository
    ▌We maintain a single Git repository
     to share library packages
     between all programs used in-house
    ▌No package-local vendoring
    ▌Updating a library is just easy!
    

    View Slide

13. Mirroring
    ▌We use `git subtree` to mirror packages on Internet
     including ours.
    ▌By mirroring, we can:
     tolerate github.com failures,
     tolerate repository deletions, and
     reproduce the same executable.
    

    View Slide

14. Frameworks
    ▌No, it doesn’t mean Web frameworks.
    ▌We think production-grade programs should:
     leak no resources,
     output enough logs to shoot troubles,
     have proper timeouts, and
     exit / restart gracefully.
    

    View Slide

15. Frameworks (contd)
    ▌To help creating production-grade Go products,
    we have two frameworks:
     github.com/cybozu-go/log
    standardize log fields and formats.
     github.com/cybozu-go/cmd
    manage goroutines, logging, signals, etc.
    ▌Both are open source.
    

    View Slide

16. Our Go Products
    

    View Slide

17. github.com/cybozu-go/log
    ▌Structured logging framework (≠ library)
    ▌Support three formats:
     plain, logfmt, JSON Lines
    ▌Simple and very fast
     449K log/s in JSON
     Plain is a bit slow
    because it sorts fields.
    

    View Slide

18. github.com/cybozu-go/cmd
    ▌ This framework imposes the use of
    context in virtually all goroutines.
    ▌ Features
     Standardized signal handling
     Graceful restart of servers
     Logging using cybozu-go/log
     etc.
    https://ymmt2005.hatenablog.com/entry/2016/09/03/Making_well-behaved_programs_in_Go_with_cybozu-go/cmd
    

    View Slide

19. transocks and usocksd
    ▌github.com/cybozu-go/transocks
     Redirect outgoing TCP connections
    to a SOCKS or HTTP proxy transparently
     Use iptables instead of LD_PRELOAD
     works for programs independent of libc.
    ▌github.com/cybozu-go/usocksd
     SOCKS4/5 server (and library)
     Dynamic IP deselection using DNSBL
    https://ymmt2005.hatenablog.com/entry/2016/03/13/Transparent_SOCKS_proxy_in_Go_to_replace_NAT
    

    View Slide

20. github.com/cybozu-go/aptutil
    ▌Provide alternatives to
     apt-cacher-ng
     apt-mirror
    ▌Features
     No inconsistent cache/mirror!
    https://ymmt2005.hatenablog.com/entry/2016/07/19/Introducing_go-apt-cacher_and_go-apt-mirror
    

    View Slide

21. github.com/cybozu-go/placemat
    ▌Virtual data center construction tool
     A product from Neco project to
    simulate CLOS & BGP network
     Under active development
    ▌Features
     Stateless (unlike libvirt)
     cloud-init & Ignition
     UEFI boot
     …
    

    View Slide

22. Vermeer
    ▌Not an OSS yet
    ▌Thumbnail generation service
     supports JPEG, GIF, TIFF, PNG, BMP
    ▌Pure Go implementation
     using github.com/disintegration/imaging
     to minimize vulnerabilities
    

    View Slide

23. Logshipper
    ▌Not an OSS yet
    ▌Export logs to Kafka
     at least once.
    ▌That’s all!
    https://www.slideshare.net/ShinyaUeoka/ss-78228751
    

    View Slide

24. We were so successful
    using Go, but
    

    View Slide

25. Journald
    Journald
    Journarld
    …
    

    View Slide

26. We are running Go programs
    as systemd services.
    

    View Slide

27. systemd unit file to run transocks
    [Unit]
    Description=transocks
    [Service]
    Type=simple
    Restart=on-failure
    ExecStart=/path/to/transocks
    

    View Slide

28. One day, transocks exited silently.
    systemd did not restart it.
    

    View Slide

29. This led us to HUGE service breakage.
    

    View Slide

30. What happened
    1. Journald died.
    2. Go got EPIPE and sent SIGPIPE to itself.
    3. transocks died with SIGPIPE.
    4. systemd did not restart transocks
    because it figured that exit with SIGPIPE
    is not a failure!!!
    

    View Slide

31. Journald dies
    ▌Journald is not PID 1.
    ▌So, the process may be killed,
    for example, by OOM killer.
    ▌Journald had bugs that killed it.
     One bug was fixed by us:
    

    View Slide

32. Go and SIGPIPE
    ▌Go masks SIGPIPE to get EPIPE errors from broken
    sockets or pipes.
    ▌For stdout & stderr, Go raises SIGPIPE manually.
    

    View Slide

33. systemd and SIGPIPE
    ▌SuccessExitStatus directive defines what should be
    considered successful exit.
    ▌The default is exit code 0, SIGHUP, SIGINT, SIGTERM, and
    SIGPIPE!
    

    View Slide

34. So, Restart=on-failure did not help.
    

    View Slide

35. Our recommendations
    ▌Output logs to files rather than journald.
    ▌Adjust OOM score of journald:
    ▌Add this line to your service unit files:
    $ cat /etc/systemd/system/systemd-journald.service.d/oom_score_adj.conf
    [Service]
    OOMScoreAdjust=-1000
    RestartForceExitStatus=SIGPIPE
    

    View Slide

36. Thank you for listening!
    Meet us at:
    

    View Slide

37. https://cybozu.connpass.com/
    

    View Slide