Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Project layout patterns in Go

Project layout patterns in Go

You completed the tour, learned the language, got your pet project on Github. Time to start a new project, a big one: how do you organize code, tests, docs? During the talk I’ll try to save you some trial and error by sharing the lessons learned during a year spent coding a non trivial Go project.

Massimiliano Pippi

October 22, 2018
Tweet

More Decks by Massimiliano Pippi

Other Decks in Programming

Transcript

  1. Project layout patterns in Go
    Massimiliano Pippi

    View full-size slide

  2. ● Engineering team lead
    ● 2+ year at Datadog, working on Agent and Integrations
    ● Python and Go
    ● OSS fan and contributor
    Hi, I’m Massi!

    View full-size slide

  3. What’s this about
    ● How we laid out one of our Go projects
    ○ Code is Open Source
    ○ ~70k lines of Go
    ○ ~10 fulltime contributors
    ○ 11 teams involved
    ○ Packaged, not deployed
    ● Your mileage may vary

    View full-size slide

  4. Trial and error
    A. Problem
    B. Solution n.1
    C. Lessons learned
    D. Solution n.2

    View full-size slide

  5. Trial and error
    A. Problem
    B. Solution n.1
    C. Lessons learned
    D. Solution n.2
    ● Deadline too tight
    ● Changing project layout impacts code
    ● Couldn’t afford a second take

    View full-size slide

  6. When in Rome...

    View full-size slide

  7. When in Rome...
    ● Looked at the Go state of the art
    ○ Kubernetes
    ○ Etcd
    ○ CockroachDB

    View full-size slide

  8. When in Rome...
    ● Looked at the Go state of the art
    ○ Kubernetes
    ○ Etcd
    ○ CockroachDB
    ● Shoot at one of similar size
    ○ Consider overhead...
    ○ ...but you might need it
    ○ Consider the amount of tooling that a layout requires

    View full-size slide

  9. Code layout
    ● Project must be easy to browse
    ○ Keep the root folder clean and sleek

    View full-size slide

  10. Code layout
    ● Project must be easy to browse
    ○ Keep the root folder clean and sleek
    ● Don’t make contributors guess where things are
    ○ Good
    ■ /docs /tests /releasenotes
    ○ Bad
    ■ /extra/build/docker/Dockerfile /common/docs
    ○ Evil
    ■ /src/bin

    View full-size slide

  11. Code layout
    ● Use packages!
    ○ They help you designing the API
    ○ They ease code reuse
    ○ They’re a powerful abstraction tool
    ○ Go stdlib is the best place to lurk

    View full-size slide

  12. Code layout
    ● Use packages!
    ○ They help you designing the API
    ○ They ease code reuse
    ○ They’re a powerful abstraction tool
    ○ Go stdlib is the best place to lurk
    ● If packages are well designed, the source tree will reflect it

    View full-size slide

  13. Code layout
    ● Partition the codebase

    View full-size slide

  14. Code layout
    ● Partition the codebase
    ○ The Framework: /pkg/
    ○ The Programs: /cmd/

    View full-size slide

  15. Code layout
    ● Partition the codebase
    ○ The Framework: /pkg/
    ■ /pkg/utils/, /pkg/config/, ...
    ■ No dependencies on /cmd/
    ■ Can be imported by external programs
    ○ The Programs: /cmd/

    View full-size slide

  16. Code layout
    ● Partition the codebase
    ○ The Framework: /pkg/
    ■ /pkg/utils/, /pkg/config/, ...
    ■ No dependencies on /cmd/
    ■ Can be imported by external programs
    ○ The Programs: /cmd/
    ■ /cmd/superduper, /cmd/miniduper, …
    ■ Domain specific logic stays close to the main() func

    View full-size slide

  17. Keep your tooling sharp

    View full-size slide

  18. Keep your tooling sharp
    ● Goals
    ○ Make obvious and robust how to build and run tests

    View full-size slide

  19. Keep your tooling sharp
    ● Goals
    ○ Make obvious and robust how to build and run tests
    ● Constraints
    ○ Make CI robots use what humans use
    ○ Tools orchestrate, don’t implement: bypassing tooling
    should always be possible
    ○ Keep number of dependencies low

    View full-size slide

  20. Keep your tooling sharp
    ● Make
    ○ Can anybody do make?
    ○ Used as task runner can get complicated

    View full-size slide

  21. Keep your tooling sharp
    ● Make
    ○ Can anybody do make?
    ○ Used as task runner can get complicated
    ● Make alternatives
    ○ Task (Go)
    ○ Pyinvoke (Python)
    ○ Rake (Ruby)

    View full-size slide

  22. Keep your tooling sharp
    ● Make
    ○ Can anybody do make?
    ○ Used as task runner can get complicated
    ● Make alternatives
    ○ Task (Go)
    ○ Pyinvoke (Python)
    ○ Rake (Ruby)
    ● Let contributors feel at home

    View full-size slide

  23. Organize tests
    ● Unit tests live within package folders

    View full-size slide

  24. Organize tests
    ● Unit tests live within package folders
    ● Other types of tests should be organized in a specific folder
    test
    ├── benchmarks
    ├── e2e
    ├── integration
    ├── kitchen
    ├── system
    └── util

    View full-size slide

  25. Keep your docs close
    ● Docs should be part of the project

    View full-size slide

  26. Keep your docs close
    ● Docs should be part of the project
    ○ Easier to update docs if all it takes is patching a file in the
    same repo
    ○ At review time, it’s easy to spot that a change to the code
    is missing the corresponding documentation update

    View full-size slide

  27. Keep your docs close
    ● Docs should be part of the project
    ○ Easier to update docs if all it takes is patching a file in the
    same repo
    ○ At review time, it’s easy to spot that a change to the code
    is missing the corresponding documentation update
    ● First time contributors to an Open Source project usually start
    with documentation
    ○ Make it obvious where to find docs
    ○ Make the update process easy

    View full-size slide

  28. Packaging
    ● Packages are part of the project
    ○ Keep packaging logic in the same repo
    ○ Make obvious where to find things
    ○ Isolate assets required by external tools in specific folders

    View full-size slide

  29. Packaging
    ● Packages are part of the project
    ○ Keep packaging logic in the same repo
    ○ Make obvious where to find things
    ○ Isolate assets required by external tools in specific folders
    ● Examples
    ○ /Dockerfiles/
    ○ /omnibus/

    View full-size slide

  30. Thanks for listening!
    Check out the Datadog Agent project on Github
    https://github.com/DataDog/datadog-agent/
    More of my elucubrations on
    @maxpippi
    http://dev.pippi.im/

    View full-size slide