Project layout patterns in Go

Project layout patterns in Go

You completed the tour, learned the language, got your pet project on Github. Time to start a new project, a big one: how do you organize code, tests, docs? During the talk I’ll try to save you some trial and error by sharing the lessons learned during a year spent coding a non trivial Go project.

Aee59cb61d6561244163f490aec50939?s=128

Massimiliano Pippi

October 22, 2018
Tweet

Transcript

  1. Project layout patterns in Go Massimiliano Pippi

  2. • Engineering team lead • 2+ year at Datadog, working

    on Agent and Integrations • Python and Go • OSS fan and contributor Hi, I’m Massi!
  3. What’s this about • How we laid out one of

    our Go projects ◦ Code is Open Source ◦ ~70k lines of Go ◦ ~10 fulltime contributors ◦ 11 teams involved ◦ Packaged, not deployed • Your mileage may vary
  4. Trial and error A. Problem B. Solution n.1 C. Lessons

    learned D. Solution n.2
  5. Trial and error A. Problem B. Solution n.1 C. Lessons

    learned D. Solution n.2 • Deadline too tight • Changing project layout impacts code • Couldn’t afford a second take
  6. When in Rome...

  7. When in Rome... • Looked at the Go state of

    the art ◦ Kubernetes ◦ Etcd ◦ CockroachDB
  8. When in Rome... • Looked at the Go state of

    the art ◦ Kubernetes ◦ Etcd ◦ CockroachDB • Shoot at one of similar size ◦ Consider overhead... ◦ ...but you might need it ◦ Consider the amount of tooling that a layout requires
  9. Code layout • Project must be easy to browse ◦

    Keep the root folder clean and sleek
  10. Code layout • Project must be easy to browse ◦

    Keep the root folder clean and sleek • Don’t make contributors guess where things are ◦ Good ▪ /docs /tests /releasenotes ◦ Bad ▪ /extra/build/docker/Dockerfile /common/docs ◦ Evil ▪ /src/bin
  11. Code layout • Use packages! ◦ They help you designing

    the API ◦ They ease code reuse ◦ They’re a powerful abstraction tool ◦ Go stdlib is the best place to lurk
  12. Code layout • Use packages! ◦ They help you designing

    the API ◦ They ease code reuse ◦ They’re a powerful abstraction tool ◦ Go stdlib is the best place to lurk • If packages are well designed, the source tree will reflect it
  13. Code layout • Partition the codebase

  14. Code layout • Partition the codebase ◦ The Framework: /pkg/

    ◦ The Programs: /cmd/
  15. Code layout • Partition the codebase ◦ The Framework: /pkg/

    ▪ /pkg/utils/, /pkg/config/, ... ▪ No dependencies on /cmd/ ▪ Can be imported by external programs ◦ The Programs: /cmd/
  16. Code layout • Partition the codebase ◦ The Framework: /pkg/

    ▪ /pkg/utils/, /pkg/config/, ... ▪ No dependencies on /cmd/ ▪ Can be imported by external programs ◦ The Programs: /cmd/ ▪ /cmd/superduper, /cmd/miniduper, … ▪ Domain specific logic stays close to the main() func
  17. Keep your tooling sharp

  18. Keep your tooling sharp • Goals ◦ Make obvious and

    robust how to build and run tests
  19. Keep your tooling sharp • Goals ◦ Make obvious and

    robust how to build and run tests • Constraints ◦ Make CI robots use what humans use ◦ Tools orchestrate, don’t implement: bypassing tooling should always be possible ◦ Keep number of dependencies low
  20. Keep your tooling sharp • Make ◦ Can anybody do

    make? ◦ Used as task runner can get complicated
  21. Keep your tooling sharp • Make ◦ Can anybody do

    make? ◦ Used as task runner can get complicated • Make alternatives ◦ Task (Go) ◦ Pyinvoke (Python) ◦ Rake (Ruby)
  22. Keep your tooling sharp • Make ◦ Can anybody do

    make? ◦ Used as task runner can get complicated • Make alternatives ◦ Task (Go) ◦ Pyinvoke (Python) ◦ Rake (Ruby) • Let contributors feel at home
  23. Organize tests • Unit tests live within package folders

  24. Organize tests • Unit tests live within package folders •

    Other types of tests should be organized in a specific folder test ├── benchmarks ├── e2e ├── integration ├── kitchen ├── system └── util
  25. Keep your docs close • Docs should be part of

    the project
  26. Keep your docs close • Docs should be part of

    the project ◦ Easier to update docs if all it takes is patching a file in the same repo ◦ At review time, it’s easy to spot that a change to the code is missing the corresponding documentation update
  27. Keep your docs close • Docs should be part of

    the project ◦ Easier to update docs if all it takes is patching a file in the same repo ◦ At review time, it’s easy to spot that a change to the code is missing the corresponding documentation update • First time contributors to an Open Source project usually start with documentation ◦ Make it obvious where to find docs ◦ Make the update process easy
  28. Packaging • Packages are part of the project ◦ Keep

    packaging logic in the same repo ◦ Make obvious where to find things ◦ Isolate assets required by external tools in specific folders
  29. Packaging • Packages are part of the project ◦ Keep

    packaging logic in the same repo ◦ Make obvious where to find things ◦ Isolate assets required by external tools in specific folders • Examples ◦ /Dockerfiles/ ◦ /omnibus/
  30. Thanks for listening! Check out the Datadog Agent project on

    Github https://github.com/DataDog/datadog-agent/ More of my elucubrations on @maxpippi http://dev.pippi.im/