Project layout patterns in Go

You completed the tour, learned the language, got your pet project on Github. Time to start a new project, a big one: how do you organize code, tests, docs? During the talk I’ll try to save you some trial and error by sharing the lessons learned during a year spent coding a non trivial Go project.

Transcript

1. Project layout patterns in Go Massimiliano Pippi
2. • Engineering team lead • 2+ year at Datadog, working on Agent and Integrations • Python and Go • OSS fan and contributor Hi, I’m Massi!
3. What’s this about • How we laid out one of our Go projects ◦ Code is Open Source ◦ ~70k lines of Go ◦ ~10 fulltime contributors ◦ 11 teams involved ◦ Packaged, not deployed • Your mileage may vary
4. Trial and error A. Problem B. Solution n.1 C. Lessons learned D. Solution n.2
5. Trial and error A. Problem B. Solution n.1 C. Lessons learned D. Solution n.2 • Deadline too tight • Changing project layout impacts code • Couldn’t afford a second take
6. When in Rome...
7. When in Rome... • Looked at the Go state of the art ◦ Kubernetes ◦ Etcd ◦ CockroachDB
8. When in Rome... • Looked at the Go state of the art ◦ Kubernetes ◦ Etcd ◦ CockroachDB • Shoot at one of similar size ◦ Consider overhead... ◦ ...but you might need it ◦ Consider the amount of tooling that a layout requires
9. Code layout • Project must be easy to browse ◦ Keep the root folder clean and sleek
10. Code layout • Project must be easy to browse ◦ Keep the root folder clean and sleek • Don’t make contributors guess where things are ◦ Good ▪ /docs /tests /releasenotes ◦ Bad ▪ /extra/build/docker/Dockerfile /common/docs ◦ Evil ▪ /src/bin
11. Code layout • Use packages! ◦ They help you designing the API ◦ They ease code reuse ◦ They’re a powerful abstraction tool ◦ Go stdlib is the best place to lurk
12. Code layout • Use packages! ◦ They help you designing the API ◦ They ease code reuse ◦ They’re a powerful abstraction tool ◦ Go stdlib is the best place to lurk • If packages are well designed, the source tree will reflect it
13. Code layout • Partition the codebase
14. Code layout • Partition the codebase ◦ The Framework: /pkg/ ◦ The Programs: /cmd/
15. Code layout • Partition the codebase ◦ The Framework: /pkg/ ▪ /pkg/utils/, /pkg/config/, ... ▪ No dependencies on /cmd/ ▪ Can be imported by external programs ◦ The Programs: /cmd/
16. Code layout • Partition the codebase ◦ The Framework: /pkg/ ▪ /pkg/utils/, /pkg/config/, ... ▪ No dependencies on /cmd/ ▪ Can be imported by external programs ◦ The Programs: /cmd/ ▪ /cmd/superduper, /cmd/miniduper, … ▪ Domain specific logic stays close to the main() func
17. Keep your tooling sharp
18. Keep your tooling sharp • Goals ◦ Make obvious and robust how to build and run tests
19. Keep your tooling sharp • Goals ◦ Make obvious and robust how to build and run tests • Constraints ◦ Make CI robots use what humans use ◦ Tools orchestrate, don’t implement: bypassing tooling should always be possible ◦ Keep number of dependencies low
20. Keep your tooling sharp • Make ◦ Can anybody do make? ◦ Used as task runner can get complicated
21. Keep your tooling sharp • Make ◦ Can anybody do make? ◦ Used as task runner can get complicated • Make alternatives ◦ Task (Go) ◦ Pyinvoke (Python) ◦ Rake (Ruby)
22. Keep your tooling sharp • Make ◦ Can anybody do make? ◦ Used as task runner can get complicated • Make alternatives ◦ Task (Go) ◦ Pyinvoke (Python) ◦ Rake (Ruby) • Let contributors feel at home
23. Organize tests • Unit tests live within package folders
24. Organize tests • Unit tests live within package folders • Other types of tests should be organized in a specific folder test ├── benchmarks ├── e2e ├── integration ├── kitchen ├── system └── util
25. Keep your docs close • Docs should be part of the project
26. Keep your docs close • Docs should be part of the project ◦ Easier to update docs if all it takes is patching a file in the same repo ◦ At review time, it’s easy to spot that a change to the code is missing the corresponding documentation update
27. Keep your docs close • Docs should be part of the project ◦ Easier to update docs if all it takes is patching a file in the same repo ◦ At review time, it’s easy to spot that a change to the code is missing the corresponding documentation update • First time contributors to an Open Source project usually start with documentation ◦ Make it obvious where to find docs ◦ Make the update process easy
28. Packaging • Packages are part of the project ◦ Keep packaging logic in the same repo ◦ Make obvious where to find things ◦ Isolate assets required by external tools in specific folders
29. Packaging • Packages are part of the project ◦ Keep packaging logic in the same repo ◦ Make obvious where to find things ◦ Isolate assets required by external tools in specific folders • Examples ◦ /Dockerfiles/ ◦ /omnibus/
30. Thanks for listening! Check out the Datadog Agent project on Github https://github.com/DataDog/datadog-agent/ More of my elucubrations on @maxpippi http://dev.pippi.im/