Project layout patterns in Go

You completed the tour, learned the language, got your pet project on Github. Time to start a new project, a big one: how do you organize code, tests, docs? During the talk I’ll try to save you some trial and error by sharing the lessons learned during a year spent coding a non trivial Go project.

Transcript

1. Project layout patterns in Go
   Massimiliano Pippi
   

   View Slide

2. ● Engineering team lead
   ● 2+ year at Datadog, working on Agent and Integrations
   ● Python and Go
   ● OSS fan and contributor
   Hi, I’m Massi!
   

   View Slide

3. What’s this about
   ● How we laid out one of our Go projects
   ○ Code is Open Source
   ○ ~70k lines of Go
   ○ ~10 fulltime contributors
   ○ 11 teams involved
   ○ Packaged, not deployed
   ● Your mileage may vary
   

   View Slide

4. Trial and error
   A. Problem
   B. Solution n.1
   C. Lessons learned
   D. Solution n.2
   

   View Slide

5. Trial and error
   A. Problem
   B. Solution n.1
   C. Lessons learned
   D. Solution n.2
   ● Deadline too tight
   ● Changing project layout impacts code
   ● Couldn’t afford a second take
   

   View Slide

6. When in Rome...
   

   View Slide

7. When in Rome...
   ● Looked at the Go state of the art
   ○ Kubernetes
   ○ Etcd
   ○ CockroachDB
   

   View Slide

8. When in Rome...
   ● Looked at the Go state of the art
   ○ Kubernetes
   ○ Etcd
   ○ CockroachDB
   ● Shoot at one of similar size
   ○ Consider overhead...
   ○ ...but you might need it
   ○ Consider the amount of tooling that a layout requires
   

   View Slide

9. Code layout
   ● Project must be easy to browse
   ○ Keep the root folder clean and sleek
   

   View Slide

10. Code layout
    ● Project must be easy to browse
    ○ Keep the root folder clean and sleek
    ● Don’t make contributors guess where things are
    ○ Good
    ■ /docs /tests /releasenotes
    ○ Bad
    ■ /extra/build/docker/Dockerfile /common/docs
    ○ Evil
    ■ /src/bin
    

    View Slide

11. Code layout
    ● Use packages!
    ○ They help you designing the API
    ○ They ease code reuse
    ○ They’re a powerful abstraction tool
    ○ Go stdlib is the best place to lurk
    

    View Slide

12. Code layout
    ● Use packages!
    ○ They help you designing the API
    ○ They ease code reuse
    ○ They’re a powerful abstraction tool
    ○ Go stdlib is the best place to lurk
    ● If packages are well designed, the source tree will reflect it
    

    View Slide

13. Code layout
    ● Partition the codebase
    

    View Slide

14. Code layout
    ● Partition the codebase
    ○ The Framework: /pkg/
    ○ The Programs: /cmd/
    

    View Slide

15. Code layout
    ● Partition the codebase
    ○ The Framework: /pkg/
    ■ /pkg/utils/, /pkg/config/, ...
    ■ No dependencies on /cmd/
    ■ Can be imported by external programs
    ○ The Programs: /cmd/
    

    View Slide

16. Code layout
    ● Partition the codebase
    ○ The Framework: /pkg/
    ■ /pkg/utils/, /pkg/config/, ...
    ■ No dependencies on /cmd/
    ■ Can be imported by external programs
    ○ The Programs: /cmd/
    ■ /cmd/superduper, /cmd/miniduper, …
    ■ Domain specific logic stays close to the main() func
    

    View Slide

17. Keep your tooling sharp
    

    View Slide

18. Keep your tooling sharp
    ● Goals
    ○ Make obvious and robust how to build and run tests
    

    View Slide

19. Keep your tooling sharp
    ● Goals
    ○ Make obvious and robust how to build and run tests
    ● Constraints
    ○ Make CI robots use what humans use
    ○ Tools orchestrate, don’t implement: bypassing tooling
    should always be possible
    ○ Keep number of dependencies low
    

    View Slide

20. Keep your tooling sharp
    ● Make
    ○ Can anybody do make?
    ○ Used as task runner can get complicated
    

    View Slide

21. Keep your tooling sharp
    ● Make
    ○ Can anybody do make?
    ○ Used as task runner can get complicated
    ● Make alternatives
    ○ Task (Go)
    ○ Pyinvoke (Python)
    ○ Rake (Ruby)
    

    View Slide

22. Keep your tooling sharp
    ● Make
    ○ Can anybody do make?
    ○ Used as task runner can get complicated
    ● Make alternatives
    ○ Task (Go)
    ○ Pyinvoke (Python)
    ○ Rake (Ruby)
    ● Let contributors feel at home
    

    View Slide

23. Organize tests
    ● Unit tests live within package folders
    

    View Slide

24. Organize tests
    ● Unit tests live within package folders
    ● Other types of tests should be organized in a specific folder
    test
    ├── benchmarks
    ├── e2e
    ├── integration
    ├── kitchen
    ├── system
    └── util
    

    View Slide

25. Keep your docs close
    ● Docs should be part of the project
    

    View Slide

26. Keep your docs close
    ● Docs should be part of the project
    ○ Easier to update docs if all it takes is patching a file in the
    same repo
    ○ At review time, it’s easy to spot that a change to the code
    is missing the corresponding documentation update
    

    View Slide

27. Keep your docs close
    ● Docs should be part of the project
    ○ Easier to update docs if all it takes is patching a file in the
    same repo
    ○ At review time, it’s easy to spot that a change to the code
    is missing the corresponding documentation update
    ● First time contributors to an Open Source project usually start
    with documentation
    ○ Make it obvious where to find docs
    ○ Make the update process easy
    

    View Slide

28. Packaging
    ● Packages are part of the project
    ○ Keep packaging logic in the same repo
    ○ Make obvious where to find things
    ○ Isolate assets required by external tools in specific folders
    

    View Slide

29. Packaging
    ● Packages are part of the project
    ○ Keep packaging logic in the same repo
    ○ Make obvious where to find things
    ○ Isolate assets required by external tools in specific folders
    ● Examples
    ○ /Dockerfiles/
    ○ /omnibus/
    

    View Slide

30. Thanks for listening!
    Check out the Datadog Agent project on Github
    https://github.com/DataDog/datadog-agent/
    More of my elucubrations on
    @maxpippi
    http://dev.pippi.im/
    

    View Slide