Taming Dependencies

Managing Problematic Dependencies in Go: A Path to Sanity

Transcript

1. Taming Dependencies Managing Problematic Dependencies: A Path to Sanity Greg Poirier - CTO - @GetOpsee
2. Who am I? Why am I here? • CTO at @GetOpsee • Go in production for more than a year. • Building a whole damn product in Go. • I like to talk to people about my problems.
3. “In the trenches…”
4. Background: Opsee • Two primary components: API and Stream Processing • Stream processing components need to be independently deployable • API downtime cannot affect stream processing downtime. • Microservices to the rescue, and it had nothing to do with organizations.
5. In the beginning…
6. HTTP JSON RPC APIs • Easy to use and build • Easily parsed by humans when they want to debug • A lot of boilerplate code to write to get them going (this was prior to go-micro being particularly stable or useful).
7. Let ‘em bleed…
8. gRPC and Protobuf • “Performance”, but not really. • Code generation mechanisms that were language independent and that we didn’t have to write. • Declarative syntax for services and objects.
9. github.com/opsee/basic
11. Shared Dependencies • Services needed a common grammar to communicate with one another. • It was easier to standardize around a shared library than vendoring most of the same code in many individual components.
12. What is the problem?
13. gRPC • Tightly coupled ourselves to the transport layer. • Dependency management is hard. • Proliferation of multiple versions of the transport. • gRPC changes its exported interface all the time.
14. It can’t be that bad… Can it?
15. gmunch • gmunch: a service to abstract away the fact that we’re using AWS Kinesis. • You send it an event, it tells you if it was able to queue it for processing. • Significantly simpler interface to Kinesis, and client authentication doesn’t require IAM privileges.
17. Let’s write a new service… package myservicethatusesgmunch import (
 gmunch “github.com/opsee/gmunch/client”
 “google.golang.org/grpc/credentials”
 )
 func main() {
 // Ignore this obviously bad TLS init.
 client, err := gmunch.New(
 “gmunch”,
 credentials.NewTLS(nil))
 }
18. Oopsie… https://github.com/grpc/grpc-go/commit/ 59486d9c172d7b02e0166fce76fd720a88c24914
21. –Chris Knight in Real Genius I was thinking of the immortal words of Socrates, who said, “… I drank what?”
22. This happens more often than you think…
23. What do we do now?
25. I am bad, and I should feel bad.
26. Let’s fix it. • Must haves: • No more gRPC or shared dependency version conflicts causing problems. • A sensible build system.
27. Let’s fix it. • Nice to haves: • `go get` still works • `go build` still builds projects • Easily update dependencies across all projects.*
28. Well that sounds EASY…
29. Solution 1: DIP
30. Dependency Inversion Principle • Abstract away the gRPC bits from service clients. • Go vendor experiment takes care of the rest. • Example: • https://gist.github.com/grepory/ 0ee8067d99797d8430502135639be097
32. Advantages • Free of the tyranny of gRPC developers. • `go get` and `go build` keep working. • We feel like we’re being good software engineers. • Don’t have to do either of the next two solutions.
33. Disadvantages • Yet another shared dependency. • Have to write generators. • Still haven’t really solved our problem, just delayed having to really address it.
34. Solution 2: Monorepo
35. “Advantages of monolithic version control” http://danluu.com/monorepo/
36. Advantages • Atomic updates to dependencies. • `go build` works. • No longer have to use vendoring tools (which are all very annoying for different reasons). • Easier to audit accretion of external dependencies and versions.
37. Disadvantages • CI sucks and we have to build a builder. • Ironically, `go get` doesn’t work very well with monorepos. • We probably have to build tools to manage the dependencies themselves.
38. Solution 3: Monolithic build container
39. https://github.com/opsee/build-go
40. Advantages • We already build in containers and this is easy to do in CI and locally. • Per-project CI/CD doesn’t change. • No VCS difficulties (because let’s face it, submodules are a VCS difficulty nobody likes).
41. Disadvantages • Bye bye `go get` and `go build`. • Developers have to know what they can and cannot vendor in their project. • Managing dependencies requires all new tooling. • Still have to setup CI fanciness for updating the build container to run tests for every project.
42. Well? What do we do?
43. –Donald Rumsfeld “I'm not into this detail stuff. I'm more concepty.”
44. Not sure yet. • We have our known knowns and our known unknowns, but we’re not sure about our unknown unknowns. • We’re aware of the problem, and we’re actively working on its resolution. • It will likely involve some dependency inversion, because that’s probably just a good idea anyway.
45. Thanks! • Greg Poirier - @grepory on Twitter • Try Opsee! https://try.opsee.com