$30 off During Our Annual Pro Sale. View Details »

Taming Dependencies

Taming Dependencies

Managing Problematic Dependencies in Go: A Path to Sanity

Greg Poirier

June 14, 2016

More Decks by Greg Poirier

Other Decks in Programming


  1. Taming Dependencies Managing Problematic Dependencies: A Path to Sanity Greg

    Poirier - CTO - @GetOpsee
  2. Who am I? Why am I here? • CTO at

    @GetOpsee • Go in production for more than a year. • Building a whole damn product in Go. • I like to talk to people about my problems.
  3. “In the trenches…”

  4. Background: Opsee • Two primary components: API and Stream Processing

    • Stream processing components need to be independently deployable • API downtime cannot affect stream processing downtime. • Microservices to the rescue, and it had nothing to do with organizations.
  5. In the beginning…

  6. HTTP JSON RPC APIs • Easy to use and build

    • Easily parsed by humans when they want to debug • A lot of boilerplate code to write to get them going (this was prior to go-micro being particularly stable or useful).
  7. Let ‘em bleed…

  8. gRPC and Protobuf • “Performance”, but not really. • Code

    generation mechanisms that were language independent and that we didn’t have to write. • Declarative syntax for services and objects.
  9. github.com/opsee/basic

  10. None
  11. Shared Dependencies • Services needed a common grammar to communicate

    with one another. • It was easier to standardize around a shared library than vendoring most of the same code in many individual components.
  12. What is the problem?

  13. gRPC • Tightly coupled ourselves to the transport layer. •

    Dependency management is hard. • Proliferation of multiple versions of the transport. • gRPC changes its exported interface all the time.
  14. It can’t be that bad… Can it?

  15. gmunch • gmunch: a service to abstract away the fact

    that we’re using AWS Kinesis. • You send it an event, it tells you if it was able to queue it for processing. • Significantly simpler interface to Kinesis, and client authentication doesn’t require IAM privileges.
  16. None
  17. Let’s write a new service… package myservicethatusesgmunch import (

 func main() {
 // Ignore this obviously bad TLS init.
 client, err := gmunch.New(
  18. Oopsie… https://github.com/grpc/grpc-go/commit/ 59486d9c172d7b02e0166fce76fd720a88c24914

  19. None
  20. None
  21. –Chris Knight in Real Genius I was thinking of the

    immortal words of Socrates, who said, “… I drank what?”
  22. This happens more often than you think…

  23. What do we do now?

  24. None
  25. I am bad, and I should feel bad.

  26. Let’s fix it. • Must haves: • No more gRPC

    or shared dependency version conflicts causing problems. • A sensible build system.
  27. Let’s fix it. • Nice to haves: • `go get`

    still works • `go build` still builds projects • Easily update dependencies across all projects.*
  28. Well that sounds EASY…

  29. Solution 1: DIP

  30. Dependency Inversion Principle • Abstract away the gRPC bits from

    service clients. • Go vendor experiment takes care of the rest. • Example: • https://gist.github.com/grepory/ 0ee8067d99797d8430502135639be097
  31. None
  32. Advantages • Free of the tyranny of gRPC developers. •

    `go get` and `go build` keep working. • We feel like we’re being good software engineers. • Don’t have to do either of the next two solutions.
  33. Disadvantages • Yet another shared dependency. • Have to write

    generators. • Still haven’t really solved our problem, just delayed having to really address it.
  34. Solution 2: Monorepo

  35. “Advantages of monolithic version control” http://danluu.com/monorepo/

  36. Advantages • Atomic updates to dependencies. • `go build` works.

    • No longer have to use vendoring tools (which are all very annoying for different reasons). • Easier to audit accretion of external dependencies and versions.
  37. Disadvantages • CI sucks and we have to build a

    builder. • Ironically, `go get` doesn’t work very well with monorepos. • We probably have to build tools to manage the dependencies themselves.
  38. Solution 3: Monolithic build container

  39. https://github.com/opsee/build-go

  40. Advantages • We already build in containers and this is

    easy to do in CI and locally. • Per-project CI/CD doesn’t change. • No VCS difficulties (because let’s face it, submodules are a VCS difficulty nobody likes).
  41. Disadvantages • Bye bye `go get` and `go build`. •

    Developers have to know what they can and cannot vendor in their project. • Managing dependencies requires all new tooling. • Still have to setup CI fanciness for updating the build container to run tests for every project.
  42. Well? What do we do?

  43. –Donald Rumsfeld “I'm not into this detail stuff. I'm more

  44. Not sure yet. • We have our known knowns and

    our known unknowns, but we’re not sure about our unknown unknowns. • We’re aware of the problem, and we’re actively working on its resolution. • It will likely involve some dependency inversion, because that’s probably just a good idea anyway.
  45. Thanks! • Greg Poirier - @grepory on Twitter • Try

    Opsee! https://try.opsee.com