Poirier Sensu, Inc. Hi friends! My name is Greg Poirier, and I work at Sensu on the open source monitoring platform. I’ve spent the last 15 years building systems for ISPs, *air quotes* “big” data companies, the US government, and SaaS providers. Today, I want to talk about one of my favorite infrastructure design patterns: the sidecar.
2016 Sidecars extend and enhance the main container. In this case, the “main container” is the actual application. But I want to frame our discussion of sidecars a little so that they maybe make a little more sense in the context of a holistic architecture. First, though…
super expensive choose your own adventure book that only ends when you run out of money. So, my suggestion is to make choices that mean you don’t run out of money trying to engineer your way out of a hole you dug yourself.
sprawling organization is now faced with addressing the mountains of technical debt that made you so successful. You: 1) Find more tires (pg. 9,345,160) 2) Reorganize people and code in a way that makes microservices make sense. (pg. 263,400) But maybe you’re reading along in your choose your own adventure book, and you ﬁnd yourself at this page.
you see immediately, “Ah yes! Of course! I see how this easily decomposes into services. Why don’t we reorganize our teams around this very, very obvious decomposition so that all of our development eﬀorts are aligned with a great new vision.” And you get to work right away on some new architecture diagrams.
services: - HTTP for transport - Rate limiting to provide backpressure to noisy internal users - A sensible authentication and authorization layer that allows for security and accountability - Uniform request logging across all services in a structured logging format that helps with request tracing through the new system - A core service discovery mechanism that makes it easy to ﬁnd the nearest copy of another service - A conﬁguration system to deliver durable conﬁguration data to applications - And a standardized metrics format and storage mechanism Everyone’s really happy about this new world, but then a few teams get together, and start to think, “Hey wait! We could build some of these things once, and then every can beneﬁt from that work!” And now it’s time to make a decision.
like he’s trying to make a decision. In this one, he’s clearly trying to decide between some shared libraries or reusable external processes that add functionality to each of your organization’s deployed applications! Wait. What was that last sentence, that sounds familiar!
organization devotes adequate resources to the building and maintenance of some shared sidecars that help developers rapidly iterate on business logic. At this point, I want to step back, though, because did you notice how I haven’t even said containers yet?
a locally deployed nginx reverse proxy. That is totally a sidecar. Nginx can provide your service with buﬀering, caching, rate limiting, and load balancing. That’s additional functionality that you didn’t have to implement yourself.
is going to be about shared libraries. It’s true, shared libraries provide code reuse as well, but there are a couple of key factors here. First, sidecars don’t care what language your service is written in. Second, sidecars are runtime dependencies, not build dependencies. Runtime dependencies can be built, tested, and deployed in isolation. So while you may have to deploy your sidecar to the world for a critical update, you won’t have to deploy the whole world. Solid integration and regression testing for your sidecars allow for a high conﬁdence that they can be deployed safely in isolation of the services that depend on them. And that’s the key to being successful.
sure that your sidecars adhere to a well-speciﬁed interface, and you need to test that interface with every build. Even if it’s Nginx, the way that applications interact with that process needs to be tested.
Twitter moment is: “Why does this have to live on the same machine as my service? Why can’t I have a pool of other hosts make this functionality available for every service?” Well, you can go that route, and many companies do so successfully, but I prefer the sidecar approach for its next set of beneﬁts. In order to talk about those beneﬁts, let’s consider these two choices.
our imaginary architecture, conﬁguration could have been a client library provided for applications, but we’ve decided to pull it out of the monolithic stack into a sidecar. If you’ve never used or built a library like this, all it eﬀectively does is call out to an external key-value store like Zookeeper, Consul, or Etcd to get conﬁguration data at runtime. It might even support hot reloading of conﬁguration data by watching those values and restart or reconﬁguring the application when they change.
left we have what it would look like if there was a centralized conﬁguration service that applications spoke to, we have the customer and order services talking directly to an external conﬁguration service. Again, this could be the key-value store directly or we could have fronted it by some kind of service providing a simpler interface. On the right, we’ve introduced a sidecar that acts as a smart proxy to our persistent data store backing the conﬁguration service. Their implementations aren’t signiﬁcantly diﬀerent—the sidecar would behave pretty similarly to an external conﬁguration service, BUT! You could implement identical interfaces, if you so wanted. There’s a very important, key diﬀerence here: the sidecar is local and shares a the physical location of the customer and order services, respectively. So instead of requiring network traversal, the two processes can communicate via any means available to the host locally. In our example, we’re going to have them communicate via the ﬁlesystem.
methods that it accepts, put and get, to a path. Your HTTP requests to the ﬁctional HTTP conﬁguration service would look like those with some JSON in the body. Our reimagined conﬁguration service that is built using a sidecar might look a little diﬀerent.
is localized to this one artifact that the maintainers of the conﬁguration service ship to your laptop and to production. Regardless of the interfaces exposed, this aﬀords us considerable ﬂexibility when implementing the conﬁguration service, because now we can do fancier things! For example, we can cache a local copy of the conﬁguration. Or we can make the sidecar poll our key-value store and update the local copy on disk whenever the value changes.
diﬀerent than a client library talking to our conﬁguration service? Well, what if someone working on a new service puts conﬁguration retrieval too early in startup. Let’s pretend for a second that they’re not using whatever framework has been provided for the lingua franca of your organization. They’ve had to write a service in Go, and everything else there is in Java. They whip together a new service that talks to a centralized conﬁguration service, but they put that action really early in startup—before any other dependencies are satisﬁed. Now let’s pretend that right after they talk to the conﬁguration service, they panic trying to test for their ﬁrst runtime dependency. The service restarts, causing it to poll the conﬁguration service again. And again. And again. Once a millisecond or so. And now your conﬁguration service is DOS’d. Even with the rate limiting you’ve put in place, it’s taking up sockets, because the process is leaving sockets around in FIN_WAIT on your conﬁg service hosts. Well.
sidecar, the service would just be opening and reading a ﬁle on disk. The sidecar is happily doing its thing, polling the key-value store occasionally or waiting on events saying the value has changed. And those are our ﬁnal, two, key beneﬁts of sidecars:
between processes and localize them to a single host. Your application may abuse the local copy of the sidecar only, and requests to the sidecar do not have to traverse the network. And, a local process has many more communication primitives to work with than a remote process. Every process having its own instance of our ﬁctional conﬁguration service means that no single service or process can dominate the pool of shared resources.
talk, I promised some practical examples. So I wandered the earth looking for some super sweet sidecars. Some of these were pointed out to me by friends, and I even wrote one myself in like a quick second as an example of a sidecar with a well-deﬁned, tested interface for a conﬁguration provider.
via Netﬂix’s rich suite of JVM client libraries over an HTTP interface for an application. This is exactly the kind of approach to sidecar development that I’m advocating for today. They have a well-deﬁned, well-formed software stack that they want to leverage across multiple programming languages. So they just pulled the stack right out and built a sidecar service for applications that can’t use client libraries.
like Google. The next sidecar I want to talk about is this oauth2 proxy. This is really cool, because it’s a such great example of an external process that provides a very speciﬁc functionality to your application. In this case, a complete OAuth2 workﬂow, and I don’t know if you’ve ever added oauth to an application, but this is sort of an annoying thing to do over and over again.
library, but with this sidecar, the experience of implementing the OAuth2 workﬂow for applications is standardized without having to ﬁgure out how to do it for whatever language I’m developing in. I can easily ship a completely uniform OAuth experience across every externally-facing application.
totally sweet sidecar that I love. Have you ever wanted to just throw an nginx proxy in front of your project and have it manage its own SSL certiﬁcates with LetsEncrypt. Yeah, well, there you go. Now you can, and it’s totally easy. This does use docker-gen though, which requires access to the Docker socket, so there’s room for improvement there, but this is totally workable immediately for experimentation.
seen the stuﬀ form buoyant.io, I highly encourage you to look at linkerd, linkerd-tcp which was recently announced, and namerd. Linkerd service-to-service routing sidecars that were built by the some of the team members that helped build Finagle at Twitter. This is a powerful construct, because it allows you to centralize all of your service discovery, load balancing, and failure handling in a single location.
take a look at metadataproxy. You can use metadataproxy for a few diﬀerent things. I’ve used it for local development of applications that I want to talk to the EC2 metadata service. In a container runtime environment where you’re deploying with Docker, you could actually route requests from containers to the metadataproxy with ﬁrewall rules and have metadataproxy deliver STS credentials for speciﬁc services. If you’re using ECS, this isn’t necessary, because ECS provides per-container IAM roles already, but keep this in mind if you’re building a container platform.