▸ HR, Payroll, Benefits software. ▸ Moves more than 1 billion in payroll monthly ▸ Around 100 Engineering Employees ▸ Moving to microservices for the past couple of years
Spinnaker (Kubernetes v2 provider) for Deployments ▸ AWS for all Infrastructure ▸ 4 k8s Environments (Integration, staging, production, ops) ▸ Istio 1.0.2 ▸ Using gRPC for all service to service calls ▸ Go, C#, Ruby, and Python services, React frontends
applications: A Rails monolith and a .NET Monolith ‣ Now we have 3 dozen (and climbing) services in our architecture ‣ We also found ourselves with the improper tooling to deploy in this new world
when you don't have the right tools to deploy ‣ SRE was responsible for all deploys. (Manually running kubectl apply -f locally) ‣ No standards on how applications deploy, when, how, etc.
file to every project and modify the Jenkinsfile to execute it at the end of a build ▸ Open 30+ pull requests to projects to install this monstrosity ▸ Boom! Continuous Deployment!
1/3rd of the solution to any business problem ▸ Installing and configuring tools like Spinnaker is the easy part ▸ Educating, documenting, productionalizing, and on- boarding is a whole different ball game
must have a source of truth that is version controlled in Git ▸ Pipelines must automatically update in Spinnaker when changes are pushed to Github ▸ Releases must be recorded for all environments for every project
wanted to use Spinnaker Pipeline's for all deployments ▸ However, Spinnaker didn't have a tool for "codifying" pipelines that supported everything we wanted ▸ Bummer, because we don't want engineers managing pipelines via the Spinnaker UI ▸ This allows drift from a set standard that we can build into tools ▸ Our decision: Build that missing tool.
what a Spinnaker pipeline is via Go structs, we now have a guaranteed JSON structure. ▸ We also can easily manipulate pipelines (more on that later). ▸ We can then render these struct values as JSON and send that payload to Gate (the Spinnaker API) for any operation we need to perform.
SPEC ▸ k8s-pipeliner includes a set of pipeline config objects ▸ These values are what compose our pipeline.yml file that gets transposed into a Spinnaker Pipeline JSON payload
thing we didn't want our pipeline.yml file to define is what gets deployed ▸ Instead, our spec only defines how something should be deployed, not the specifics of what. ▸ Our file references Kubernetes manifest files to define what get's deployed.
Original Kubernetes provider had it's own flavor of Kubernetes objects as JSON ▸ This is because the provider hit the Kubernetes API with a Java client as opposed to just using kubectl (which is what v2 does) ▸ This caused massive pain creating these definitions ▸ We had to create every Spinnaker Kubernetes V1 object manually
managed via the UI (with the exception of pasting the JSON) ▸ All of the pipeline.yml files are checked into source control ▸ Reverts to the codebase allows us to revert pipelines easily ▸ Track record of who changed what in pipelines ▸ The wins are numerous.
and rolled that way without version control. ▸ We could have let teams own their own process. ▸ But the long term spend of that is enormous ▸ Configuring via the UI would be tedious and error prone ▸ Linear scale of time spent as we add more applications
▸ Disclaimer: I love Spinnaker. We use it for every project we deploy to Kubernetes (which is basically everything) ▸ But the UX for managing pipelines is clunky ▸ UI/UX is hard to get right for something that can do so much (See: AWS) ▸ Spinnaker suffers from this (great) problem because it can do so much.
pasting JSON into a text field in the pipeline configuration page, this should be automatic. ▸ Whenever a merge is performed into master for a project, we'll send a Github webhook to an internal service ▸ That project will then clone the project, read the pipeline config file(s), and then update the application's pipeline
meets the stream. WHAT IS ESTUARY ▸ Estuary is a simple HTTP server that receives GitHub webhooks, builds a pipeline config, and then updates the Spinnaker Pipeline via the Gate API. ▸ Written in Go ▸ Imports k8s-pipeliner to generate the JSON ▸ Can inject values into pipelines automatically as well
Gate to authenticate via Github and A X509 Certificate ▸ Using the X509 Certificate allows us to send POST requests to update the pipeline from Estuary ▸ But what does this request look like exactly?
a Swagger file for it's API ▸ However... we found it easier to just look at what the UI sends in its XHR requests ▸ Once we figured that out, we can easily update the Pipeline
entry of what was deployed, when, and other metadata ▸ Stored forever, most recent first ▸ Can be used for identifying what release broke what ▸ Metrics! ▸ Release managers love it
a module called Echo that can send webhook requests on task events ▸ We've configured Echo to send webhooks to our "Release Management Software" ▸ Release Management Service parses the webhook and creates log entries when pipeline stages complete
We could not find a single page of documentation about what this payload looks like in full ▸ Little did we know a typical payload for one of our pipelines was 870kb of JSON (that's enormous) ▸ So, we just sent the payload to http://webhook.site
puppets ▸ On their own, they're lifeless, they provide very little or no business value on their own ▸ We as engineers must be the puppeteers and give these projects the personality we want them to have ▸ Building tools around them is just one of those ways to give a project like Spinnaker the life we want it to have
bobbytables
kashioka
kanaru0928
high_g_engineer
pkshadeck
komatsunaqa
hayama17
johnroyer
kentaroutakeda
mu7889yoon
ydah
yosh1
tomohisa
chrisshort
brad_frost
reverentgeek
varn
charlesmeaden
bkeepers
jeffersonlam
inesmontani
aemeredith
jacobian
jonoalderson
