Spinnaker Wants Tools, So Build Them!

Transcript

1. BUILDING TOOLS FOR SPINNAKER A STORY ABOUT
2. ROBERT ROSS ME MY NAME IS I AM A SOFTWARE ENGINEER @ NAMELY I'M ALSO A MARCHING BAND NERD
3. ROBERT ROSS ME MY NAME IS I AM A SOFTWARE ENGINEER @ NAMELY I'M ALSO A MARCHING BAND NERD
4. BUILDING TOOLS FOR SPINNAKER THIS IS A STORY ABOUT NAMELY ▸ HR, Payroll, Benefits software. ▸ Moves more than 1 billion in payroll monthly ▸ Around 100 Engineering Employees ▸ Moving to microservices for the past couple of years
5. THE INFRASTRUCTURE ▸ Kubernetes 1.9 ▸ Jenkins for tests ▸ Spinnaker (Kubernetes v2 provider) for Deployments ▸ AWS for all Infrastructure ▸ 4 k8s Environments (Integration, staging, production, ops) ▸ Istio 1.0.2 ▸ Using gRPC for all service to service calls ▸ Go, C#, Ruby, and Python services, React frontends
6. FROM TWO SERVICES TO TOO MANY ‣ We had 2 applications: A Rails monolith and a .NET Monolith ‣ Now we have 3 dozen (and climbing) services in our architecture ‣ We also found ourselves with the improper tooling to deploy in this new world
7. FROM TWO SERVICES TO TOO MANY ‣ This causes problems when you don't have the right tools to deploy ‣ SRE was responsible for all deploys. (Manually running kubectl apply -f locally) ‣ No standards on how applications deploy, when, how, etc.
8. SO HOW DO YOU IMPROVE THAT?
9. OUR INITIAL SOLUTION?
10. DEPLOY FROM JENKINS!
11. BUILDING TOOLS FOR SPINNAKER GREAT IDEA! ▸ Copy a kube-deploy.sh file to every project and modify the Jenkinsfile to execute it at the end of a build ▸ Open 30+ pull requests to projects to install this monstrosity ▸ Boom! Continuous Deployment!
13. IT WAS AT THIS MOMENT WE KNEW... WE HAD F'ED UP
14. INTRODUCTION OF SPINNAKER
15. "YEAH I CAN INSTALL THAT IN 3 WEEKS" SRE SHRAY SAID
17. BUILDING TOOLS FOR SPINNAKER WHY? ▸ Open Source tools are 1/3rd of the solution to any business problem ▸ Installing and configuring tools like Spinnaker is the easy part ▸ Educating, documenting, productionalizing, and on- boarding is a whole different ball game
18. WRITE A DESIGN DOC Mike Hamrah, Chief Architect @ Namely BUILDING TOOLS FOR SPINNAKER
19. BUILDING TOOLS FOR SPINNAKER RULES AND REQUIREMENTS ▸ Pipeline configurations must have a source of truth that is version controlled in Git ▸ Pipelines must automatically update in Spinnaker when changes are pushed to Github ▸ Releases must be recorded for all environments for every project
20. PIPELINE CONFIGURATION PART 1
21. BUILDING TOOLS FOR SPINNAKER WHAT IS A PIPELINE? ▸ We wanted to use Spinnaker Pipeline's for all deployments ▸ However, Spinnaker didn't have a tool for "codifying" pipelines that supported everything we wanted ▸ Bummer, because we don't want engineers managing pipelines via the Spinnaker UI ▸ This allows drift from a set standard that we can build into tools ▸ Our decision: Build that missing tool.
22. BUILDING TOOLS FOR SPINNAKER WHERE TO START? ▸ Spinnaker has a nifty "Edit as JSON" feature for Pipeline's in the UI ▸ We configured pipelines for projects by hand and then took the resulting JSON
23. BUILDING TOOLS FOR SPINNAKER WHERE TO START?
24. BUILDING TOOLS FOR SPINNAKER THE ENTIRE PIPELINE IS JSON
25. BUILDING TOOLS FOR SPINNAKER CONVERTING THE JSON ▸ We then used that JSON to create concrete Go structs for our tool ▸ This tool became k8s-pipeliner
26. GO(ING) NUTS FOR STRUCTS K8S-PIPELINER GO STRUCTS ▸ By defining what a Spinnaker pipeline is via Go structs, we now have a guaranteed JSON structure. ▸ We also can easily manipulate pipelines (more on that later). ▸ We can then render these struct values as JSON and send that payload to Gate (the Spinnaker API) for any operation we need to perform.
27. CREATING OUR PIPELINE DEFINITION NOW WE JUST NEED A CONFIG SPEC ▸ k8s-pipeliner includes a set of pipeline config objects ▸ These values are what compose our pipeline.yml file that gets transposed into a Spinnaker Pipeline JSON payload
28. IT LOOKS LIKE THIS
29. WHICH ALLOWS US TO DO THIS
30. GETTING INTO FLOW DEFINE PIPELINE FLOW, NOT SPECIFICS ▸ One thing we didn't want our pipeline.yml file to define is what gets deployed ▸ Instead, our spec only defines how something should be deployed, not the specifics of what. ▸ Our file references Kubernetes manifest files to define what get's deployed.
31. REFERENCING A MANIFEST FILE
32. GO(ING) NUTS FOR STRUCTS ONE PROBLEM... KUBERNETES V1 PROVIDER ▸ Original Kubernetes provider had it's own flavor of Kubernetes objects as JSON ▸ This is because the provider hit the Kubernetes API with a Java client as opposed to just using kubectl (which is what v2 does) ▸ This caused massive pain creating these definitions ▸ We had to create every Spinnaker Kubernetes V1 object manually
33. FOR EXAMPLE: CONTAINER ENVS
34. OR POD SECURITY CONTEXT WE JUST ADDED THESE AS WE DISCOVERED THEM...
36. WE GOT WHAT WE WANTED HOWEVER!
37. SPINNAKER PIPELINE JSON!
38. HAS JSON EVER MADE YOU THAT HAPPY? THIS JSON LETS US CONFIGURE PIPELINES IN THE UI ▸ We can take the output of this command, copy it, and paste it into the Spinnaker UI via the "Edit as JSON" option
39. BADA BING BADA BOOM ▸ Our Spinnaker Pipeline never is managed via the UI (with the exception of pasting the JSON) ▸ All of the pipeline.yml files are checked into source control ▸ Reverts to the codebase allows us to revert pipelines easily ▸ Track record of who changed what in pipelines ▸ The wins are numerous.
40. ALTERNATIVES CONSIDERED ▸ We could have configured every project manually and rolled that way without version control. ▸ We could have let teams own their own process. ▸ But the long term spend of that is enormous ▸ Configuring via the UI would be tedious and error prone ▸ Linear scale of time spent as we add more applications
41. MAKE THE CHANGE EASY THEN MAKE THE EASY CHANGE Kent Beck A SIMPLE PHILOSOPHY
42. A GREAT START WE MADE THE CHANGE EASY By building k8s-pipeliner and creating concrete types to represent a Spinnaker pipeline, we set ourselves up for even bigger successes with little work.
43. AUTOMATIC PIPELINE UPDATES PART 2
44. STICK SHIFT TO AUTOMATIC AVOIDING THE UI AS A SERVICE ▸ Disclaimer: I love Spinnaker. We use it for every project we deploy to Kubernetes (which is basically everything) ▸ But the UX for managing pipelines is clunky ▸ UI/UX is hard to get right for something that can do so much (See: AWS) ▸ Spinnaker suffers from this (great) problem because it can do so much.
45. STICK SHIFT TO AUTOMATIC OUR IDEAL FLOW ▸ Instead of pasting JSON into a text field in the pipeline configuration page, this should be automatic. ▸ Whenever a merge is performed into master for a project, we'll send a Github webhook to an internal service ▸ That project will then clone the project, read the pipeline config file(s), and then update the application's pipeline
46. INTRODUCING ESTUARY
47. GITHUB WEBHOOK ESTUARY GIT CLONE RUN K8S-PIPELINER POST TO SPINNAKER-GATE
48. The tidal mouth of a large river, where the tide meets the stream. WHAT IS ESTUARY ▸ Estuary is a simple HTTP server that receives GitHub webhooks, builds a pipeline config, and then updates the Spinnaker Pipeline via the Gate API. ▸ Written in Go ▸ Imports k8s-pipeliner to generate the JSON ▸ Can inject values into pipelines automatically as well
49. SNOOPING AROUND FIGURING OUT THE API REQUEST ▸ We've configured Gate to authenticate via Github and A X509 Certificate ▸ Using the X509 Certificate allows us to send POST requests to update the pipeline from Estuary ▸ But what does this request look like exactly?
50. SNOOPING AROUND FIGURING OUT THE API REQUEST ▸ Spinnaker has a Swagger file for it's API ▸ However... we found it easier to just look at what the UI sends in its XHR requests ▸ Once we figured that out, we can easily update the Pipeline
51. AUTO PIPELINE UPDATE WHAT WE'LL SEE ▸ You'll see a simple pipeline.yml file get updated ▸ Estuary in the background receives the github webhook ▸ Estuary then clones and updates the pipeline accordingly
53. RELEASE LOGS PART 3
54. LOG ME MAYBE WHAT ARE RELEASE LOGS ▸ A structured entry of what was deployed, when, and other metadata ▸ Stored forever, most recent first ▸ Can be used for identifying what release broke what ▸ Metrics! ▸ Release managers love it
55. HOW? HOW DO YOU CREATE THESE LOGS? ▸ Spinnaker has a module called Echo that can send webhook requests on task events ▸ We've configured Echo to send webhooks to our "Release Management Software" ▸ Release Management Service parses the webhook and creates log entries when pipeline stages complete
56. SPINNAKER ECHO RELEASE MANAGEMENT SERVICE STORAGE
57. SPINNAKER ECHO RELEASE MANAGEMENT SERVICE STORAGE SO WHAT SHOULD WE USE FOR STORAGE?
59. I'M NOT KIDDING NO
60. OH YEAH
61. THE PAYLOAD HOW DO YOU EVEN GET THIS PAYLOAD? ▸ We could not find a single page of documentation about what this payload looks like in full ▸ Little did we know a typical payload for one of our pipelines was 870kb of JSON (that's enormous) ▸ So, we just sent the payload to http://webhook.site
62. HTTPS://WEBHOOK.SITE CREATE A NEW URL BY VISITING THE SITE
63. USE THE WEBHOOK MODIFY ECHO'S CONFIG TO INCLUDE IT Kick off a pipeline (preferably one without sensitive data) and watch webhook.site fill up.
64. DEFINED GO STRUCTS
65. CLOSING UP MY THOUGHTS ▸ All Open Source projects are puppets ▸ On their own, they're lifeless, they provide very little or no business value on their own ▸ We as engineers must be the puppeteers and give these projects the personality we want them to have ▸ Building tools around them is just one of those ways to give a project like Spinnaker the life we want it to have
66. BUILD ALL THE TOOLS. FUTURE YOU WILL LOVE YOU MORE FOR IT
67. @BOBBYTABLES
 THANKS! BOBBYTABLES