[2018.09 Meetup] [TALK #1] Filipe Dobreira - Scaling Development with Kubernetes

An overview of how Impraise built on Kubernetes and ChatOps to scale developer automation to a growing team. The problems we wanted to solve. The different solutions we tried, from Kubernetes to Qarla, through Kapow and Robin. The mistakes we made, the lessons we learned, and our plans for the future.

Filipe Dobreira is co-founder and Platform team lead at Impraise, a people enablement platform helping companies improve their performance reviews and feedback culture.

Transcript

1. Scaling Development with Kubernetes Filipe Dobreira | September 2018
2. We’re hiring! - Front-End Developer (React) - Back-end Developer (Ruby) - DevOps Engineer More positions and details at:
 jobs.impraise.com
3. Planning Analysis & Design Implementation Testing Maintenance
4. Planning Analysis & Design Implementation Testing Maintenance ???
5. Planning Analysis & Design Implementation Testing Maintenance
6. Blunder years A slow, painful and error-prone development process
9. A workflow fit for a small startup AWS Elastic Beanstalk Cumbersome interfaces, slow and unreliable update procedure Zero visibility Knowing what was deployed where was mostly done by asking through Slack or in person Manual everything Development and production deployments were almost completely manual
13. Buzz words to the rescue Docker, Kubernetes and our vision of a better workflow
14. why docker?
15. why docker?
16. Consistency and predictability
 
 Jul 20, 1969 - 10 reasons you should use Docker. Number 10 will SHOCK you! About 28,400,000 results (0.31 seconds) All Portability
 
 A brownie recipe. But first, my entire life story. Orchestration
 
 Sep 17, 1991 - The Docker secret doctors don’t want you to know! why docker?
18. Why Kubernetes? Unified management As our needs were growing, Kubernetes offered a predictable solution for managing our stack at our current and future scale Support Official and community documentation, tooling was solid Extremely flexible Kubernetes’ declarative approach and complete API means it’s easy to build on
19. Kubernetes filled our orchestration gap, and gave us a foundation to build on
20. Our assumptions about Kubernetes & developers A lot to learn We thought developers wouldn’t want to learn an entire new platform just to have their code deployed Hard to manage We thought developers wouldn’t want to deal with Kubernetes’ particulars in day-to-day operations No structure We thought Kubernetes’ bare-bones resource tooling wasn’t enough for us
21. The K stands for Kubernetes
22. The Kapow stands for whatever this is
23. Kapow Deployment es Deployment Kubernetes Ingress Kubernetes Service Kubernetes S Kubernetes Jeff Bezos Cloud Magic
24. A tool for deployment management at Impraise Tailored to fit Kapow speaks our language, and is built for our workflow Complete toolchain It includes a full toolchain for managing the development lifecycle, including monitoring and troubleshooting An API to build on Additional tools and services can build on its high- level API
25. manifestVersion: “1.1”
 name: my-app
 args:
 version: latest
 endpoints:
 exposes:
 - name: app
 host: app.impraise. containers:
 - name: app
 image: “app:{{ versio resources:
 requests:
 cpu: 1m
 env:
 - name: NODE_ENV
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 A unified configuration file
26. 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 manifestVersion: “1.1”
 name: my-app
 args:
 version: latest
 endpoints:
 exposes:
 - name: app
 host: app.impraise.io
 containers:
 - name: app
 image: “app:{{ version }}”
 resources:
 requests:
 cpu: 1m
 env:
 - name: NODE_ENV

27. 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 manifestVersion: “1.1”
 name: my-app
 args:
 version: latest
 endpoints:
 exposes:
 - name: app
 host: app.impraise.io
 containers:
 - name: app
 image: “app:{{ version }}”
 resources:
 requests:
 cpu: 1m
 env:
 - name: NODE_ENV

28. apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: myapp spec: containers: - name: myapp-container image: node resources: requests:
 cpu: 1m
 env:
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 manifestVersion: “1.1”
 name: my-app
 args:
 version: latest
 endpoints:
 exposes:
 - name: app
 host: app.impraise.io
 containers:
 - name: app
 image: “app:{{ version }}”
 resources:
 requests:
 cpu: 1m
 env:
 - name: NODE_ENV

29. 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 manifestVersion: “1.1”
 name: my-app
 args:
 version: latest
 endpoints:
 exposes:
 - name: app
 host: app.impraise.io
 containers:
 - name: app
 image: “app:{{ version }}”
 resources:
 requests:
 cpu: 1m
 env:
 - name: NODE_ENV

30. Sane(r) service configuration
 Service configuration is centrally managed, and can be injected at deployment time NODE_ENV ASSET_HOST “cdn.foo App Container 1 SECRET_KEY “xR3edxE49sA4dg0” “product App Container 2
31. It’s APIs all the way down Authentication layer Before more powerful authentication solutions for Kubernetes were widely available and supported Service discovery Development releases can automatically find different versions of backend services to work with Built for our use-cases Because we know our bottlenecks, we can build Kapow to help us overcome them
32. Kapow implements our idea of what a deployment looks like at Impraise
33. Robin Kapow CLI client
34. $ robin env pr-3071-filp RAILS_ENV=production Environment variables updated for next deployment version
35. $ robin deploy impraise/project -a version:latest Created deployment with plan ID f76efe16-093b-41e5-afa0
36. $ robin ls $ robin logs pr-3071 —follow $ robin exec pr-3071 sh
37. Qarla ChatOps at Impraise
38. Visibility is central to a healthy DevOps process
39. qarla 11:36 AM APP Deployed https://github.com/impraise/project/pull/30 Auto deployed by Qarla
 Deployment of #3071 (branch-name)
 
 status commit
 pending 0fb347c
 
 commit message
 Fix everything
 
 Use ‘@qarla help’ for commands to interact with this deployment Channels
 #development
 #deploys
 #dogs 1
42. qarla 02:01 AM APP I started that deployment for you, check #deploys
 for updates Channels
 #development
 #deploys
 #dogs filipe 02:01 AM @qarla deploy web#3071
43. qarla 11:00 AM APP The deployment train is ready to depart! @filipe @arnaud your work is on the train, please review
 the changes and stick around in case of issues.
 
 The train will depart in 15 minutes. Changes for web
 5 commits (2 merge commits) Channels
 #development
 #deploys
 #dogs 1 Changes for next
 18 commits (6 merge commits) 3
44. The deployment train explained On time Developers know exactly when it’ll start, and by when they must have their work ready to catch it Strict, but forgiving Developers can ask for a 30 minute delay twice per day, but otherwise the train departs with what’s ready Highly visible The deployment train is loud and visible, and makes successful deployments a team effort and core part of the workflow
46. Lessons learned Misguided expectations, and a lesson in keeping it simple, stupid
47. Lesson 1 Resist the urge to build everything from scratch Templating and Versioning Tools like Helm grew to offer complete and capable suites for managing large Kubernetes projects Authorization RBAC and integration with solutions like Amazon Web Services’ IAM are more secure and flexible DNS and Service Discovery Tools like ExternalDNS and Kubernetes’ native service layer are a powerful toolset on their own
48. Lesson 2 You’re building a product, and the developer is the client Talk to the client Include developers in the early stages of the process, and use their feedback Start small Give yourself time and space to iterate, and resist over-building from the start Find the line Find where the line lies between Development and Ops for your team, and decide where it should be blurred
49. Lesson 3 Good visibility makes the bad days suck less Share a common language Talking about your processes is easier if everyone’s using the same words Be loud The strength of ChatOps lies in its ability to be visible and loud, use it to rally the team around what’s important, and bring clarity to confusing situations
50. Get in touch: Filipe Dobreira
 Head of Infrastructure & Security
 filipe@impraise.com Thank you!