[2019.11 Meetup] [TALK] Robert Kuhr - CI_CD at MAN Truck & Bus

Dream, Code, Test, Deploy, Repeat. These are the steps of today's modern product teams. MAN-IT is growing and needed to provide its technologists with a modern way to achieve their goals.

MAN's platform team needs to support cloud, on-premise and embedded-engineering development teams. After surveying the CICD landscape and after much consideration, the decision was to base our CICD infrastructure on Gitlab. Gitlab not only provides the centralized source code repositories for all of our team, but also provides a core tool chain for all development teams, regardless of which technological stack they are developing for providing pipelines for building, testing, deploying and other functionality. In conjunction with Gitlab, MAN is providing SonarQube for best-practices and Static Source Code Analysis along with JFrog Artifactory for hosting artifacts for Maven, NPM, and docker images (with maybe more things in the future).

One challenge MAN faces is the onboarding of users and synchronizing the userbase across these additional platforms. This talk will dive into how we are using Gitlab, Sonar, JFrog-Artifactory, where we started with it, where we are going, how we are using AWS Serverless functionality (lambda, streams, step functions, etc) to get us to where we are going and how the CICD/Cloud-team are dog-fooding all of our tools to make sure we can feel the same joys and pains as our user base. It will go from a higher level to start and quickly get technical when discussing our serverless solutions.

Robert Kuhr is a Cloud Architect at MAN Truck & Bus AG. Robert architects pragmatic and valuable solutions in the cloud as well as on-prem and in the embedded world. Additionally, he guides teams in crafting solutions to accomplish their goals reasonably while taking account for best practices in CICD and DevOps processes.

Transcript

1. DEVOPS MEETUP LISBON HOW MAN CICDS • Robert Kuhr Photo by Alex Paganelli on Unsplash MAN | Digital Hub MAN Truck & Bus´s HQ
2. 34x + 6x + We drive transportation to the next level by creating a startup within our corporation • MAN Digital Hub: Our journey so far Jan 2018 Kick-off Nov 2017 Go! 2020 Jul 2018 Start Operations Aug 2018 New Office 100x Office Opening Oct 2018 Nov 2018 Volkswagen Press Event Today Setup, Growing, Recruiting, Scouting, Entering Scene, … We are currently growing up our Cloud Platform and CI/CD Platform Team! MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs
3. Agenda 1 How do we get this project out the door? 2 How do we use our tools? 3 What are our CICD team’s goals? 4 How good does our dogfood taste? 5 Examples under the hood 6 Growing our platform 7 Q & A
4. HOW DO WE GET THIS PROJECT OUT THE DOOR? •1I HAVE SO MUCH MAGIC TO SHARE WITH MY USERS!
5. What tools are in your toolbox? • Version Control • Github/Bitbucket/Stash • AWS Code Commit • SVN/CVS • Build, Test & Deploy Tools • jenkins • Gocd/TravisCI/CircleCI • AWS Code Pipelines • How do you generate output? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we get this PROJECT out the door?
6. What does MAN need? • Centralized core functionality • Allow for sophisticated and yet simple CICD definitions • Customizable and expandable • “output” agnostic • Challenges and Goals at MAN How do we get this PROJECT out the door? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs
7. MAN is using Gitlab! (Self Hosted - Starter) J It is a centralized J CI/CD is built-in (Gitlab CI). No extra software required for pipelines J Gitlab Runners can be customized and flexible for your targets J Pipelines are defined as simple or as complicated as you need L Not using SaaS due to data being stored only in US regions (GDPR) J In Process of being rolled out to the whole company • Why? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we get this PROJECT out the door?
8. MAN’s CICD platform has a “support cast” • 2 more pieces to the puzzle MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we get this PROJECT out the door? SonarQube (Self hosted – Community Ed) Was a requirement setup by a small section of developers Could be removed if we move to Gitlab Ultimate Jfrog Artifactory (SaaS – Enterprise Ed) The following image stores were required to be supported: • Maven • Npm • Docker Images • Debian Packages • NuGet Packages • VM Images (like Vagrant)
9. HOW DO WE USE OUR TOOLS? •2
10. Gitlab Pipelines and Jobs • What is it we want to do today? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we use our tools? Gitlab Pipelines are a list of jobs to be run in a particular stage order The can be triggered by, commits, by schedules, manually and other automated events Gitlab Jobs are defined packages of scripts that run in a container. The minimal information a Job must define: • image: docker image to contain the job • script: list of commands to be run in the container • stage: which step/stage you want the job to run +
11. Gitlab Runners • It executes your jobs, so you don’t have to! MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we use our tools? Examples of MAN Gitlab Runners : • Auto Scaling Group of EC2 runner instances • VMWare instances in our on-premise data centers deploying to Tomcat clusters • Local machines running embedded hardware tests • More in the works….. • Gitlab runners are demons that poll Gitlab for jobs to run and then executes them • Runs on any OS that can make a Go binary • Only needs your Gitlab URL and a token to register • Runners can be tied to a group or a singular project or shared for everyone to use
12. Gitlab CICD Pipeline Variables • Passing information to your job about the environment MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we use our tools? “pre-defined” environment variables examples Gitlab has “pre-defined” environment variables along with developer defined “custom” ones CI_COMMIT_SHA commit revision for which project is built CI_JOB_TOKEN Token used for auth to gitlab docker reg CI_PROJECT_PATH The namespace with project name CI_REGISTRY GitLab’s Container Registry AWS_ACCNUM An account number for AWS PROD_URL A production URL for something TTL_PROD A number in seconds for a time to live Etc. etc. etc …….. “custom” environment variables can live at different levels.
13. Gitlab Job Containers • What needs to be in a container? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we use our tools? Are you familiar with creating docker containers like this? Or this? Then you already know how to build images for Gitlab Job!
14. Simple Example • Test and Deploy MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we use our tools? A simple example using python that tests and deploys to AWS
15. Start making it more `DevOpsy` • Validating Deployments MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How do we use our tools? Surround the previous pipeline with these
16. WHAT ARE OUR CICD TEAM’S GOALS? •3 • Not just a ‘platform team’ • Team of enablers in Cloud, CICD and DevOps Methodologies. • Hands-on consultancy approach with onboarding and software development • Creating example templates for teams to use (populating CICD Central) • Platform Management will become a very small part of team
17. Onboarding - Web Teams • AWS Cloud or On-Premise? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How we are hitting the nets! AWS Cloud: • The default cloud accounts per project are ‘development’, ’integration’, and ‘production’ • AWS Cloud Formation Stack, consisting of an autoscaling group for the runners, deployed to each account. • Target environments for running jobs (and deploying) are determined by job tags On Premise: • ‘development’ and ‘production’ targets are used for on-premise tomcat server clusters • There is 1 runner for each target • Not much room for flexibility in what can be deployed
18. Onboarding - Embedded Engineering Teams • Embedded engineering teams are more involved to work with • Depend on a lot of virtualization of HW environments • Most virtualization is done with EC2 in AWS when possible • No production ‘targets’ (yet) • Still evolving for best practices • Ongoing process MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs How we are hitting the nets!
19. HOW GOOD DOES OUR DOG FOOD TASTE? •4Would you want to share in the bowl of MAN-CICD?
20. CICD Team is not tasting the food L • In the beginning… MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs Is our dogfood yummy? The CICD Team used Gitlab to: • A store for all of the platform infrastructure templates • Simple pipelines to run scheduled tasks to synchronize users/projects across github, sonar and jfrog • Generate a few custom containers We were not REALLY USING the tools we had other than to cut examples for development teams
21. Houston, our CICD platform has a problem… • Our sync’ing is not done yet? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs Is our dogfood yummy? Gitlab has system hooks… Maybe.. we use our tool… Users, projects & group syncs take 12+ hours to complete …to update our tool…
22. Requirements: • Use Gitlab CICD pipelines • Use Gitlab for Project Management • Use AWS Serverless • A.B.D.… always be deploying. The team decides to eat a bowl • Let’s provide an example for how to use the toolchain MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs Is our dogfood yummy?
23. CICD Team Deploying Pattern • What branches where, how? MAN Truck & Bus | DevOps Meetup Lisbon | Robert Kuhr | November 2019 | How MAN CICDs Is our dogfood yummy?
24. EXAMPLES UNDER THE HOOD •5 Let’s put our nerd-caps on and look at how we did it!
25. GROWING OUR PLATFORM •6 • Possibly upgrade to Gitlab Ultimate • Azure AD Integration • More sophisticated runners and containers for jobs • More tooling for testing and rollbacks of deployments • More shared templating to onboard projects faster • More! More! More! Progress does not sleep!
26. Q & A •7 … and thank you!