2020 - Mastering hordes of CI jobs using Jenkins plugins

Jenkins plugins provide ability to automate CI test automation jobs creation (using Pipeline plugin and Job plugin) and this allows to effectively create and maintain lots of test automation configurations

Presentation for Test Automation Days conference

Transcript

1. eng.evolutiongaming.com Mastering Hordes of CI Jobs using Jenkins plugins by Vyacheslav Lukashevich
2. ABOUT ME • I am a Tester • Studied Test Automation extensively • Worked on the number of great projects with Test Automation • Last couple of years did a deep dive into CI/CD topic 2
3. 3 The world leader in live dealer gaming
4. For 11 consecutive years, Evolution Gaming has been crowned “Live Casino Supplier of the Year” at the EGR B2B Awards 4
5. 5 34 new games (2017 – 2020) 300+ top tier operators Thousands of users 24/7/365 operations Lowest latency video in industry Hardware & equipment Cameras, microphones In-house developed 7000+ EVOlutioners 14 countries
6. Evolution Gaming Engineering Hubs Riga ! 2006 Amsterdam " Minsk # 2019 290+ Evolutioneers 100+ Evolutioneers 14+ Evolutioneers Tallinn $ 2016 25+ Evolutioneers
7. 7 Tech Stack • Functional Scala • Low latency event sourcing platform (kafka, Cassandra, akka) • TypeScript, React (with hooks), BabylonJS • Infrastructure as Code based oт Kubernetes • Hadoop 3.x + Hive & Spark • C++ for video recognition solutions • Python + OpenCV for automated UI testing using image recognition.
8. Test Execution Nowadays is About 8 CI/CD
9. Motivation for This Speech 9 QA engineers usually focus on the tests part Knowledge of CI allows to automate much more things!
10. Jenkins 10 Open Source 1500+ plugins Large community https://www.jenkins.io/artwork/
11. Agenda 11 1) Automating Jenkins • Configuration • Pipelines • Libraries • Jobs 2) Moving to CI/CD • Jenkins + GitLab
12. Problem in the Past Lots of Jenkins Jobs to Maintain 12
13. ~300 Test Jobs For bi-weekly Release process 13 E2E regression packs • Java, TestNG • All teams, all products, integrations with clients • ~10-40 minutes each job Executed • Nightly • After deployments
14. Manual Configuration is Cumbersome 14 • Bunch of input fields to enter • Copy-paste • Multiple jobs management • Migrate to new Java version..
15. Automate Jenkins! 15 Solution – Automated Jenkins!
16. “AS A CODE” - Power of Programming Language - Source Control Management - Code Reviews Jenkins automation 16
17. Terminology quick intro to Jenkins 17
18. Payments smoke tests on UAT 18 Job/Project https://jenkins.io/doc/book/glossary Job/Project Defines work that Jenkins need to perform
19. Build 19 Payments smoke tests on UAT Job/Project Build Execution on 17.01.2018 Result of Job execution https://jenkins.io/doc/book/glossary
20. Pipeline-Stage-Step 20 https://jenkins.io/doc/book/glossary Payments smoke tests on UAT Job/Project Build Pipeline Stage Step Execution on 17.01.2018 Smoke test pipeline • Prepare environment • Run tests • Report • Generate report • Run Maven build • Create directory to store results • Execute shell command
21. Agent 21 https://jenkins.io/doc/book/glossary Payments smoke tests on UAT Job/Project Build Pipeline Stage Step Execution on 17.01.2018 Smoke test pipeline • Prepare environment • Run tests • Report • Generate report • Run Maven build • Create directory to store results • Execute shell command Node/Agent • Machine capable to execute pipeline steps • Unix box • Docker container
22. Automation Framework for Jenkins Building Blocks • Jenkins Pipeline DSL • Job DSL • Shared libraries • Configuration as Code • 22
23. Step 1 Local Pre-Configured Jenkins 23
24. Jenkins for Experiments Quickly start Jenkins • in Docker • with Plugins • & Configuration 24
25. Configuration as Code (YAML) https://github.com/jenkinsci/configuration-as-code-plugin 25
26. Step 2 Write Pipeline Scripts Jenkins pipeline plugin https://jenkins.io/doc/book/pipeline/ 26
27. ü Get agent where to run tests ü Pull Test Code from GIT ü Build it ü Run Tests using provided parameters ü Report results Here comes most differences! Common Testing Pipeline Script 27
28. Jenkins Pipeline DSL “Pipeline as a Code” Uses Groovy language 28
29. Groovy • Jenkins supported language • Yet another “Java like” language to learn • Data types • Strings • Lists, Maps • Functions, Closures • Work with files, parse json 29
30. Step 3 Move Reusable Code Into Pipeline Library Manage Jenkins -> 30
31. 1) Create function to reuse across different pipelines 2) Use it in pipeline script Shared Library (global vars) https://www.jenkins.io/doc/book/pipeline/shared-libraries/ ü Stored in /vars directory ü File name = function name ü Implements call method 31
32. Creating Own Pipeline DSL Via reusable library functions 32
33. • Custom reporting • Messages to Slack • Gitlab comments • Parse test reports • Store data in Influx DB • Build optimizations • Trigger jobs in parallel • Pre-build test code • Create test environment 33 Libraries We Have
34. Step 4 Jenkins Job DSL Job DSL plugin “Jobs as Code” 34
35. “Seed Job” The seed job is a Jenkins job which runs a DSL script, and then generates a new job. https://tech.gogoair.com/jenkins-jobs-as-code-with-groovy-dsl-c8143837593a 35
36. Seed Script JOB DSL structure mimics Create Job UI 36
37. http://<your_jenkins_host>/plugin/job-dsl/api-viewer/index.html Documentation 37
38. Master Seed Job executes all seed scripts Executes all seed scripts in target 38
39. Define Similar Jobs Data in JSON Job name Build parameters Slack channel to report {JSON} Seed template script Job Job Job 39
40. • Users work with data in JSON • Master Seed” run on merge • Jobs automatically • Generated • Included into dashboard • Removed Regular CI Activities Simplified 40
41. • Split test scope to smaller jobs • informative dashboards • simple re-run Better Test Suite Organization 41
42. 42 Job DSL scripts Shared Libraries Pipeline scripts Local development Jenkins JSON files for similar jobs Framework structure • IDE support • Compilation • Tests
43. • Jenkins fully managed from code • Code reviews by QA, DevOps, developers • Community that understands CI slightly increased 43 Results 43
44. ~50 seed scripts ~50 pipeline scripts 23 JSON files ~2100 jobs 107 views 44
45. 45 • Simple technical migrations • Common features for all teams • Development of advanced CI/CD setup CI automation unlocks improvements
46. Next Journey Switch to Continuous Delivery (CD) 46 Time to Market Time to Recover
47. • ”You build it, you run it” • Microservices • Delivery pipelines • Testing Pyramid • Trunk based development • …. 47 47 Moving Organization to CD
48. One of biggest challenges we had Setup Continuous Delivery for frontend mono-repo in GitLab 48
49. TypeScript, React, Babylon.js GitLab repository >40 code packages >10 Teams >200 contributors 49 Frontend Mono-Repo Baccarat Poker Roulette Shared UI components Blackjack … …
50. Focus QA on exact code changes 50 50
51. • Code Review • Test Automation • Manual Testing 51 Build QA around Merge Request workflow Testing code from GIT branch
52. Changing our approach for Frontend Test Automation • Each package contains • Unit & Integration tests • Few E2E tests • Visual regression tests 52 Poker Shared UI components … src tests src tests
53. Visual Regression Tests screenshot comparison • Separate components • Games layout 53 ~1200 test cases X number of devices/ browsers baseline difference
54. Changing our CI Setup To support work in Merge Requests 1. Store CI configuration in same mono-repo 2. Launch tests for impacted packages 3. Use MR comments for communication with CI 4. Add automated quality gates before merge 54 Shared UI components poker blackjack Crazy time code change code change
55. Integrating Gitlab & Jenkins + Collaboration Code review Quality gates Builds Tests Automation 55
56. Basic Needs Have Jenkins jobs displayed in Gitlab MR 56
57. Jenkins Gitlab Plugin Webhook integration https://plugins.jenkins.io/gitlab-plugin/ 57
58. Gitlab triggers Jenkins jobs via webhook on different events 58 Gitlab Webhook Integration
59. Report Pipeline Status from Jenkins Wrap code into gitlabCommitStatus 59
60. 1. Test config in each package • Test parameters • Paths to watch 60 roulette- common french roulette code change code change american roulette
61. 2. Launch Tests for Changed Code 61
62. Test Orchestration Job Selects needed test jobs for code changes MR created Commit added 1) Get changed packages in GIT 2) Read all job configs in repo 3) Find matching jobs (between 1,2) 4) Trigger jobs from 5 to ~20 jobs triggered 62
63. Generic Test Job for Frontend Tests 1. Build code 2. Deploy it 3. Run tests based on JSON config 4. Report to Gitlab 63
64. 3. Use MR Comments for Collaboration Between CI & Engineers 64
65. Add Ability to Trigger Each Job via Comment Job responds back with results & suggestions 65
66. gitlabTriggerPhrase string with GitLab comment that triggered Jenkins 66 Jenkins Side
67. Trigger Screenshots Update from MR comment 67
68. 4. Implement Support for “Code Review“ 68 68 Additional quality gate
69. Gitlab REST API Gitlab exposes secured API to get data about: • Merge Requests • Commits • Pipelines • Users & groups • etc. https://docs.gitlab.com/ee/api/ 69
70. GIT: Package->users mapping file REST API: Get list of users who MR merge_requests/{MR_ID}/award_emoji Compare list of users with mapping Check Code Owner’s Approval Before Merge 70
71. Other things we check/do • All needed pipelines were executed • Images are compressed enough • Reporting flaky tests by rerunning failed tests 71
72. Result: QA System Around Mono-Repo Relevant tests executed on MR Can’t merge without: • All pipelines green • Code owners approved change MR comments • provide status update • used to trigger needed jobs 72
73. 40 new Jenkins jobs From 4 seed scripts 1600 runs per day 73 20-40 merges in frontend repo daily Daily deployments to production QAs focus on changes instead of regression
74. What’s next? 74
75. 1. Jenkins automation needs testing 2. Complexity of pipelines grows 3. Throughput of Jenkins could become an issue Problems 75
76. 1. Plug more automation into GitLab 2. Improve changed code footprint calculation • smarter test scope selection 3. Spawn production Jenkins instances • on as needed basis Future Plans 76
77. • CI tools need to be automated • Jenkins has rich capabilities for automating it “As A Code” • QA engineers in CI/CD field improve the overall DevOps expertise 77 Short Summary Thank You!
78. GitHub Jenkins Automation https://github.com/vjacheslavl/jenkins-automation 78