Leveraging the power of GitLab CI and DDEV to manage hundreds of sites

https://www.drupalcamp.be/en/drupal-dev-days-2022/session/leveraging-power-gitlab-ci-and-ddev-manage-hundreds-sites

At 1xINTERNET, we manage hundreds of drupal sites, some of them in them in Drupal 7, others in Drupal 8 (and a multisite in Drupal 6 that we don't talk about).

To manage all these sites, and allow our developers to just start working on their tickets, while allowing our project management to efficiently perform QA activities, we have built a set of tools based on community best-practices that reduced the effort necessary to deploy our sites to our servers, or in some cases to client servers.

The tools we use include

GitLab CI
DDEV
backstopJS
drush
DRD
etc.
In this talk I will present the tools that we have open-sourced at https://gitlab.com/1xinternet/gitlab-ci-common/ and https://gitlab.com/1xinternet/drupal-vr-testing-backstopjs that allow us to efficiently deploy a site and perform regression testing with minimum effort from the part of the project developers. Because the tools are public, we won't focus on the tools themselves, but on the process of using them, and how any Drupal project can leverage them to achieve the same productivity gains that we have.

These are tools that highly configurable, and which we keep evolving, that we want to share with the community to make them more robust, and so that we can gain from the experience of others. If you're interested in learning about them and how to collaborate with us, please attend this session.

https://www.youtube.com/watch?v=wI9SjFXL_ck&ab_channel=Drupalcamp-DrupalUserGroupBelgium

Transcript

1. Leveraging Gitlab and DDEV Open-source tools to manage 100s of sites
2. We are 1xINTERNET International team of 65 employees From all corners of the world: South America, North America, Asia and Europe Over 18 different languages Offices in Frankfurt (Headquarters), Berlin, Reykjavik and Conil de la Frontera Founded in 2013
3. João Ventura • From Faro, Portugal • Living near Darmstadt, Germany • Developing with Drupal since 4.7 • Senior Back-end Developer at 1xINTERNET • Co-organizer of Drupal Dev Days Lisbon 2018 • In orga team of Drupal Europe • Scuba instructor joao.ventura@1xinternet.de @jcnventura
4. Devops team (in 2022) • Stefan Weber 󰎈/󰎧 (🏠: 󰎲) • João Ventura 󰐨 (🏠: 󰎲) • Denis Dmitriiev 󰑒 (🏠: 󰑒 -> 󰎲) • Francisco 'Curro' Sánchez 󰎼 (🏠: 󰎼) Only 3 when this talk was first submitted. • 1 Drupal 6 multi-site 🙈 • Several Drupal 7 legacy projects 😎 • A few Drupal 8 projects waiting to be migrated 💰 • Multiple Drupal 9 projects 😊
5. Objectives • Keep our developers/project managers happy ◦ Hopefully, keep clients happy as well • Best tools to enable colleagues to focus on productive work • Leverage project commonality • DevOps should be invisible
6. Why GitLab? Before: • Code version: Bitbucket • Issues: Jira • CI: Jenkins After: • Self-hosted community edition of GitLab Commits link to issues, which link to MRs. Webhooks allow us to use /spend to trigger time-tracking.
7. Onboarding developers • Local setup ◦ Representative environment? • docker-compose.yml in each project ◦ Project switching Long setup times (~1d) Runs on my machine…
8. Local development environment tools • Lando • DDEV (our choice) • Docksal • Launchpad, Lagoon, … If your company uses DDEV, please sponsor: https://github.com/sponsors/rfay
9. DDEV use Project onboarding is fast (1-2 hours) • git pull • ddev start • ddev drush sql-sync @test @self • drush sfp-en test (custom command) Need to keep everyone at similar tool level.
10. Deploying • Hosting environments we support ◦ 1xHosting ◦ Acquia ◦ Pantheon ◦ AWS ◦ … • Drupal ◦ 7 ◦ 8 ◦ 9 ◦ … • Environments ◦ Test ◦ Stage ◦ Production
11. Keep it simple
12. Example: creating a project $ composer create-project dxp/core-composer-project ddd2022example --remove-vcs --ignore-platform-reqs --repository-url=<url> $ cd ddd2022example $ ddev config --project-name="ddd2022example" --php-version="8.1" --mariadb-version="10.4” $ ddev get drud/ddev-elasticsearch $ mv web/sites/default/settings.local.php.example web/sites/default/settings.local.php $ pushd themes/custom/granite && ./build.sh dev && popd $ drush site-install xi_profile -y --site-name=DDD2022 --site-mail=f.last@example.com
13. Contents of the project skeleton ├ .gitignore <- ignore all files added by composer ├ .gitlab-ci.yml.example <- example of .gitlab-ci file ├── composer.json <- based on drupal/recommended-project ├── drush │ ├── Commands │ │ ├── PolicyCommands.php <- prevents sql-sync to overwrite @live │ │ └── StageFileProxyCommands.php <- drush sfp-en @test │ ├── drush.yml <- sets cache tables to structure-only, etc. │ └── sites │ └── self.site.yml.example <- example of @live @stage @test ├── phpcs.xml.dist <- PHP code standards ├── README.md <- Instructions on skeleton use └── web └── sites └── default ├── services.yml <- sets cookie_lifetime to 0 because of persistent_login ├── services.yml.example <- example with debug on └── settings.local.php.example <- example of local settings (config_split, env indicator, etc.)
14. Example project .gitlab-ci.yml (1xHosting) stages: - install - deploy include: - project: '1xinternet/gitlab-ci-common' file: 'gitlab-ci-d8.yml' composer_install: stage: install script: - composer install deploy_to_live: stage: deploy extends: .drupal8_deploy variables: env: live create_backup: "true" environment: url: https://www.example.com only: - main https://gitlab.com/1xinternet/gitlab-ci-common
15. Show and tell
16. GitLab CI 💖 Composer • GitLab provides integrated composer packagist ◦ { "type": "composer", "url": "https://gitlab.example.com/api/v4/group/42/-/packages/composer/"}, • Automatic creation of packages via CI: script: - version=$([[ -z "$CI_COMMIT_TAG" ]] && echo "branch=$CI_COMMIT_REF_NAME" || echo "tag=$CI_COMMIT_TAG") - 'curl --header "Job-Token: ${CI_JOB_TOKEN}" --data $version https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/composer' • Private repos will require read-only token in composer.json
17. GitLab + Tugboat = 😍 • Tugboat is a product by Lullabot that integrates with GitHub, Gitlab or BitBucket • Base preview for each site using Stage File Proxy module and sanitized daily backup • Automatic preview for each Merge Request ◦ 1x Project Managers: • Integrated Visual Regression Testing • Custom Automatic Updates setup ◦ drush pm:security ◦ composer update ◦ drush updb ◦ drush cex -y
18. Other tools • Acquia: BLT • Pantheon: Terminus • Code Quality: SonarQube • Monitoring: Zabbix • Automation: Ansible AWX • Visualization: Grafana • SSH key management: Bastillion • Knowledge management: Drupal 😎
19. Thank you To our wonderful sponsors, our awesome community and fantastic volunteers! Platinum sponsors
20. Thank you Gold sponsors Silver sponsors
21. Questions? João Ventura joao.ventura@1xinternet.de drupal.org/u/jcnventura @jcnventura BTW, we’re hiring 😊
22. Stay in touch #ddd2022 on Drupal slack @drupaldevdays /drupaldevdays