Develop it, Test it, Ship it, Measure it - Building an end-to-end CI solution for Neos projects

The development of a complex Neos project is not done by single developers. A team of specialists will be involved in different parts of your project workflow. And not all of them are developers.

How can you provide a great development (and operational) experience that automates as much as possible? And how can end-to-end tests or measurements for core web vitals be integrated into the development pipeline? Are containers the answer? And what can a CI tool do for you?

In this talk we will have a 360° look at the DevOps lifecycle and build the perfect project pipeline using tools like GitLab CI, Kubernetes, Helm, Playwright and Lighthouse CI.

Transcript

1. Develop it Test it Ship it Measure it Christopher Hlubek (@hlubek) Building an end-to-end CI 
 solution for Neos projects.
2. 💖 Automation
3. 😡 Automation It's always the little things...
4. Why automate?
5. Care about 
 Developer experience Creator
6. Agile >> Fast feedback loops
7. Capture knowledge
8. Continuous Integration Check that changes don't 
 break anything
9. Run a build Compile code Build frontend assets Continuous Integration
10. Static checks Linter Typechecker TypeScript Continuous Integration
11. Verify code With tests Unit Tests Functional Tests or Fusion Snapshot Tests [https://github.com/suf fl e/Suf fl e.Snapshot] Continuous Integration
12. Make Deployments Easy and safe So you can do it often.
13. Build artifacts For delivery
14. Get early Feedback
15. Review deployments Preview deployments
16. Verify Correctness
17. Do manual tests On disposable instances
18. Run end-to-end Tests on environments
19. Measure performance Web vitals Metrics
20. How we do it
21. Monorepo
22. 1 Repository Many components Monorepo
23. Monorepo JS app Native app Neos Microservice Documentation
24. GitLab CI
25. Project = Repository GitLab CI
26. .gitlab-ci.yml CI Con fi guration in Git Repo GitLab CI Yay, YAML!
27. Pipeline A CI run with multiple Jobs in Stages GitLab CI
28. Job Runs in a container GitLab CI Artifacts Rules Cache Script
29. GitLab CI Build assets composer Test php:unit php:functional Deploy Docker-build app js:unit integration Acceptance acceptance Stage by stage
30. GitLab CI assets composer php:unit php:functional app js:unit integration acceptance app: stage: docker-build needs: - composer - assets With graph 
 (DAG)
31. Deployments / Environments
32. Review Auto-deployed per 
 merge request Reset on each deploy Dynamic URL Deployments / Environments https:/ /my-project-task-update.cluster.dev
33. Integration Auto-deployed for 
 main branch Reset on each deploy Fixed URL Deployments / Environments https:/ /my-project-integration.cluster.dev
34. Staging Auto-deployed for tags Data / DB is persistent Fixed URL Deployments / Environments https:/ /my-project-staging.cluster.dev
35. Integration vs. Staging Deployments / Environments Most current state Always fresh Reference system Preview release Test migrations Use prod content / data
36. Production Auto-deploy for 
 version tags 
 3.7.0 Manual deploy for 
 release candidates 
 3.7.2-rc Deployments / Environments
37. Kubernetes For CI jobs and dev deployments.
38. Worker 1 Kubernetes Worker 2 Worker 3 Worker 4 Controlplane 1 Controlplane 2 Controlplane 3 Physical view
39. Kubernetes Namespace A Logical view Namespace B Deployment Service Ingress Pod Container Pod Container
40. Kubernetes Namespace dev-my-project CI and dev deployments Namespace gitlab-ci Deployment gitlab-runner Pod job-123 Pod job-124 Deployment integration Ingress integration.domain.dev Deployment review-foo Ingress review-foo.domain.dev
41. Access from 
 GitLab CI Service account of namespace
 can be connected via certi fi cate Kubernetes Will be replaced by 
 GitLab Kubernetes Agent
42. Manifests = Lots of YAML Kubernetes Declarative Resources Metadata How to manage?
43. Helm
44. Templates YAML Kubernetes Helm
45. Charts Bundled in monorepo Templates 
 for resources Customize through values Helm
46. Releases Metadata for release Can be managed via Helm CLI Easy to use via CI Helm
47. Unit Test Snapshots & assertions Test different values Verify YAML is correct Helm https://github.com/quintush/helm-unittest
48. Values Example default: image: # A default image tag for the project images (excluding Nginx) tag: 'latest' postgres: # A locally deployed PostgreSQL database (without persistence) enabled: false image: repository: "postgres" tag: "13" pullPolicy: Always env: POSTGRES_USER: neos POSTGRES_PASSWORD: only-for-development service: type: ClusterIP port: 5432 neos: replicaCount: 1 revisionHistoryLimit: 0 strategyType: Recreate # Neos Deploymnent uses 2 containers for PHP FPM and a Nginx webserver php: image: repository: 'registry.networkteam.com/chlubek/neos-ci-end-to-end/neos' # tag: 'latest' pullPolicy: Always env: FPM_FASTCGI_ADDRESS: localhost:9000 FLOW_HTTP_TRUSTED_PROXIES: "*" resources: {} nginx: image: repository: 'registry.networkteam.com/networkteam/docker/nginx' tag: '1.3' pullPolicy: Always env: FPM_FASTCGI_ADDRESS: localhost:9000 resources: {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following # lines, adjust them as necessary, and remove the curly braces after 'resources:'. # limits: Helm
49. Overriding from CI deploy:integration: extends: .helm-deploy-job # Use Helm image and run chart in ci/helm with overrides from env vars image: $HELM_IMAGE stage: deploy rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH environment: name: integration url: https://my-project-integration.cluster.dev variables: RELEASE_NAME: integration script: - | cat <<EOF > overrides.yaml neos: php: env: NEOS_CREATE_ADMIN_USER_NAME: admin NEOS_CREATE_ADMIN_USER_PASSWORD: password NEOS_IMPORT_SITE_PACKAGE_KEY: 'Neos.Demo' ingress: hostnames: - my-project-$RELEASE_NAME.cluster.dev tls: - secretName: wildcard-cluster-dev hosts: - my-project-$RELEASE_NAME.cluster-dev default: image: tag: $CI_REGISTRY_TAG annotations: app.gitlab.com/env: $CI_ENVIRONMENT_SLUG app.gitlab.com/app: $CI_PROJECT_PATH_SLUG postgres: enabled: true EOF # Make sure the previous release is completely removed - helm delete $RELEASE_NAME > /dev/null 2>&1 || echo "No previous release found" # Install the chart and wait until everything is ready, # otherwise print logs of failed pods and fail the build - | (helm install --wait -f overrides.yaml $RELEASE_NAME . && echo "Done.") \ || (inspect-failed-release $RELEASE_NAME && false) Helm
50. Kaniko https://github.com/GoogleContainerTools/kaniko
51. Unprivileged Image builds No Docker-in-Docker Runs in a container Shared build cache Kaniko
52. Custom Kaniko Image Build script Registry auth Push to Harbor 
 (for tags) Kaniko Built using low-level tools 
 (Skopeo, Umoci)
53. Bonus 
 Reproducible Builds Exact digest (hash) with same file content Kaniko
55. Playwright
56. Code tests In TypeScript Playwright import { test as base, expect, Page, Locator } from '@playwright/test'; // ... test.describe('ContactForm', () => { test('success', async ({ contactForm }) => { await contactForm.fillField('Name', 'Christopher'); await contactForm.fillField('Email', 'j.doe@example.com'); await contactForm.fillField('Message', 'Hi there, this is a test!'); await contactForm.getSubmitButton().click(); await expect(contactForm.message).toHaveText('Thank you!'); }); test.describe('validation', () => { test('required field', async ({ contactForm }) => { await contactForm.fillField('Name', 'Christopher'); await contactForm.fillField('Email', 'j.doe@example.com'); await contactForm.getSubmitButton().click(); await expect(contactForm.errorFor("Message")) .toHaveText('This property is required'); }); test('email field', async ({ contactForm }) => { await contactForm.fillField('Email', 'invalid'); await contactForm.getSubmitButton().click(); await expect(contactForm.errorFor("Email")) .toHaveText('Please specify a valid email address'); }); ContactForm.spec.ts
57. Use Page Object Model 
 To abstract selectors Playwright Especially for forms! import { expect, Locator } from "@playwright/test"; export class FormObject { protected form: Locator; constructor(form: Locator) { this.form = form; } async isVisible() { await expect(this.form).toBeVisible(); } getField(label: string) { return this.form.locator( `.form-group:has(label:has-text("${label}")) .form-control` ); } errorFor(label: string) { return this.form.locator( `.form-group:has(label:has-text("${label}")) .errors` ); } async fillField(label: string, value: string) { const field = this.getField(label); await expect(field).toBeVisible(); await field.fill(value); } getSubmitButton() { return this.form.locator('button[type="submit"]'); } }
58. Use integration / review For acceptance tests Playwright
59. Config Example import { PlaywrightTestConfig } from "@playwright/test"; const config: PlaywrightTestConfig = { testMatch: "DistributionPackages/**/*.spec.ts", use: { baseURL: process.env.BASE_URL || "http://localhost:8081", trace: "on-first-retry", }, // Enable retries for CI and to capture videos, but not for local development retries: process.env.CI ? 2 : undefined, // Enable list reporter (defaults to dot) and junit for CI reporter: process.env.CI ? [["list"], ["junit", { outputFile: "test-results/report.xml" }]] : "list", }; export default config; Playwright
60. Test 
 Presentational components Via Monocle Playwright
61. Neos Fixtures Playwright Site dump with 
 fixed content Provide Rest API to 
 set up fixture data
62. Reliable Tests Playwright Tests can be retried Auto-wait with timeout
63. Traces For debugging Playwright Saved as artifact Can be viewed locally
64. ... and more Playwright Extend Playwright fixtures (type-safe) Intercept and assert XHR requests Easy to run headless in CI via container
65. Lighthouse CI LHCI for short.
66. LHCI Lighthouse CI Standalone server Build results can be reported to LHCI Maintains history to compare reports
67. Deploy the Server Lighthouse CI Deployed in dev cluster Integrated into GitLab via webhook
68. Measure Lighthouse CI Run for deployed environments Job template is added via include 🚫 Do not run on shared instances / containers
69. Compare Lighthouse CI
70. What it looks like
71. Demo project https://github.com/networkteam/neos-ci-end-to-end
82. Seriously YAML is okay You need to know the rules. 
 There are strange ones.
83. What we learned
85. Reliability is key
86. Pipelines 
 Must not be slow
87. Share common CI configuration But beware having 
 too much indirection
88. Cache what you can
89. Building Docker images Is Hard™
90. Fin Let's get in contact about Neos and CI 
 Christopher Hlubek (@hlubek) https://github.com/networkteam/neos-ci-end-to-end