Develop it, Test it, Ship it, Measure it - Building an end-to-end CI solution for Neos projects

The development of a complex Neos project is not done by single developers. A team of specialists will be involved in different parts of your project workflow. And not all of them are developers.

How can you provide a great development (and operational) experience that automates as much as possible? And how can end-to-end tests or measurements for core web vitals be integrated into the development pipeline? Are containers the answer? And what can a CI tool do for you?

In this talk we will have a 360° look at the DevOps lifecycle and build the perfect project pipeline using tools like GitLab CI, Kubernetes, Helm, Playwright and Lighthouse CI.

Transcript

1. Develop it
   
   
   Test it
   
   
   Ship it
   
   
   Measure it Christopher Hlubek (@hlubek)
   Building an end-to-end CI
   

   solution for Neos projects.
   

   View Slide

2. 💖 Automation
   

   View Slide

3. 😡 Automation
   It's always the little things...
   

   View Slide

4. Why automate?
   

   View Slide

5. Care about
   

   Developer experience
   Creator
   

   View Slide

6. Agile >> Fast feedback loops
   

   View Slide

7. Capture knowledge
   

   View Slide

8. Continuous
   
   
   Integration
   Check that changes don't
   

   break anything
   

   View Slide

9. Run a build
   Compile code
   
   
   Build frontend assets
   Continuous Integration
   

   View Slide

10. Static checks
    Linter
    
    
    Typechecker
    
    
    TypeScript
    Continuous Integration
    

    View Slide

11. Verify code
    
    
    With tests
    Unit Tests
    
    
    Functional Tests
    or Fusion Snapshot Tests [https://github.com/suf
    fl
    e/Suf
    fl
    e.Snapshot]
    Continuous Integration
    

    View Slide

12. Make
    
    
    Deployments
    
    
    Easy and safe
    So you can do it often.
    

    View Slide

13. Build artifacts
    
    
    For delivery
    

    View Slide

14. Get early
    
    
    Feedback
    

    View Slide

15. Review deployments
    Preview deployments
    

    View Slide

16. Verify
    
    
    Correctness
    

    View Slide

17. Do manual tests
    On disposable instances
    

    View Slide

18. Run end-to-end
    
    
    Tests on environments
    

    View Slide

19. Measure performance
    Web vitals Metrics
    

    View Slide

20. How we do it
    

    View Slide

21. Monorepo
    

    View Slide

22. 1 Repository
    Many components
    Monorepo
    

    View Slide

23. Monorepo
    JS app
    Native app
    Neos
    Microservice
    Documentation
    

    View Slide

24. GitLab CI
    

    View Slide

25. Project
    = Repository
    GitLab CI
    

    View Slide

26. .gitlab-ci.yml
    CI Con
    fi
    guration in Git Repo
    GitLab CI
    Yay, YAML!
    

    View Slide

27. Pipeline
    A CI run with multiple Jobs in Stages
    GitLab CI
    

    View Slide

28. Job
    Runs in a container
    GitLab CI
    Artifacts
    Rules Cache
    Script
    

    View Slide

29. GitLab CI
    Build
    assets
    composer
    Test
    php:unit
    php:functional
    Deploy
    Docker-build
    app
    js:unit
    integration
    Acceptance
    acceptance
    Stage by stage
    

    View Slide

30. GitLab CI
    assets
    composer php:unit
    php:functional
    app
    js:unit
    integration acceptance
    app:
    
    
    stage: docker-build
    
    
    needs:
    
    
    - composer
    
    
    - assets
    
    
    With graph
    

    (DAG)
    

    View Slide

31. Deployments /
    
    
    Environments
    

    View Slide

32. Review
    Auto-deployed per
    

    merge request
    
    
    Reset on each deploy
    
    
    Dynamic URL
    Deployments / Environments
    https:/
    /my-project-task-update.cluster.dev
    

    View Slide

33. Integration
    Auto-deployed for
    

    main branch
    
    
    Reset on each deploy
    
    
    Fixed URL
    Deployments / Environments
    https:/
    /my-project-integration.cluster.dev
    

    View Slide

34. Staging
    Auto-deployed for tags
    
    
    Data / DB is persistent
    
    
    Fixed URL
    Deployments / Environments
    https:/
    /my-project-staging.cluster.dev
    

    View Slide

35. Integration vs. Staging
    Deployments / Environments
    Most current state
    
    
    Always fresh
    
    
    Reference system
    Preview release
    
    
    Test migrations
    
    
    Use prod content / data
    

    View Slide

36. Production
    Auto-deploy for
    

    version tags
    

    3.7.0
    
    Manual deploy for
    

    release candidates
    

    3.7.2-rc
    Deployments / Environments
    

    View Slide

37. Kubernetes For CI jobs and dev
    
    
    deployments.
    

    View Slide

38. Worker 1
    Kubernetes
    Worker 2
    Worker 3
    Worker 4
    Controlplane 1
    Controlplane 2
    Controlplane 3
    Physical view
    

    View Slide

39. Kubernetes
    Namespace A
    Logical view
    Namespace B
    Deployment
    Service
    Ingress
    Pod
    Container
    Pod
    Container
    

    View Slide

40. Kubernetes
    Namespace dev-my-project
    CI and dev deployments
    Namespace gitlab-ci
    Deployment gitlab-runner
    Pod job-123 Pod job-124
    Deployment integration
    Ingress integration.domain.dev
    Deployment review-foo
    Ingress review-foo.domain.dev
    

    View Slide

41. Access from
    

    GitLab CI
    Service account of namespace

    can be connected via certi
    fi
    cate
    Kubernetes
    Will be replaced by
    

    GitLab Kubernetes Agent
    

    View Slide

42. Manifests
    = Lots of YAML
    Kubernetes
    Declarative Resources Metadata
    How to manage?
    

    View Slide

43. Helm
    

    View Slide

44. Templates
    
    
    YAML
    
    
    Kubernetes
    Helm
    

    View Slide

45. Charts
    Bundled in monorepo
    
    
    Templates
    

    for resources
    
    
    Customize through values
    Helm
    

    View Slide

46. Releases
    Metadata for release
    
    
    Can be managed via
    Helm CLI
    
    
    Easy to use via CI
    Helm
    

    View Slide

47. Unit Test
    Snapshots & assertions
    
    
    Test different values
    
    
    Verify YAML is correct
    Helm
    https://github.com/quintush/helm-unittest
    

    View Slide

48. Values
    
    
    Example
    default:
    
    
    image:
    
    
    # A default image tag for the project images (excluding Nginx)
    
    
    tag: 'latest'
    
    
    postgres:
    
    
    # A locally deployed PostgreSQL database (without persistence)
    
    
    enabled: false
    
    
    image:
    
    
    repository: "postgres"
    
    
    tag: "13"
    
    
    pullPolicy: Always
    
    
    env:
    
    
    POSTGRES_USER: neos
    
    
    POSTGRES_PASSWORD: only-for-development
    
    
    service:
    
    
    type: ClusterIP
    
    
    port: 5432
    
    
    neos:
    
    
    replicaCount: 1
    
    
    revisionHistoryLimit: 0
    
    
    strategyType: Recreate
    
    
    # Neos Deploymnent uses 2 containers for PHP FPM and a Nginx webserver
    
    
    php:
    
    
    image:
    
    
    repository: 'registry.networkteam.com/chlubek/neos-ci-end-to-end/neos'
    
    
    # tag: 'latest'
    
    
    pullPolicy: Always
    
    
    env:
    
    
    FPM_FASTCGI_ADDRESS: localhost:9000
    
    
    FLOW_HTTP_TRUSTED_PROXIES: "*"
    
    
    resources: {}
    
    
    nginx:
    
    
    image:
    
    
    repository: 'registry.networkteam.com/networkteam/docker/nginx'
    
    
    tag: '1.3'
    
    
    pullPolicy: Always
    
    
    env:
    
    
    FPM_FASTCGI_ADDRESS: localhost:9000
    
    
    resources: {}
    
    
    # We usually recommend not to specify default resources and to leave this as a conscious
    
    
    # choice for the user. This also increases chances charts run on environments with little
    
    
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    
    
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    
    
    # limits:
    
    
    
    Helm
    

    View Slide

49. Overriding
    
    
    from CI
    deploy:integration:
    
    
    extends: .helm-deploy-job
    
    
    # Use Helm image and run chart in ci/helm with overrides from env vars
    
    
    image: $HELM_IMAGE
    
    
    stage: deploy
    
    
    rules:
    
    
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    
    environment:
    
    
    name: integration
    
    
    url: https://my-project-integration.cluster.dev
    
    
    variables:
    
    
    RELEASE_NAME: integration
    
    
    script:
    
    
    - |
    
    
    cat < overrides.yaml
    
    
    neos:
    
    
    php:
    
    
    env:
    
    
    NEOS_CREATE_ADMIN_USER_NAME: admin
    
    
    NEOS_CREATE_ADMIN_USER_PASSWORD: password
    
    
    NEOS_IMPORT_SITE_PACKAGE_KEY: 'Neos.Demo'
    
    
    ingress:
    
    
    hostnames:
    
    
    - my-project-$RELEASE_NAME.cluster.dev
    
    
    tls:
    
    
    - secretName: wildcard-cluster-dev
    
    
    hosts:
    
    
    - my-project-$RELEASE_NAME.cluster-dev
    
    
    default:
    
    
    image:
    
    
    tag: $CI_REGISTRY_TAG
    
    
    annotations:
    
    
    app.gitlab.com/env: $CI_ENVIRONMENT_SLUG
    
    
    app.gitlab.com/app: $CI_PROJECT_PATH_SLUG
    
    
    postgres:
    
    
    enabled: true
    
    
    EOF
    
    
    # Make sure the previous release is completely removed
    
    
    - helm delete $RELEASE_NAME > /dev/null 2>&1 || echo "No previous release found"
    
    
    # Install the chart and wait until everything is ready,
    
    
    # otherwise print logs of failed pods and fail the build
    
    
    - |
    
    
    (helm install --wait -f overrides.yaml $RELEASE_NAME . && echo "Done.") \
    
    
    || (inspect-failed-release $RELEASE_NAME && false)
    
    
    Helm
    

    View Slide

50. Kaniko
    https://github.com/GoogleContainerTools/kaniko
    

    View Slide

51. Unprivileged
    
    
    Image builds
    No Docker-in-Docker
    
    
    Runs in a container
    
    
    Shared build cache
    Kaniko
    

    View Slide

52. Custom
    
    
    Kaniko Image
    Build script
    
    
    Registry auth
    
    
    Push to Harbor
    

    (for tags)
    Kaniko
    Built using low-level tools
    

    (Skopeo, Umoci)
    
    
    

    View Slide

53. Bonus
    

    Reproducible
    
    
    Builds
    Exact digest (hash)
    with same file
    content
    Kaniko
    

    View Slide

54. View Slide

55. Playwright
    

    View Slide

56. Code tests
    
    
    In TypeScript
    Playwright
    import { test as base, expect, Page, Locator } from '@playwright/test';
    
    
    // ...
    
    
    test.describe('ContactForm', () => {
    
    
    test('success', async ({ contactForm }) => {
    
    
    await contactForm.fillField('Name', 'Christopher');
    
    
    await contactForm.fillField('Email', 'j.doe@example.com');
    
    
    await contactForm.fillField('Message', 'Hi there, this is a test!');
    
    
    await contactForm.getSubmitButton().click();
    
    
    await expect(contactForm.message).toHaveText('Thank you!');
    
    
    });
    
    
    test.describe('validation', () => {
    
    
    test('required field', async ({ contactForm }) => {
    
    
    await contactForm.fillField('Name', 'Christopher');
    
    
    await contactForm.fillField('Email', 'j.doe@example.com');
    
    
    await contactForm.getSubmitButton().click();
    
    
    await expect(contactForm.errorFor("Message"))
    
    
    .toHaveText('This property is required');
    
    
    });
    
    
    test('email field', async ({ contactForm }) => {
    
    
    await contactForm.fillField('Email', 'invalid');
    
    
    await contactForm.getSubmitButton().click();
    
    
    await expect(contactForm.errorFor("Email"))
    
    
    .toHaveText('Please specify a valid email address');
    
    
    });
    
    
    
    
    ContactForm.spec.ts
    

    View Slide

57. Use Page Object Model
    

    To abstract selectors
    Playwright
    Especially for forms!
    
    
    import { expect, Locator } from "@playwright/test";
    
    
    export class FormObject {
    
    
    protected form: Locator;
    
    
    constructor(form: Locator) {
    
    
    this.form = form;
    
    
    }
    
    
    async isVisible() {
    
    
    await expect(this.form).toBeVisible();
    
    
    }
    
    
    getField(label: string) {
    
    
    return this.form.locator(
    
    
    `.form-group:has(label:has-text("${label}")) .form-control`
    
    
    );
    
    
    }
    
    
    errorFor(label: string) {
    
    
    return this.form.locator(
    
    
    `.form-group:has(label:has-text("${label}")) .errors`
    
    
    );
    
    
    }
    
    
    async fillField(label: string, value: string) {
    
    
    const field = this.getField(label);
    
    
    await expect(field).toBeVisible();
    
    
    await field.fill(value);
    
    
    }
    
    
    getSubmitButton() {
    
    
    return this.form.locator('button[type="submit"]');
    
    
    }
    
    
    }
    
    
    

    View Slide

58. Use integration / review
    
    
    For acceptance tests
    Playwright
    

    View Slide

59. Config
    
    
    Example
    import { PlaywrightTestConfig } from "@playwright/test";
    
    
    const config: PlaywrightTestConfig = {
    
    
    testMatch: "DistributionPackages/**/*.spec.ts",
    
    
    use: {
    
    
    baseURL: process.env.BASE_URL || "http://localhost:8081",
    
    
    trace: "on-first-retry",
    
    
    },
    
    
    // Enable retries for CI and to capture videos, but not for local development
    
    
    retries: process.env.CI ? 2 : undefined,
    
    
    // Enable list reporter (defaults to dot) and junit for CI
    
    
    reporter: process.env.CI
    
    
    ? [["list"], ["junit", { outputFile: "test-results/report.xml" }]]
    
    
    : "list",
    
    
    };
    
    
    export default config;
    Playwright
    

    View Slide

60. Test
    

    Presentational components
    
    
    Via Monocle
    Playwright
    

    View Slide

61. Neos
    
    
    Fixtures
    Playwright
    Site dump with
    

    fixed content
    
    
    Provide Rest API to
    

    set up fixture data
    

    View Slide

62. Reliable
    
    
    Tests
    Playwright
    Tests can be retried
    
    
    Auto-wait with
    timeout
    

    View Slide

63. Traces
    
    
    For debugging
    Playwright
    Saved as artifact
    
    
    Can be viewed
    locally
    

    View Slide

64. ... and more
    Playwright
    Extend Playwright
    fixtures (type-safe)
    
    
    Intercept and assert
    XHR requests
    
    
    Easy to run headless
    in CI via container
    

    View Slide

65. Lighthouse CI
    LHCI for short.
    

    View Slide

66. LHCI
    Lighthouse CI
    Standalone server
    
    
    Build results can be
    reported to LHCI
    
    
    Maintains history to
    compare reports
    

    View Slide

67. Deploy the
    
    
    Server
    Lighthouse CI
    Deployed in dev
    cluster
    
    
    Integrated into
    GitLab via webhook
    

    View Slide

68. Measure
    Lighthouse CI
    Run for deployed
    environments
    
    
    Job template is
    added via include
    
    
    🚫 Do not run on
    shared instances /
    containers
    

    View Slide

69. Compare
    Lighthouse CI
    

    View Slide

70. What it looks like
    

    View Slide

71. Demo project
    https://github.com/networkteam/neos-ci-end-to-end
    

    View Slide

72. View Slide

73. View Slide

74. View Slide

75. View Slide

76. View Slide

77. View Slide

78. View Slide

79. View Slide

80. View Slide

81. View Slide

82. Seriously
    
    
    YAML is okay
    You need to know the rules.
    

    There are strange ones.
    
    
    

    View Slide

83. What we learned
    

    View Slide

84. View Slide

85. Reliability is key
    

    View Slide

86. Pipelines
    

    Must not be slow
    

    View Slide

87. Share common
    
    
    CI configuration
    But beware having
    

    too much indirection
    
    
    

    View Slide

88. Cache what you can
    

    View Slide

89. Building Docker images
    
    
    Is Hard™
    

    View Slide

90. Fin
    Let's get in contact about Neos and CI
    

    Christopher Hlubek (@hlubek)
    https://github.com/networkteam/neos-ci-end-to-end
    

    View Slide