Cloud Run CI/CD + QA at ソウゾウ - Cloud Run Casual Talk !

2023/02/15: Cloud Run Casual Talk！〜 #1 カウシェ・ソウゾウ の CI/CD + QA 環境にDeep Dive！〜
- https://kauche.connpass.com/event/273291/
- https://www.youtube.com/watch?v=LrX3kQ-8_0Q

参考記事:
- メルカリShops の CI/CD と Pull Request 環境
- メルカリShops における Cloud Run service の Canary Deployment

Transcript

1. 1
   Cloud Run CI/CD + QA at ソウゾウ
   Ryuzo Yamamoto
   Cloud Run Casual Talk！
   

   View Slide

2. 2
   山本 竜三
   自己紹介
   @dragon3
   Software Engineer
   Lead Architect / SRE at Souzoh
   in Fukuoka
   

   View Slide

3. 3
   ソウゾウ / メルカリShops
   

   View Slide

4. 4
   ● Architecture, Tech Stack
   ● CI/CD with GitHub Actions self-hosted runner
   ● Pull Request Environment
   ● Deploy & Canary Rollout
   Agenda
   

   View Slide

5. 5
   Architecture
   Next.js
   Cloud Run
   GraphQL
   Cloud Run
   imgproxy
   Cloud Run
   microservice
   Cloud Run
   microservice
   Cloud Run
   Cloud
   Storage
   Cloud Load
   Balancing
   Cloud SQL
   Memorystore
   Cloud Run
   (70~ services)
   microservice(s)
   Cloud Run
   

   View Slide

6. 6
   Tech Stack
   ● Monorepo
   ○ Go, TypeScript, Python, Java
   ○ 70~ microservices
   ● Bazel, Turborepo
   ● GraphQL / gRPC
   ● Serverless (Cloud Run)
   ● PostgreSQL, Redis
   ● Cloud PubSub, Tasks, Workflows, Scheduler, VertexAI
   

   View Slide

7. 7
   ● Architecture, Tech Stack
   ● CI/CD with GitHub Actions self-hosted runner
   ● Pull Request Environment
   ● Deploy & Canary Rollout
   Agenda
   

   View Slide

8. 8
   CI/CD with GitHub Actions self-hosted runner
   monorepo
   self-hosted runners
   runner
   runner
   job
   job
   Cloud Run
   Service
   Deploy
   GCR
   Push
   development
   Same as development
   production
   External
   Services
   NAT
   runner
   job
   Bazel
   Remote Cache
   

   View Slide

9. 9
   ● Architecture
   ● CI/CD with GitHub Actions self-hosted runner
   ● Pull Request Environment
   ● Deploy & Canary Rollout
   Agenda
   

   View Slide

10. 10
    Pull Request Environment
    Pull Request による変更をマージする前にデプロイし、
    End-to-End でテストできる環境
    Stable Environment (main)
    Pull Request Environment #123
    Pull Request Environment #465
    

    View Slide

11. 11
    Pull Request Environment
    Next.js
    Cloud Run
    Cloud Load
    Balancing
    stable
    pr123
    Host: example.com
    X-PR-ENV: 123
    Host: pr123.nextjs.example.com
    GraphQL
    Cloud Run
    stable
    pr123
    Service A
    Cloud Run
    stable
    pr123
    URL mask routing
    ..example.com
    

    View Slide

12. 12
    Pull Request Environment
    Next.js
    Cloud Run
    Cloud Load
    Balancing
    stable
    pr123
    Host: example.com
    X-PR-ENV: 123
    Path: /graphql
    Host: pr123.graphql.example.com
    URL mask routing
    GraphQL
    Cloud Run
    stable
    pr123
    Service A
    Cloud Run
    stable
    pr123
    Cloud Run Tag URL
    https://pr123---echo-XXXXXXXXXX-an.a.run.app
    ..example.com
    

    View Slide

13. 13
    Pull Request Environment
    self-hosted runners
    runner
    GCR
    development
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    main
    pr123
    Service A
    Cloud Run
    stable
    pr123
    Deploy
    job
    

    View Slide

14. 14
    ● Architecture
    ● CI/CD with GitHub Actions self-hosted runner
    ● Pull Request Environment
    ● Deploy & Canary Rollout
    Agenda
    

    View Slide

15. 15
    Deploy - Custom deploy tool
    # Production 環境に deploy
    cli deploy main \
    --spec production/services.yaml \
    --service echo \
    --image-tag 1.0.0
    # Pull Request 環境 #123 に deploy
    cli deploy pr \
    --spec development/services.yaml \
    --service echo \
    --number 123
    # production/services.yaml
    project: awesome-project
    environment: production
    location: asia-northeast1
    services:
    - name: echo
    image: gcr.io/awesome-project/echo
    env:
    - name: LOG_LEVEL
    value: info
    auto_scaling:
    min: 0
    max: 100
    capacity:
    memory: 512Mi
    cpu: 1
    concurrency: 100
    request_timeout: 30
    connection:
    use_http2: true
    ...
    

    View Slide

16. 16
    Canary Rollout
    canaryrollout
    Cloud Run
    Service A
    Cloud Run
    stable canary
    50% 50%
    Service A
    Cloud Run
    stable
    100%
    Service A
    Cloud Run
    stable canary
    100% 0%
    Service B
    Cloud Run
    stable canary
    50% 50%
    Service B
    Cloud Run
    stable
    100%
    Service B
    Cloud Run
    stable canary
    100% 0%
    Error
    

    View Slide

17. 17
    Service A
    Cloud Run
    Cloud Scheduler
    Canary Rollout
    canaryrollout
    Cloud Run
    Service A
    Cloud Run
    stable
    canary
    Trigger every minute
    Update traffic
    Custom deploy tool
    Monitoring
    Get metrics
    labels:
    canaryrollout: enabled
    

    View Slide

18. 18
    Canary Rollout
    # production/services.yaml
    ...
    services:
    - name: echo
    ...
    canary_rollout:
    enable: true
    rollout_percent_steps: [10, 30, 60, 100]
    min_requests: 50
    metrics_provider_type: grpc
    time_between_rollouts: 300s
    max_error_rate: 0.1
    ...
    # Cloud Run Service
    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
    name: echo
    ...
    labels:
    canaryrollout: enabled
    ...
    annotations:
    canaryrollout.souzoh.com/rolloutPercentSteps: 10,30,60,100
    canaryrollout.souzoh.com/minRequests: '50'
    canaryrollout.souzoh.com/metricsProviderType: grpc
    canaryrollout.souzoh.com/timeBetweenRollouts: 300s
    canaryrollout.souzoh.com/maxErrorRate: '0.1'
    ...
    Deploy
    

    View Slide

19. 19
    ● Architecture
    ● CI/CD with GitHub Actions self-hosted runner
    ● Pull Request Environment
    ● Deploy & Canary Rollout
    Wrap Up
    

    View Slide