Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Prow! Leveraging Developer-Centric CI for Your OSS Project!

Prow! Leveraging Developer-Centric CI for Your OSS Project!

Prow is a CI system maintained by Kubernetes SIG Testing to test Kubernetes on Kubernetes. Prow is designed as a pluggable system of components and it can be used as a generic CI system. The robust architecture of Prow can lead to challenges in deploying it. In the talk, we will navigate the challenges faced when deploying and using Prow, including setting up the Prow control plane components, configuring access for GitHub repos, and enabling Prow plugins. Prow is used by large projects in the CNCF landscape like Kubernetes, Knative, cert-manager, Falco, to name a few. Even though a lot of these projects have deployed Prow successfully, it is a challenge to set up Prow. The talk will highlight the common pitfalls and gotchas that one will run into when deploying Prow. The talk would cover * A roundup of Prow Architecture * Cloud resources required for Prow and setting them up * Capabilities of Prow like running tests, using GitHub comments for interaction, auto merging pull requests.

Nabarun Pal

May 18, 2022
Tweet

More Decks by Nabarun Pal

Other Decks in Technology

Transcript

  1. Prow! Leveraging Developer
    Centric CI for your OSS project!
    Arsh Sharma & Nabarun Pal
    KubeCon + CloudNativeCon Europe 2022 1

    View full-size slide

  2. $ whoarewe
    KubeCon + CloudNativeCon Europe 2022 2

    View full-size slide

  3. Arsh Sharma
    • Developer Experience Engineer at Okteto
    • Kubernetes 1.25 CI Signal Lead
    • New Contributor Ambassador for Kubernetes SIG Docs
    KubeCon + CloudNativeCon Europe 2022 3

    View full-size slide

  4. Nabarun Pal
    • Senior Engineer at VMware
    • Kubernetes Release Engineering
    • Kubernetes Code of Conduct Committee
    KubeCon + CloudNativeCon Europe 2022 4

    View full-size slide

  5. What is Prow?
    KubeCon + CloudNativeCon Europe 2022 5

    View full-size slide

  6. What is Prow?
    • Kubernetes based CI/CD system
    KubeCon + CloudNativeCon Europe 2022 5

    View full-size slide

  7. What is Prow?
    • Kubernetes based CI/CD system
    • GitHub automation through policies
    KubeCon + CloudNativeCon Europe 2022 5

    View full-size slide

  8. What is Prow?
    • Kubernetes based CI/CD system
    • GitHub automation through policies
    • Configuration as code
    KubeCon + CloudNativeCon Europe 2022 5

    View full-size slide

  9. What is Prow?
    • Kubernetes based CI/CD system
    • GitHub automation through policies
    • Configuration as code
    • Highly pluggable
    KubeCon + CloudNativeCon Europe 2022 5

    View full-size slide

  10. What is Prow?
    • Kubernetes based CI/CD system
    • GitHub automation through policies
    • Configuration as code
    • Highly pluggable
    • ChatOps via /honk style commands
    KubeCon + CloudNativeCon Europe 2022 5

    View full-size slide

  11. ChatOps
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  12. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  13. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  14. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  15. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    • Provide approval for PRs
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  16. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    • Provide approval for PRs
    • Run tests for a PR
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  17. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    • Provide approval for PRs
    • Run tests for a PR
    • A lot more stuff!
    KubeCon + CloudNativeCon Europe 2022 6

    View full-size slide

  18. Architecture
    KubeCon + CloudNativeCon Europe 2022 7

    View full-size slide

  19. Tide
    KubeCon + CloudNativeCon Europe 2022 8

    View full-size slide

  20. Tide
    • Manages pool of GitHub PRs
    KubeCon + CloudNativeCon Europe 2022 8

    View full-size slide

  21. Tide
    • Manages pool of GitHub PRs
    • Automatically retests PRs when they meet the criteria
    KubeCon + CloudNativeCon Europe 2022 8

    View full-size slide

  22. Tide
    • Manages pool of GitHub PRs
    • Automatically retests PRs when they meet the criteria
    • Merges PRs without user intervention when tests pass
    KubeCon + CloudNativeCon Europe 2022 8

    View full-size slide

  23. Testgrid
    KubeCon + CloudNativeCon Europe 2022 9

    View full-size slide

  24. Testgrid
    • Interactive dashboard to view test results
    KubeCon + CloudNativeCon Europe 2022 9

    View full-size slide

  25. Testgrid
    • Interactive dashboard to view test results
    • And they are shown in a grid!
    KubeCon + CloudNativeCon Europe 2022 9

    View full-size slide

  26. KubeCon + CloudNativeCon Europe 2022 10

    View full-size slide

  27. Configuration
    KubeCon + CloudNativeCon Europe 2022 11

    View full-size slide

  28. Inrepo
    KubeCon + CloudNativeCon Europe 2022 12

    View full-size slide

  29. Inrepo
    • Define jobs in the same repo where the source code is
    KubeCon + CloudNativeCon Europe 2022 12

    View full-size slide

  30. Inrepo
    • Define jobs in the same repo where the source code is
    • In a .prow directory or .prow.yaml file at the root of the
    repo
    KubeCon + CloudNativeCon Europe 2022 12

    View full-size slide

  31. Inrepo
    • Define jobs in the same repo where the source code is
    • In a .prow directory or .prow.yaml file at the root of the
    repo
    • Jobs are NOT defined centrally in this approach
    KubeCon + CloudNativeCon Europe 2022 12

    View full-size slide

  32. Centralizing configuration
    KubeCon + CloudNativeCon Europe 2022 13

    View full-size slide

  33. Centralizing configuration
    • Have a separate repo for all your prow jobs across different
    repos
    KubeCon + CloudNativeCon Europe 2022 13

    View full-size slide

  34. Centralizing configuration
    • Have a separate repo for all your prow jobs across different
    repos
    • Jobs are defined centrally in this approach
    KubeCon + CloudNativeCon Europe 2022 13

    View full-size slide

  35. Centralizing configuration
    • Have a separate repo for all your prow jobs across different
    repos
    • Jobs are defined centrally in this approach
    • Let's look at kubernetes/test-infra!
    KubeCon + CloudNativeCon Europe 2022 13

    View full-size slide

  36. kubernetes/test-infra
    KubeCon + CloudNativeCon Europe 2022 14

    View full-size slide

  37. Deploying your own
    instance of Prow
    KubeCon + CloudNativeCon Europe 2022 15

    View full-size slide

  38. Creating the GitHub app
    KubeCon + CloudNativeCon Europe 2022 16

    View full-size slide

  39. Add hook URL
    KubeCon + CloudNativeCon Europe 2022 17

    View full-size slide

  40. Give Repository and Org Permissions
    KubeCon + CloudNativeCon Europe 2022 18

    View full-size slide

  41. Subscribe to all events
    KubeCon + CloudNativeCon Europe 2022 19

    View full-size slide

  42. Note App ID and Private Key
    KubeCon + CloudNativeCon Europe 2022 20

    View full-size slide

  43. Run tackle
    Set up the GCP credentials before running tackle
    $ tackle -skip-github
    ...
    Let it run and pause where it asks you to specify location for
    starter.yaml.
    KubeCon + CloudNativeCon Europe 2022 21

    View full-size slide

  44. Apply the Prow CRDs
    $ kubectl apply --server-side=true -f https://raw.githubusercontent.com/kubernetes/test-infra/master/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml
    customresourcedefinition.apiextensions.k8s.io/prowjobs.prow.k8s.io serverside-applied
    KubeCon + CloudNativeCon Europe 2022 22

    View full-size slide

  45. Create a GCS bucket
    $ gcloud iam service-accounts create prow-gcs-publisher
    $ identifier="$(gcloud iam service-accounts list --filter 'name:prow-gcs-publisher' --format 'value(email)')"
    $ gsutil mb gs://master-shifus-knowledge/
    $ gsutil iam ch allUsers:objectViewer gs://master-shifus-knowledge
    $ gsutil iam ch "serviceAccount:${identifier}:objectAdmin" gs://master-shifus-knowledge
    $ gcloud iam service-accounts keys create --iam-account "${identifier}" service-account.json
    $ kubectl -n test-pods create secret generic gcs-credentials --from-file=service-account.json
    $ kubectl -n prow create secret generic gcs-credentials --from-file=service-account.json
    KubeCon + CloudNativeCon Europe 2022 23

    View full-size slide

  46. Generate a HMAC token
    $ openssl rand -hex 20
    f9abc03ce9...cb3d92df96d2f4c0466a2cf
    KubeCon + CloudNativeCon Europe 2022 24

    View full-size slide

  47. Fetch the starter yaml, make necessary
    changes and then specify to tackle
    https://git.k8s.io/test-infra/config/prow/cluster/starter/starter-
    gcs.yaml
    KubeCon + CloudNativeCon Europe 2022 25

    View full-size slide

  48. Point your domain to the Ingress
    IP
    KubeCon + CloudNativeCon Europe 2022 26

    View full-size slide

  49. Update the HMAC Token in your
    GitHub App
    KubeCon + CloudNativeCon Europe 2022 27

    View full-size slide

  50. Install the app on a repo
    KubeCon + CloudNativeCon Europe 2022 28

    View full-size slide

  51. Enjoy!
    KubeCon + CloudNativeCon Europe 2022 29

    View full-size slide

  52. Benefits of using Prow
    KubeCon + CloudNativeCon Europe 2022 30

    View full-size slide

  53. Benefits of using Prow
    • Enables Effective Collaboration
    KubeCon + CloudNativeCon Europe 2022 30

    View full-size slide

  54. Benefits of using Prow
    • Enables Effective Collaboration
    • Accessible!
    KubeCon + CloudNativeCon Europe 2022 30

    View full-size slide

  55. Benefits of using Prow
    • Enables Effective Collaboration
    • Accessible!
    • New Contributor Friendly
    KubeCon + CloudNativeCon Europe 2022 30

    View full-size slide

  56. Contribute
    Come join us in Kubernetes SIG Testing to maintain Prow.
    • Mailing List: https://groups.google.com/forum/#!forum/
    kubernetes-sig-testing
    • Slack: #sig-testing on Kubernetes Slack
    KubeCon + CloudNativeCon Europe 2022 31

    View full-size slide

  57. Links
    • Deploying Prow
    • Setting up GitHub App
    KubeCon + CloudNativeCon Europe 2022 32

    View full-size slide