Prow! Leveraging Developer-Centric CI for Your OSS Project!

Prow is a CI system maintained by Kubernetes SIG Testing to test Kubernetes on Kubernetes. Prow is designed as a pluggable system of components and it can be used as a generic CI system. The robust architecture of Prow can lead to challenges in deploying it. In the talk, we will navigate the challenges faced when deploying and using Prow, including setting up the Prow control plane components, configuring access for GitHub repos, and enabling Prow plugins. Prow is used by large projects in the CNCF landscape like Kubernetes, Knative, cert-manager, Falco, to name a few. Even though a lot of these projects have deployed Prow successfully, it is a challenge to set up Prow. The talk will highlight the common pitfalls and gotchas that one will run into when deploying Prow. The talk would cover * A roundup of Prow Architecture * Cloud resources required for Prow and setting them up * Capabilities of Prow like running tests, using GitHub comments for interaction, auto merging pull requests.

Transcript

1. Prow! Leveraging Developer
   Centric CI for your OSS project!
   Arsh Sharma & Nabarun Pal
   KubeCon + CloudNativeCon Europe 2022 1
   

   View Slide

2. $ whoarewe
   KubeCon + CloudNativeCon Europe 2022 2
   

   View Slide

3. Arsh Sharma
   • Developer Experience Engineer at Okteto
   • Kubernetes 1.25 CI Signal Lead
   • New Contributor Ambassador for Kubernetes SIG Docs
   KubeCon + CloudNativeCon Europe 2022 3
   

   View Slide

4. Nabarun Pal
   • Senior Engineer at VMware
   • Kubernetes Release Engineering
   • Kubernetes Code of Conduct Committee
   KubeCon + CloudNativeCon Europe 2022 4
   

   View Slide

5. What is Prow?
   KubeCon + CloudNativeCon Europe 2022 5
   

   View Slide

6. What is Prow?
   • Kubernetes based CI/CD system
   KubeCon + CloudNativeCon Europe 2022 5
   

   View Slide

7. What is Prow?
   • Kubernetes based CI/CD system
   • GitHub automation through policies
   KubeCon + CloudNativeCon Europe 2022 5
   

   View Slide

8. What is Prow?
   • Kubernetes based CI/CD system
   • GitHub automation through policies
   • Configuration as code
   KubeCon + CloudNativeCon Europe 2022 5
   

   View Slide

9. What is Prow?
   • Kubernetes based CI/CD system
   • GitHub automation through policies
   • Configuration as code
   • Highly pluggable
   KubeCon + CloudNativeCon Europe 2022 5
   

   View Slide

10. What is Prow?
    • Kubernetes based CI/CD system
    • GitHub automation through policies
    • Configuration as code
    • Highly pluggable
    • ChatOps via /honk style commands
    KubeCon + CloudNativeCon Europe 2022 5
    

    View Slide

11. ChatOps
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

12. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

13. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

14. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

15. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    • Provide approval for PRs
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

16. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    • Provide approval for PRs
    • Run tests for a PR
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

17. ChatOps
    • /yolo style commands which allow interacting with GitHub
    issues and PRs
    • Assign issues to contributors or PRs to reviewers
    • Label issues and PRs
    • Provide approval for PRs
    • Run tests for a PR
    • A lot more stuff!
    KubeCon + CloudNativeCon Europe 2022 6
    

    View Slide

18. Architecture
    KubeCon + CloudNativeCon Europe 2022 7
    

    View Slide

19. Tide
    KubeCon + CloudNativeCon Europe 2022 8
    

    View Slide

20. Tide
    • Manages pool of GitHub PRs
    KubeCon + CloudNativeCon Europe 2022 8
    

    View Slide

21. Tide
    • Manages pool of GitHub PRs
    • Automatically retests PRs when they meet the criteria
    KubeCon + CloudNativeCon Europe 2022 8
    

    View Slide

22. Tide
    • Manages pool of GitHub PRs
    • Automatically retests PRs when they meet the criteria
    • Merges PRs without user intervention when tests pass
    KubeCon + CloudNativeCon Europe 2022 8
    

    View Slide

23. Testgrid
    KubeCon + CloudNativeCon Europe 2022 9
    

    View Slide

24. Testgrid
    • Interactive dashboard to view test results
    KubeCon + CloudNativeCon Europe 2022 9
    

    View Slide

25. Testgrid
    • Interactive dashboard to view test results
    • And they are shown in a grid!
    KubeCon + CloudNativeCon Europe 2022 9
    

    View Slide

26. KubeCon + CloudNativeCon Europe 2022 10
    

    View Slide

27. Configuration
    KubeCon + CloudNativeCon Europe 2022 11
    

    View Slide

28. Inrepo
    KubeCon + CloudNativeCon Europe 2022 12
    

    View Slide

29. Inrepo
    • Define jobs in the same repo where the source code is
    KubeCon + CloudNativeCon Europe 2022 12
    

    View Slide

30. Inrepo
    • Define jobs in the same repo where the source code is
    • In a .prow directory or .prow.yaml file at the root of the
    repo
    KubeCon + CloudNativeCon Europe 2022 12
    

    View Slide

31. Inrepo
    • Define jobs in the same repo where the source code is
    • In a .prow directory or .prow.yaml file at the root of the
    repo
    • Jobs are NOT defined centrally in this approach
    KubeCon + CloudNativeCon Europe 2022 12
    

    View Slide

32. Centralizing configuration
    KubeCon + CloudNativeCon Europe 2022 13
    

    View Slide

33. Centralizing configuration
    • Have a separate repo for all your prow jobs across different
    repos
    KubeCon + CloudNativeCon Europe 2022 13
    

    View Slide

34. Centralizing configuration
    • Have a separate repo for all your prow jobs across different
    repos
    • Jobs are defined centrally in this approach
    KubeCon + CloudNativeCon Europe 2022 13
    

    View Slide

35. Centralizing configuration
    • Have a separate repo for all your prow jobs across different
    repos
    • Jobs are defined centrally in this approach
    • Let's look at kubernetes/test-infra!
    KubeCon + CloudNativeCon Europe 2022 13
    

    View Slide

36. kubernetes/test-infra
    KubeCon + CloudNativeCon Europe 2022 14
    

    View Slide

37. Deploying your own
    instance of Prow
    KubeCon + CloudNativeCon Europe 2022 15
    

    View Slide

38. Creating the GitHub app
    KubeCon + CloudNativeCon Europe 2022 16
    

    View Slide

39. Add hook URL
    KubeCon + CloudNativeCon Europe 2022 17
    

    View Slide

40. Give Repository and Org Permissions
    KubeCon + CloudNativeCon Europe 2022 18
    

    View Slide

41. Subscribe to all events
    KubeCon + CloudNativeCon Europe 2022 19
    

    View Slide

42. Note App ID and Private Key
    KubeCon + CloudNativeCon Europe 2022 20
    

    View Slide

43. Run tackle
    Set up the GCP credentials before running tackle
    $ tackle -skip-github
    ...
    Let it run and pause where it asks you to specify location for
    starter.yaml.
    KubeCon + CloudNativeCon Europe 2022 21
    

    View Slide

44. Apply the Prow CRDs
    $ kubectl apply --server-side=true -f https://raw.githubusercontent.com/kubernetes/test-infra/master/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml
    customresourcedefinition.apiextensions.k8s.io/prowjobs.prow.k8s.io serverside-applied
    KubeCon + CloudNativeCon Europe 2022 22
    

    View Slide

45. Create a GCS bucket
    $ gcloud iam service-accounts create prow-gcs-publisher
    $ identifier="$(gcloud iam service-accounts list --filter 'name:prow-gcs-publisher' --format 'value(email)')"
    $ gsutil mb gs://master-shifus-knowledge/
    $ gsutil iam ch allUsers:objectViewer gs://master-shifus-knowledge
    $ gsutil iam ch "serviceAccount:${identifier}:objectAdmin" gs://master-shifus-knowledge
    $ gcloud iam service-accounts keys create --iam-account "${identifier}" service-account.json
    $ kubectl -n test-pods create secret generic gcs-credentials --from-file=service-account.json
    $ kubectl -n prow create secret generic gcs-credentials --from-file=service-account.json
    KubeCon + CloudNativeCon Europe 2022 23
    

    View Slide

46. Generate a HMAC token
    $ openssl rand -hex 20
    f9abc03ce9...cb3d92df96d2f4c0466a2cf
    KubeCon + CloudNativeCon Europe 2022 24
    

    View Slide

47. Fetch the starter yaml, make necessary
    changes and then specify to tackle
    https://git.k8s.io/test-infra/config/prow/cluster/starter/starter-
    gcs.yaml
    KubeCon + CloudNativeCon Europe 2022 25
    

    View Slide

48. Point your domain to the Ingress
    IP
    KubeCon + CloudNativeCon Europe 2022 26
    

    View Slide

49. Update the HMAC Token in your
    GitHub App
    KubeCon + CloudNativeCon Europe 2022 27
    

    View Slide

50. Install the app on a repo
    KubeCon + CloudNativeCon Europe 2022 28
    

    View Slide

51. Enjoy!
    KubeCon + CloudNativeCon Europe 2022 29
    

    View Slide

52. Benefits of using Prow
    KubeCon + CloudNativeCon Europe 2022 30
    

    View Slide

53. Benefits of using Prow
    • Enables Effective Collaboration
    KubeCon + CloudNativeCon Europe 2022 30
    

    View Slide

54. Benefits of using Prow
    • Enables Effective Collaboration
    • Accessible!
    KubeCon + CloudNativeCon Europe 2022 30
    

    View Slide

55. Benefits of using Prow
    • Enables Effective Collaboration
    • Accessible!
    • New Contributor Friendly
    KubeCon + CloudNativeCon Europe 2022 30
    

    View Slide

56. Contribute
    Come join us in Kubernetes SIG Testing to maintain Prow.
    • Mailing List: https://groups.google.com/forum/#!forum/
    kubernetes-sig-testing
    • Slack: #sig-testing on Kubernetes Slack
    KubeCon + CloudNativeCon Europe 2022 31
    

    View Slide

57. Links
    • Deploying Prow
    • Setting up GitHub App
    KubeCon + CloudNativeCon Europe 2022 32
    

    View Slide

58. Thank You!
    

    View Slide