oSC24 - Pagure CI integration with AWS CodePipeline

Pagure is a lightweight git centered forge based on Python with a long usage history in the Fedora project. It's well known in the openSUSE project too with an instance on code.opensuse.org. I demonstrate in this talk the current state of the Pagure Plugin system. How I implemented AWS CodePipeline as additional CI type. And how this can serve as blueprint to optimize the Plugin system and add other CI types. I will share the current status of my work to run Pagure on Kubernetes. And some good-first-issues you can tackle if you want to contribute to Pagure.

Presented at openSUSE Conference 2024: https://events.opensuse.org/conferences/oSC24/program/proposals/4587

Transcript

1. Dominik Wombacher dominik@wombacher.cc wombelix on libera.chat wombelix @wombelix:matrix.org pagure CI with AWS CodePipeline Sr. Partner Solutions Architect, AWS @wombelix@fosstodon.org
2. That’s me Linus loves Geeko • Passionate Engineer • Open source contributor • bugfixes / features • packager • maintainer • Dog person & Dad
3. pagure? pagure!
4. pagure? pagure! • community git forge • lightweight • hackable • extensible • scaleable
5. pagure? pagure! • First commit: 2014-03-14 17:03:48 UTC • Since then: • 9441 commits • 220 Contributors • Latest release: 5.14.1 (2024-05-24) • Major release 6.0: Q4 2024
6. pagure? pagure! • powers: • code.openSUSE.org • src.fedoraproject.org • git.centos.org • pagure.io
7. pagure? pagure! • Tech Stack • Python • Flask • Celery • libgit2 / pygit2 • Redis => Valkey • PostgreSQL / MariaDB
8. pagure? pagure! • challenges • 2 years between 5.13.3 and 5.14.1 • low amount of active contributors • technical depts • user experience
9. pagure plugin system
10. The term “plugin” in pagure • Hook • class BaseHook • Plugin • Flask blueprint
11. pagure hooks • scope: projects • routes and ui • pre-defined structure • extends BaseHook • override methods
12. Flask blueprints • scope: pagure • Env vars / config variables • PAGURE_PLUGINS_CONFIG • PLUGINS • Example: pagure-dist-git
13. additional ci types
14. pagure CI hook • The plugin inside the plugin system • pagure/hooks/pagure_ci.py • Celery triggers the CI • pagure/lib/tasks_services.py • trigger_ci_build()
15. build a new CI type • Remember: celery triggers the CI • You have to implement: • pagure/api/ci/<ci_type>.py • trigger_build()
16. build a new CI type • What else? • pagure/api/ci/<ci_type>.py • @API.route() • Handling of incoming responses
17. ongoing improvements • Done: • Make pagure CI more generic • Ongoing: • Make pagure CI easier extendable
18. pagure CI future pagure CI Build all the integrations you want! AWS CodePipeline
19. AWS CodePipeline
20. AWS CodePipeline Image source: https://docs.aws.amazon.com
21. AWS CodePipeline ci type • Short Term credentials • EC2 Instance Profiles • IAM for Service Accounts on k8s
22. EC2 Instance IAM Role S3 Bucket Role EC2 Instance Container 1 Container 2 Container 3
23. IAM for Service Accounts
24. pagure on kubernetes
25. “classic” pagure deployment traditional server Users
26. pagure on k8s architecture forge.xlii.cc pagure server storage, db pagure worker
27. [WIP] pagure on k8s • Container • Python 3.9 venv • entrypoint scripts • Kubernetes • sample Deployment manifest
28. pagure on k8s – next steps • Container • publish to quay.io • latest == master branch • release tags • Kubernetes • Helm chart
29. contribute / good-first-issues
30. Get started • Install and use pagure J • Read the source and docs • Most important: Local testing • Start with small improvements or fixes contributing
31. Ask questions! • Join our Matrix room: #pagure:fedora.im • Subscribte to pagure-devel@lists.pagure.io contributing
32. good-first-issues / easyfix
33. Thank you! Feedback? Questions? https://pulse.aws/survey/ANUMYAGL