OpenStack Summit Requiremens project update

OpenStack Summit requiremens project update

Transcript

1. OpenStack Requirements What we are doing, what to expect and what’s next Swapnil Kulkarni @coolsvap Davanum Srinivas @dims
2. Agenda • What is requirements project ◦ Why global requirements ◦ Lifecycle of requirements • Global Requirements and Upper Constraints • Accomplishments • Contribution Summary • Periodic Updates • Current Challenges • Whats Next? • How you can contribute • Resources
3. What is requirements project? Requirements project help maintain a central list of all the requirements (global-requirements.txt) that are allowed in OpenStack projects.
4. Dependencies in a typical project requirements.txt test-requirements.txt setup.cfg
5. Global Requirements and Upper Constraints
6. Why global requirements? Different projects may have different version ranges for same python library Order of project installs may break randomly Multiple installs of same library say during a single devstack run What exactly are we testing in our CI? Long term maintenance issues Is the library maintained? Is it under a compatible license? Are there existing libraries in use that already have similar functionality? Does this already exist in popular distros? So we need a single place to maintain version ranges and blessed versions
7. Lifecycle of Requirements New requirements are added to g-r by teams Component version ranges are updated in g-r as required New component is released on pypi triggers updates to u-c Updates to u-c affects all CI jobs for projects that are subscribed to g-r Bot proposes updates to all subscribed projects Requirements team considers pruning components from g-r/u-c Works with individual projects to remove dependencies Cleans up g-r and u-c, when all projects have removed those components
8. openstack-infra/project-config/zuul/layout.yaml requirements change for oslo-utils merged in requirements Subsequent change proposed by bot in nova bot to update requirements versions Individual project team reviewers then take decision on merging this requirement.
9. G-R and U-C maintenance Bot picks up new releases from pypi and updates u-c Bot picks up new indirect dependencies (of things in g-r) and updates u-c Humans may propose the following (all of which will need updates to u-c): New components New version ranges of components Exclusion of specific versions Gate/Check jobs to validate and merge Bot and Human proposed updates Another Bot proposes checks each project and proposes updates to requirements.txt, test-requirements.txt and setup.cfg
10. openstack-infra pypi mirror Periodic job to check upper constraints requirements project team core reviewers then take decision on merging this requirement. Cross-project jobs to verify upper-constraint changes work
11. How do i subscribe to global requirements process? Compare with global-requirements.txt Submit reviews for missing requirements that you need for your project Sync your requirements.txt with the version ranges in global-requirements.txt Add check-requirements job for the repository in project-config Add an entry in projects.txt in requirements repository Watch for reviews from OpenStack CI Bot
12. Adding/upgrading requirements - Why adhere to the requirements questionnaire for adding new requirements? - Upgrading requirements versions - upper-constraint changes
13. Review Guidelines - Early feedback of broken/unstable upstream requirements - Known minimum versions - Better commit messages - What are blacklist requirements? Why we need them? When to use them?
14. Contribution Matrix (Newton) Commits #570 Reviews #2600
15. Accomplishments (in Newton) - Creation of formal team and Tony Breeds (tonyb) elected as PTL - Formation of active core-team for requirements - Improved cross-project integration tests with projects using upper constraints - Improved reviews and stakeholders to actively track down issues arising with new requirements - Better co-ordination with release engineering team for releases of openstack projects and related libraries. - Removed most unused requirements from the list.
16. Pitfalls Some projects start using a “feature” just added to u-c without updating the lower bound of g-r. We don’t deal well with forks (pycrypto vs pycryptodome), Difficult to switch projects from one to another We do not test lower bounds of version ranges (u-c by definition is latest and greatest versions usable) Reverts trigger a lot of projects to be updated. How do we get a larger set of jobs (to prevent bad versions getting in). Since the CI jobs test a small subset of tests.
17. Current Challenges - Requirements sanitation - Requirements with duplicate functionality - Requirements without active maintainers and finding replacements (if possible) - Advocate project teams to use upper-constraints - Optimize propose changes
18. Looking ahead (Ocata priorities) - Introduce lower bounds and testing in projects - Python 3 compatibility and testing - https://wiki.openstack.org/wiki/Python3 - Better communication and impact analysis when merging changes related to complex binaries.
19. Ways to contribute - Manage global requirements for your projects in requirements repo - Reviews - Project integrations for u-c
20. Resources GitHub: https://github.com/openstack/requirements Developer Docs: http://docs.openstack.org/developer/requirements/ Gerrit Reviews: https://review.openstack.org/#/q/project:openstack/requirements To-dos: https://etherpad.openstack.org/p/requirements-tasks Weekly IRC meeting : https://wiki.openstack.org/wiki/Meetings/Requirements