7.0 KiB
Requirements management
Our current requirements management policy is causing significant gate downtime, developer issues and packaging headaches. We can do better.
Problem description
There are a number of interacting things we do today which are causing issues.
We run our tests with unpinned dependencies. This means that any patch being tested actually tests two things: the change in the patch, and all new releases of dependencies (within the ranges relevant to the branch in question). With nearly 300 (transitive) libraries in use, an incompatibility rate of only one per library per year can break us daily. We are suffering regular firedrills, and we've burnt out many gate fixers so far.
We require that projects have their install_requires exactly match our top level package specifiers that we use in gate jobs. This leads to releases of stable branches that have much narrower dependency specifiers than may work in practice. This becomes a problem when one package of a set needs to upgrade a given dependency post-release to fix a bug: the new package can be outside the set of versions dictated by the union of specifiers across all our packages that use it, which causes a cascade where new releases are required across the entire set to permit it to be used.
We override project local install_requires during testing, which means that our co-installability check is often returning a false positive: our actual install_requires may be incompatible but the gate won't report on this.
Additionally we have some constraints on solutions:
- Provide “some” expression to downstream packagers/users
- Configure test jobs (unit, integration, & functional) (devstack & non-devstack)
- Encourage convergence/maintain co-installability
- Not be fiction
- pip install python-novaclient works
- Stop breaking all the time (esp. stable)
- Don’t make library release management painful
Finally there are plenty of things that could be done that aren't addressed in this specification: it's the minimal self consistent set of improvements to address the ongoing firedrills we currently suffer.
Proposed change
tl;dr: Use exact pins for testing. Use open ended specifiers in project install_requires.
Globally we need to maintain global-requirements as we do today. This remains our policy control for which libraries are acceptable in OpenStack projects. As we start using extras, we'll need to track extras[1] in global-requirements as well. We want to preserve an axiom: that projects have wider-or-equal requirements than the coordinated release which has wider-or-equal requirements to the test pinned list.
We'll add a new pip freeze file to openstack/requirements, called upper-constraints.txt. This will contain a pinned list of the entire set of transitive requirements for that branch of OpenStack (defined by the projects within the projects.txt file in openstack/requirements). All CI jobs will use this to constrain the versions of Python projects that can be tested with. Changes to that file will be tested by running a subset of the same jobs that would consume it, as well as a policy checker that checks it is compatible with global-requirements.txt. Changes to either global-requirements.txt or upper-constraints.txt will have to be compatible with each other.
We'll tighten up the policy checks on projects to require that there be no dependencies outside of those listed in global-requirements. This is needed to allow centralised calculations about potential upgrades and co-installability calculations. We'll change the check from 'identical line to global-requires' to be 'compatible with both global-requires and upper-constraints'. The existing requirements sync job will continue to propose converged requirements for projects that have opted into converged dependency management.
We'll create a periodic job that takes global-requirements, expands it out transitively, and then proposes a merge to global-requirements bringing in any new releases (and any new or removed transitive-only dependencies that that implies) as a patch to upper-constraints.
Releases are a multi week process in OpenStack as different servers fork and setup their branches one at a time. During that period we'll gate any requirements changes on both master and any branched projects, branching openstack/requirements last when we're finally ready to decouple the release from master. This is aimed at the changes involved in using new releases of oslo etc.
Lastly, the pip resolver work will increase the accuracy of our constraints pinning, but this spec is not dependent on that.
Alternatives
The null option will continue to burn people out.
No other options have been proposed.
Implementation
Assignee(s)
- Primary assignee:
-
lifeless
Work Items
- Open up install-requires in master and stable/kilo. [lifeless]
- Generate pip glue to honour upper-constraints.txt. There are a few different ways this might be cut, lifeless will figure a working low hanging fruit implementation with upstream and/or infra. [lifeless]
- Create an initial upper-constraints.txt for master and stable/kilo. [lifeless]
- Change g-r self test to ensure consistency between g-r and upper-constraints.txt. [lifeless]
- Create jobs to prevent project local install_requires being incompatible with global-requirements. [fungi/lifeless]
- Teach devstack how to honour a constraints file. [lifeless]
- Teach unittests / tox how to honour a constraints file. [lifeless]
- Generate zuul glue to mangle constraints files when a project from within the constraints is part of the queue being assessed. [fungi/lifeless]
- Create jobs that use constraints files for everything. experimental for local project contexts, non-voting when triggered from g-r. [fungi]
- Create script to determine transitive dependencies of global-requirements and propose upgrades to upper-constraints.txt. [lifeless]
- Turn that script into a periodic job. [fungi]
- Debug and fix as needed until we get reasonably reliable passes on the non-voting g-r jobs. [lifeless]
- Flip the constraints using jobs to voting on g-r, and non-voting everywhere else. [fungi]
- Switch over to the constraints using jobs everywhere else in a more controlled fashion. [fungi]
- Update release documentation to branch openstack/requirements last, and setup the job that will validate requirements changes against the projects that have branched already, as well as master itself. [fungi]
Dependencies
- None.
History
| Release Name | Description |
|---|---|
| Liberty | Introduced |
References
Note
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode