Resolve the following warnings issued in recent versions of SQLAlchemy:
SADeprecationWarning: Implicit coercion of SELECT and textual SELECT
constructs into FROM clauses is deprecated; please call .subquery() on
any Core select or ORM Query object in order to produce a subquery
object.
The resolution is simple: instead of using 'FromClause.alias', use
'Query.subquery'. A warning filter is included to prevent us
reintroducing this issue in the future. It can be dropped once we start
using SQLAlchemy 2.0 where this behavior will be removed.
This change requires bumping SQLAlchemy to 1.4.0, which is the first
version to introduce the selectable '.subquery()' method. We must also
bump oslo.db to 8.6.0, which is the first version to support SQLAlchemy
1.4.x. The alternative would be to introduce a switch based on
SQLAlchemy versions, but that's tech debt that'll have to be cleaned up.
Change-Id: I6300be962dc3ce391f70d7c7e8af7cb4d6ce9baf
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
To be compatible with recent setuptools 58.0.0 which removed the support
for use_2to3 we need to bump the minimum version we support for
decorator from 3.4.0 to 4.0.0.
This is needed to unblock the gate.
Change-Id: I456f8136f882a2c6621549b2b0f80e213e5430df
The latest os-traits release introduced 3 new standard traits
While bumping the os-traits requirements it turned out that the
lower-constraints become extra slow causing a timeout. This result is a
catch-22 situation. The lower-constraint job cannot be fixed alone as
the global requirement bump already happened for os-traits 2.6.0
requiring fixing the placement tests. But fixing the placement tests
alone is not possible as the lower-constraints job will time out.
So this patch squashes in Icd0a304fc989c9535db125cd483ecac92deb011d:
Pip is slow to resolve version constraints with many unrestricted
dependencies. The recent slowness seems to be due to amqp is being
unrestricted and pip starts with amqp 5.0.3 but the real minimum is
somewhere around amqp 2.5.0. This patch adds amqp 2.5.0 to
lower-constraints to speed up the pip version resolution.
Change-Id: I89b330d0647b0883eecaf40b988a1a4779bcb1dd
os-resource-classes 1.1.0 was recently released and upper
constraints were bumped. This brings 3 new standard resource classes so
the tests are adjusted.
blueprint: qos-minimum-guaranteed-packet-rate
Change-Id: I6d1c84ddc4094635bebe001ea046cd01f1c98bbb
Since 3.7.0, oslo policy started the DeprecationWarning[1] if
deprecated_reason and deprecated_since param are not passed
in DeprecatedRule or they are passed in RuleDefault object.
These warnings are logged for every test.
[1] https://github.com/openstack/oslo.policy/blob/3.7.0/oslo_policy/policy.py#L1538
Change-Id: I38c91afdfd773af96db9b2254281c383023481e4
Placement wants to always use the latest os-traits and
os-resource-classes libs. See
Idd4920885b10f0b21b2b67d9526f1048b06f7db5 .
The reason of the os-resource-classes major bump was dropping python 2
support:
$ git log 0.5.0..1.0.0 --oneline
3dd3506 (tag: 1.0.0) Switch to hacking 2.x
b99ab33 setup.cfg: Configure 'python-requires'
50b4160 Stop testing with py2
19b3411 Switch to Ussuri jobs
aaccfd9 Bump the openstackdocstheme extension to 1.20
fcd1be5 Merge "Build pdf docs"
432563f Merge "Update bug link in docs to point to storyboard"
718adcc Update the constraints url
65b74ff Build pdf docs
040e9c1 Update bug link in docs to point to storyboard
367698e Merge "Update api-ref link to canonical location"
d616e55 Update api-ref link to canonical location
8866073 Sync Sphinx requirement
4387894 Update api-ref location
Placement already dropped that so no extra measures needed.
Change-Id: I826ee41f7ef01c936d669024c6802091eb9c3bed
When running the functional tests locally:
tox -efunctional
the following error is raised from 6 tests in the
placement.tests.functional.db.test_migrations module:
RuntimeError: 'cryptography' package is required for sha256_password
or caching_sha2_password auth methods
This adds the 'cryptography' package to the list of required packages
for tests.
Story: 2008832
Task: 42317
Change-Id: I78fbdd723be089364b8ee61432eeb74ed6f24094
In I0438e39a3c61253f781761a5aae895993f897adb,
I333147dcd47c6d0b926338a5a0c545f5adc63961 , and
Ia4d1871b719037174506fb784b77572e63303762 six new
traits were added, and a new os-traits was released
triggering the canary test for updating constraints.
Change-Id: I4fbab60f04f85d70c703385d7b43457045e3661d
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: Ia7365cc3ae09e2ff916ab9f9ff0ba4fef0dc446b
This ensures we have a version available that contains the deprecated
markers for the Wallaby release.
This commit also updates the minimum version of oslo.context to 2.21.0.
The oslo.log library depends 2.20.0 and we'll need version 2.21.0 in a
subsequent patch.
I bumped additional packages to ensure the minimum versions for
lower-constraints work with the newer oslo dependencies.
Change-Id: I6d1e39886d177d8729bc2db9e0018b5c772f8b0e
There are few failure in l-c jobs due to
- The conflict is caused by:
The user requested pyflakes>=2.1.1
hacking 0.12.0 depends on pyflakes==0.8.1
- The conflict is caused by:
The user requested requests>=2.14.2
keystonemiddleware 4.18.0 depends on requests>=2.14.2
oslo-config 6.7.0 depends on requests>=2.18.0
Remove the flake8 and pyflake from lower constraints as they are
maintained in hacking side[1]. Maintaining in hacking aas well as
in project side can lead to version conflict. Also few more deps
from l-c as they are not used in placement directly.
Update requests version to 2.25 to fix the http connection failure with
2.18
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dff/766994/6/check/openstack-tox-lower-constraints/dff1943/testr_results.html
This commits also moves the code to latest hacking 4.0.0 with W504 in
ignore list and fix W503 and other checks.
[1] 4069b0c4f5/lower-constraints.txt (L9)
Change-Id: Id2761300060597c1744e76ea77740ed380d98943
As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).
Fixing:
- bug#1885825
Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)
Ubuntu Focal (20.04) has mysql 8.0 and with mysql 8.0 there
is no implicit user creation with GRANT. We need to
create the user first before using GRANT command.
- bug#1886298
Bump the lower constraints for required deps which added python3.8 support
in their later version.
- bug#1886296
pyflake issue and some pep8 fixes.
Story: #2007865
Task: #40209
Related-Bug: #1885825
Related-Bug: #1886298
[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal
Change-Id: Iac9652b0d36ef208075da506957cc87f22912c45
These translation sections are not needed anymore,
Babel can generate translation files without them.
Change-Id: I0dbcb9c497144a248aefff0355d03dd7cddc77c9
The upper constraint for os-traits was bumped to 2.4.0 in change
Idf9ae2ea576f4ed882489f03e4dbc94193243c98 and additional traits
are available.
Update the tests and requirements accordingly.
Change-Id: I2d67ea2025bab811b00d98759ac4b97ad0737377
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we don't need it
in lower-constraints.
These changes will help us to drop `mock` from openstack/requirements
Change-Id: Ib48e422635c2d8957d865f8af4ca730161c40e8c
os-traits has a new release 2.2.0 with an additional trait; update tests
and requirements accordingly.
Change-Id: I549a1ae99daa524ec881796f4442d781d4cf50bf
os-traits has a new release 2.1.0 with an additional trait; update tests
and requirements accordingly.
Change-Id: I7de7d2bc32b46a505a1837ff3af5861e2ccbd553
os-traits has a new release 2.0.0 with an additional trait; update tests
and requirements accordingly.
Change-Id: If9a26b4d85bd902457228bafc28abb75980b168d
This was explored as part of discussion with zzzeek on various
ways to make a statement with a large in_ be more performant.
Batching the queries by chunking rp_ids did not help, but an
expanding bindparam did: it makes the surrounding method about
50% faster [1].
bindparam expanding was added in sqlalchemy 1.2.0 so we bump
the requirements here to the latest 1.2.x bugfix release,
1.2.19.
[1] With the caveat that profiling and benchmarking are very
different things, and in this case it is profiling results
which are being analysed.
Change-Id: Ic4e0cdd87f8f2d76b921059ac4bf16a838913abf
os-resource-classes 0.5.0 was recently released and upper
constraints were bumped. We have decided to always pin placement
to the latest versions of os-resource-classes and os-traits.
This patch makes it so, again.
Change-Id: Idd4920885b10f0b21b2b67d9526f1048b06f7db5
os-traits 0.14.0 hit upper-constraints [1] so placement's lower bounds
and canary test need to be updated accordingly.
[1] https://review.opendev.org/663507
Change-Id: Ib6f08a681162101f91c0de4ce1d2eed35975ef96
Because os-traits and os-resource-classes are libraries that
provide enumerations, we usually want whatever the latest
version is, even in lower-constraints.txt. We haven't, however,
been good about keeping those up to date.
This updates both requriements.txt and lower-constraints.txt to
the latest versions of both os-traits and os-resource-clases.
No change is required is global upper-constraints. That already
has the latest.
Change-Id: I0bd0e8e071d6275c3e21556018d476c97e8533ae
Story: 2005651
Task: 30939
Turn ResourceProvider and ResourceProviderList into
classical Python objects.
There are two changes here which deserve particular
attention because they are more changing than the other
OVO removals:
* There were two deepcopy calls in the allocation request
handling. When using OVO, the __deepcopy__ handling
built into it prevents deep recursion. I changed them
to copy and things still work as expected. This will
be because using the nested objects by reference is
acceptable.
* The removal of OVO removes the detection of changed fields.
This was being used when creating and saving resource
providers (at the object level) to automagically detect
and prevent writing fields we don't want to.
This change removes that functionality. Instead if bad
data has made it as far as the create or save calls, we
simply don't write it. The HTTP layer continues to
maintain the guards it already had in place to prevent
badness. Tests which were testing the object layer
are removed.
The create_provider function in functional/db/test_base
allowed (trying to) create a provider with a root
but that functionality was only called from one place.
That place and the functionality in create_provider
is removed.
Other things to note:
* As with the other changes, where context is actually used
by the object it is required in the constructor. This
cascades some changes to tests.
* A test that checks to see if adding traits resets the
changes on a rp has been removed, because we don't
track that any more if we haven't got OVO.
* The last_modified handling in placement/util no longer
need NotImplemented handling, that was an artifact of
OVO.
oslo.versionedobjects is removed from requirements. This
doesn't have a huge impact on the size of the virtualenv:
114M -> 107M [1] but it does take us from 132 python
dependencies (via pip list) to 119, removing plenty of
stuff that was only being called in because stuff that
we don't use depended on it.
lower-constraints.txt is updated to reflect the removal
of dependencies that are no longer needed.
[1] Of note, 26M of that is babel. Do we need to translate
exceptions? See email disussion at
http://lists.openstack.org/pipermail/openstack-discuss/2019-February/thread.html#3002
Change-Id: Ie0a9351e0d7c762c9c96c45cbe50132a0fbd1486
Because of a conflict between how the oslo.upgradecheck library uses
oslo config and how we want to use it in placement, two different
ConfigOpts were needed to avoid an args already parsed error. The
new release of the oslo.upgradecheck library is change to allow
the two steps in main (registering the CLI opts and running the
command) to be called separately if desired. Doing so allows us
to use just one ConfigOpts.
Requirements and constraints are updated.
Change-Id: I792df18cb17da95659628bfe7f7a69897c6f37ab
The 0.2.0 release of os-resource-classes has happened, adding the
'PCPU' class. It's already updated in global upper-constraints,
so update it in requirements.txt and lower-constraints and change
the gabbi tests which count resource classes.
Change-Id: I4a189aca59485d65ad8a7c9bfbeca7ac995ed336
os-resource-classes is a python library in which the standardized
resource classes are maintained. It is done as a library so that
multiple services (e.g., placement and nova) can use the same stuff.
It is used and managed here in the same way the os-traits library is
used: At system start up we compare the contents of the
resource_classes table with the classes in the library and add any
that are missing. CUSTOM resource classes are added with a high id
(and always were, even before this change). Because we need to
insert standard resource classes with an id of zero, so we need to
protect against mysql thinking 0 on a primary key id is "generate
the next one". We don't need a similar thing in os-traits because
we don't care about the ids there. And we don't need to guard
against postgresql or sqlite at this point because they do not have
the same behavior.
The resource_class_cache of id to string and string to id mappings
continues to be maintained, but now it looks solely in the database.
As part of confirming that code, it was discovered that the reader
context manager was being entered twice, this has been fixed.
Locking around every access to the resource class cache is fairly
expensive (changes the perfload job from <2s to >5s). Prior to this
change we would only go to cache if the resource classes in the
query were not standards. Now we always look at the cache so rather
than locking around reads and writes we only lock around writes.
This should be okay, because as long as we do a get (intead of the
previous two separate accesses) on the cache's dict that operation
is safe and if it misses (because something else destroyed the
cache) the fall through is to refresh the cache, which still has the
lock.
While updating the database fixture to ensure that the resource
classes are synched properly, it was discovered that addCleanup was
being called twice with the same args. That has been fixed.
In objects/resource_provider.py the ResourceClass field is changed to a
StringField. The field definition was in rc_field and we simply don't
need it anymore. This is satisfactory because we don't do any validation
on the field internal to the objects (but do elsewhere).
Based on initial feedback, 'os_resource_classes' is imported as 'orc'
throughout to avoid unwieldy length.
Change-Id: Ib7e8081519c3b310cd526284db28c623c8410fbe
With oslo.config 6.7.0 configuration options may be found in environment
variables. Because of the small number of options that placement
requires to run it is straightforward to source all config from the
environment, useful in container-based environments.
However, prior to this change, placement-the-wsgi-app required that a
configuration file existed, placement.conf, either in /etc/placement or
in a custom directory.
This change makes it so that the placement.conf can be any of:
* in the custom directory
* in the default project locations (including /etc/placement)
* nowhere
The change maintains the behavior that if
[placement_database]/connection is not set the application will fail to
start.
Though this change is orthogonal to the oslo.config change, requirements
and lower-constraints are updated to ensure the desired behavior.
Closes-Bug: #1802925
Change-Id: Iefa8ad22dcb6a128293ea71ab77c377db56e8d70
While exploring removing unused packages from lower-constraints.txt it
became clear that the lower-constraints job was not working as expected:
Because our tox config has usedevelop=True 'setup.py develop' is called
to install the placement package after the install command is called.
This means that the lower-constraints are clobbered.
I had mistakenly assumed that turning off 'usedevelop', which causes
'setup.py install' would not make any difference, because it usually
installs dependencies too. It turns out however, that when using pbr
and within a git working dir, it does not. That took some time to
figure out. Oh well.
This change makes it so that we create the tox environment using
usedevelop=False and with our own install_command, to avoid upper
constraints conflicting with lower constraints.
This flagged up a few changes, the main one being that we did not have
a new enough version of keystonemiddleware in order to require use of
www_authenticate_uri. requirements.txt is updated for this as well.
And PasteDeploy needed to be updated to work with Python 3's notion
of namespace packages.
psycopg2 need a newer version to work with Postgresql 10.
oslotest needs to be raised to 3.4.0 because the tests in
cmd.test_manage use features to control what is capture by the Output
fixture from oslotest. Note that the lower-constraints job found
this problem and also demonstrates why we must run the lower-constraints
job without upper-constraints being involved. upper-constraints will
"win" and we don't want that. The point of the job is find packages
where lower-constraints are wrong, so it must "win".
The end result here is a lower-constraints.txt file that starts from
the lower-constraints.txt defined by nova, and then is adapted to
update the versions of packages that were not up to date, remove
those packages which are no longer present, and add some that are
now required.
Change-Id: Id66a28f7ace6fc2adf0e1201d9de5f901234d870
This adds the basic framework for the
placement-status upgrade check command which
is a community-wide goal for the Stein release.
A simple placeholder check is added which should
be replaced later when we have a real upgrade
check.
Change-Id: I9291386fe7fcbfc035c104ea9fdbe5eb875c4776
Story: 2003657
Task: 27518
With the migrations now gone, we can safely remove sqlalchemy-migrate
from requirements.txt and lower-constraints.txt.
Change-Id: Id8d03d9532bcf4c4f7ad1c1039fa6a42990f8d5d
This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.
Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
This changes the max_concurrent_live_migrations handling
to use a ThreadPoolExecutor so that we can control a bounded
pool of Futures in order to cancel queued live migrations
later in this series.
There is a slight functional difference in the unlimited
case since starting in python 3.5, ThreadPoolExecutor will
default to ncpu * 5 concurrently running threads. However,
max_concurrent_live_migrations defaults to 1 and assuming
compute hosts run with 32 physical CPUs on average, you'd
be looking at a maximum of 160 concurrently running live
migrations, which is probably way above what anyone would
consider sane.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Part of blueprint abort-live-migration-in-queued-status
Change-Id: Ia9ea1e164fb3b4a386405538eed58d94ad115172
This is the first change that implements basic virt.driver methods
to allow nova-compute process start successfully.
A set of subsequent changes will implement spawn, snapshot, destroy
and instance power actions.
Change-Id: Ica6117c2c64f7518b78b7fb02487622250638e88
blueprint: add-zvm-driver-rocky
Ghanshyam Mann