This is being reverted because it's overly strict and complaining
that upgrade-related work has not been done before it should have or
needs to have been done. This may be re-added later when we start
depending on these linkages.
Closes-Bug: #2039597
This reverts commit 27f384b7ac.
Change-Id: Ifa5b82ca3b83d0ba481aa7a062827bd8e838989a
The URLs had the wrong order of "/latest/nova" instead of the correct
one, leading to "404 not found" errors.
Closes-Bug: 2036530
Change-Id: I083381ad2649c06be9443f5ed6a55bddafab4df8
The 'force' parameter of os-brick's disconnect_volume() method allows
callers to ignore flushing errors and ensure that devices are being
removed from the host.
We should use force=True when we are going to delete an instance to
avoid leaving leftover devices connected to the compute host which
could then potentially be reused to map to volumes to an instance that
should not have access to those volumes.
We can use force=True even when disconnecting a volume that will not be
deleted on termination because os-brick will always attempt to flush
and disconnect gracefully before forcefully removing devices.
Closes-Bug: #2004555
Change-Id: I3629b84d3255a8fe9d8a7cea8c6131d7c40899e8
We have droped the system scope from Nova policy
and keeping the legacy admin behaviour same. This
commit adds the releasenotes and update the policy
configuration documentation accordingly.
Also, remove the upgrade check for policy which was
added for the system scope configuration protection.
Change-Id: I127cc4da689a82dbde07059de90c451eb09ea4cf
Use of this library has significant performance implications. While
we're probably not too badly affected, we don't actually need to use it
here. The 'parse_version' utility it exposes is intended to parse
PEP440-compliant version identifiers, not the simple microversions
placement uses, which the 'microversion_parse' library can competently
parse for us.
Change-Id: I9b7281caec6fa53600dea316492d052787cf799b
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The TooOldComputeService upgrade check currently produces a warning,
which may be missed if the upgrade process only checks the exit code of
the upgrade check command.
Since this can lead to Nova control services failing to start, make the
upgrade check a failure instead, so it results in a non-zero exit code.
Closes-Bug: #1956983
Change-Id: Ia3ce6a0b0b810667ac0a66502a43038fe43c5aed
After moving the nova APIs policy as per the new guidlines
where system scoped token will be only allowed to access
system level APIs and will not be allowed any operation
on project level APIs. With that we do not need below
base rules (who have hardcoded 'system_scope:all' check_str):
- system_admin_api
- system_reader_api
- system_admin_or_owner
- system_or_project_reader
At this stage (phase-1 target), we allow below roles as targeted
in phase-1 [1]
1. ADMIN(this is System Administrator with scope_type 'system'
when scope enabled otherwise legacy admin)
2. PROJECT_ADMIN
3. PROJECT_MEMBER
4. PROJECT_READER
& below one specific to nova
5. PROJECT_READER_OR_ADMIN (to allow system admin and project reader
to list flavor extra specs)
This complete the phase-1 of RBAC community-wide goal[2] for nova.
Add release notes too.
[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#how-operator
[2] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#yoga-timeline-7th-mar-2022
Partial implement blueprint policy-defaults-refresh-2
Change-Id: I075005d13ff6bfe048bbb21d80d71bf1602e4c02
Resolve the following RemovedIn20Warning warnings:
The Executable.scalar() method is considered legacy as of the 1.x
series of SQLAlchemy and will be removed in 2.0. Scalar execution in
SQLAlchemy 2.0 is performed by the Connection.scalar() method of
Connection, or in the ORM by the Session.scalar() method of Session.
The Executable.execute() method is considered legacy as of the 1.x
series of SQLAlchemy and will be removed in 2.0. All statement
execution in SQLAlchemy 2.0 is performed by the Connection.execute()
method of Connection, or in the ORM by the Session.execute() method of
Session.
Note that while we're resolving these issues, we also head off other
issues that are currently ignored but will ultimately need to be
resolved such as switching to the modern calling style of 'select()'
Change-Id: Idebcba02a6704df21e5520fdbf60296b8187e79c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
To implement the usage of same_subtree query parameter in the
allocation candidate request first the minimum requires placement
microversion needs to be bumped from 1.35 to 1.36. This patch makes such
bump and update the related nova upgrade check. Later patches will
modify the query generation to include the same_subtree param to the
request.
Change-Id: I5bfec9b9ec49e60c454d71f6fc645038504ef9ef
blueprint: qos-minimum-guaranteed-packet-rate
Introduce a new 'nova.db.api.api' module to hold API database-specific
helpers, plus a generic 'nova.db.utils' module to hold code suitable for
both main and API databases. This highlights a level of complexity
around connection management that is present for the main database but
not for the API database. This is because we need to handle the
complexity of cells for the former but not the latter.
Change-Id: Ia5304c552ce552ae3c5223a2bfb3a9cd543ec57c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.
The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.
Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change If90d9295b231166a28c2cc350d324691821a696b kicked off this effort
but only change the migrations. This change completes the job.
Change-Id: Ic0f2c326ebce8d7c89b0debf5225cbe471daca03
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This change introduces a new nova-status check to ensure a machine type
has been recorded for each instance within an environment.
nova-status will fail with a warning when instances are found, directing
the operator to use the previously added nova-manage list_unset and
update commands to set a machine type for these instances. The logic for
this check comes entirely from the list_unset command.
It is noted in the warning output that this can be ignored if no libvirt
or HyperV based computes are present in the environment as
hw_machine_type is only used by these two virt drivers at present.
blueprint: libvirt-default-machine-type
Change-Id: Ic3ae48c57e61c4e45883fbae1328a448be025953
This command was helpful to assist users FFUing past the Pike release,
however, it's no longer helpful and should be removed now. Do just that.
Change-Id: Ib42f65dbcf61ead571e9107e7ffbba2b29f48d64
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Replace six.text_type with str.
A subsequent patch will replace other six.text_type.
Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
Report a warning during upgrade checks if there are computes older than
the previous major nova release in the system.
So if code is upgraded to Wallaby and the upgrade check was run before
the restart of the services with W code then the check warns for Ussuri
computes in the system.
Change-Id: I873b0c1e6e695ae88241bbf75ac9f80ecc6f5664
Default value of 'CONF.oslo_policy.policy_file' config option
has been changed from 'policy.json' to 'policy.yaml'. If new default
file 'policy.yaml' does not exist but old default 'policy.json' exist
then fallback to use old default file.
An upgrade checks is added to check the policy_file format and
fail upgrade checks if it is JSON formatted.
Added a warning in policy doc about JSON formatted file is deprecated,
also removed all the reference to policy.json file in doc as well as
in tests.
Related Blueprint: policy-json-to-yaml
Closes-Bug: #1875418
Change-Id: Ic4d3b998bb9701cb1e3ef12d9bb6f4d91cc19c18
These particular imports are no longer needed in a Python 3-only world.
Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
There are cases where policy file is re-generated freshly
and end up having the new defaults only but expectation is that
old deprecated rule keep working.
If a rule is present in policy file then, that has priority over
its defaults so either rules should not be present in policy file
or users need to update their token to match the overridden rule
permission.
This issue was always present when any policy defaults were changed
with old defaults being supported as deprecated. This is we have
changed all the policy for new defaults so it came up as broken case.
Adding nova-status upgrade check also to detect such policy file.
Related-Bug: #1875418
Change-Id: Id9cd65877e53577bff22e408ca07bbeec4407f6e
Placement microversion 1.35 gives us the root_required queryparam to GET
/allocation_candidates, allowing us to filter out candidates where the
*root* provider has/lacks certain traits, independent of traits
specified in any of the individual request groups.
Use it.
And add affordance for specifying such traits to the RequestSpec.
Which allows us to fix up the couple of request filters that were
hacking traits into the RequestSpec.flavor.
Change-Id: I44f02044ce178e84c23d178e5a23a3aa1208e502
This commit cuts us over to using placement microversion 1.34 for GET
/allocation_candidates, thereby supporting string request group suffixes
(added in 1.33) when specified in flavor extra_specs.
The mappings (added in 1.34) are not used in code yet, but a future
patch will tie the group suffixes to the RequestGroup.requester_id so
that it can be correlated after GET /a_c. This will allow us to get rid
of map_requested_resources_to_providers, which was a hack to bridge the
gap until we had mappings from placement.
Change-Id: I52499ff6639c1a5815a8557b22dd33106dcc386b
This patch modifies Nova objects that will allow Destination object to
store list of forbidden aggregates that placement should ignore in
'GET /allocation_candidates' API at microversion 1.32, including the
code to generate the placement querystring syntax from them.
Change-Id: Ic2bcee40b41c97170a8603b27b935113f0633de7
Implements: blueprint placement-req-filter-forbidden-aggregates
The "Request Spec Migration" upgrade status check was added
in Rocky [1] and the related online data migration was removed
in Stein [2]. Now that we're in Train we should be able to
remove the upgrade status check.
The verify install docs are updated to remove the now-removed
check along with "Console Auths" which were removed in Train [3].
[1] I1fb63765f0b0e8f35d6a66dccf9d12cc20e9c661
[2] Ib0de49b3c0d6b3d807c4eb321976848773c1a9e8
[3] I5c7e54259857d9959f5a2dfb99102602a0cf9bb7
Change-Id: I6dfa0b226ab0b99864fc27209ebb7b73e3f73512
We're going to be removing the configuration option so the advice from
this check will no longer make sense.
Part of blueprint remove-consoleauth
Change-Id: I5c7e54259857d9959f5a2dfb99102602a0cf9bb7
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The compute API has required cinder API >= 3.44 since Queens [1] for
working with the volume attachments API as part of the wider
volume multi-attach support.
In order to start removing the compatibility code in the compute API
this change adds an upgrade check for the minimum required cinder API
version (3.44).
[1] Ifc01dbf98545104c998ab96f65ff8623a6db0f28
Change-Id: Ic9d1fb364e06e08250c7c5d7d4bdb956cb60e678
This patch adds the translation of `RequestGroup.in_tree` to the
actual placement query and bumps microversion to enable it.
The release note for this change is added.
Change-Id: I8ec95d576417c32a57aa0298789dac6afb0cca02
Blueprint: use-placement-in-tree
Related-Bug: #1777591
There were a lot of workarounds here to ensure we didn't switch to the
new model (vs. the old 'nova-consoleauth' service) if users were on
cells v1. These can go now, along with the old 'nova-consoleauth'
service (though that's a later, separate change).
Part of blueprint remove-cells-v1
Change-Id: I1b8f411b050d34e4e77e9a4f1e613135eb5f74b7
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The nova-status upgrade check code is intentionally written
to not use versioned objects and just work with the data model
directly.
However, CellMapping database_connection template support was
added in Rocky and the template URL is only formatted when using
the object on read from the database, which means if you are using
a template for the database_connection nova-status will break since
the get_engine() code is getting a template string rather than a
properly formatted URL.
This change fixes the bug by using the CellMapping object code to
pull the mappings from the database which will format the database
connection URL. Note that we cannot simply update the sqlalchemy
RowProxy objects we get normally since those are read-only, and
because of how the mappings are used as objects with attribute
access rather than as a dict we cannot just convert the RowProxy
to a dict - we would have to put it in some kind of object for
attribute access and if we are going to do that we might as well
just use the CellMapping objects we have so that's what we do in
this change.
Change-Id: I5ce175517f6feb6e82ba507078a565b71427a4b0
Closes-Bug: #1818047
With change I11083aa3c78bd8b6201558561457f3d65007a177
the code for the API Service Version upgrade check no
longer exists and therefore the upgrade check itself
is meaningless now.
Change-Id: I68b13002bc745c2c9ca7209b806f08c30272550e
With placement being extracted from nova, the
"Resource Providers" nova-status upgrade check no
longer works as intended since the placement data
will no longer be in the nova_api database. As a
result the check can fail on otherwise properly
deployed setups with placement extracted.
This check was originally intended to ease the upgrade
to Ocata when placement was required for nova to work,
as can be seen from the newton/ocata/pike references
in the code.
Note that one could argue the check itself as a concept
is still useful for fresh installs to make sure everything
is deployed correctly and nova-compute is properly
reporting into placement, but for it to be maintained
we would have to change it to no longer rely on the
nova_api database and instead use the placement REST API,
which while possible, might not be worth the effort or
maintenance cost.
For simplicity and expediency, the check is simply removed
in this change.
Related mailing list discussion can be found here [1].
[1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000454.html
Change-Id: I630a518d449a64160c81410245a22ed3d985fb01
Closes-Bug: #1808956
The common upgrade check code has been moved to oslo.upgradecheck.
This change switches the Nova upgrade checks to use the common code
and removes the tests for functionality that is now the responsibility
of the library.
Change-Id: I0dc2044286dbe78314c650a92c4654f7f50642d2
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
This will check if a deployment is currently using consoles and warns
the operator to set [workarounds]enable_consoleauth = True on their
console proxy host if they are performing a rolling upgrade which is
not yet complete.
Partial-Bug: #1798188
Change-Id: Idd6079ce4038d6f19966e98bcc61422b61b3636b
When counting rows in the compute_nodes table to compare against
resource records, those which have been marked as deleted should
not be counted, otherwise the result is artificially high.
Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
Closes-Bug: #1757207
The way in which this check counted compute nodes was
broken because of an incorrect for/else condition. If
the check is run with a nova.conf like we have in
devstack, where the API database is configured but
the [database]/connection is pointing at cell0, where
there are no compute nodes, the check passes saying
there are no compute nodes even if the are compute
nodes found in the cell databases (in the for loop).
This is because the else executes because the for loop
doesn't break, and then _count_compute_nodes returns 0
for cell0 and overwrites the num_computes variable.
This fixes the issue by checking if we have cell mappings
before running the loop, else we hit the else block as
was originally intended.
Change-Id: I1a706d028a9ca894348a19b7b3df1ea673e4ec90
Partial-Bug: #1790721
Add a thin wrapper to invoke the POST /reshaper placement API with
appropriate error checking. This bumps the placement minimum to the
reshaper microversion, 1.30.
Change-Id: Idf8997d5efdfdfca6967899a0882ffb9ecf96915
blueprint: reshape-provider-tree
If we're updating existing allocations for an instance due
to the project_id/user_id not matching the instance, we should
use the consumer_generation parameter, new in placement 1.28,
to ensure we don't overwrite the allocations while another
process is updating them.
As a result, the include_project_user kwarg to method
get_allocations_for_consumer is removed since nothing else
is using it now, and the minimum required version of placement
checked by nova-status is updated to 1.28.
Change-Id: I4d5f26061594fa9863c1110e6152069e44168cc3
We can't easily add a blocker db sync migration to make
sure the migrate_instances_add_request_spec online data
migration has been run since we have to iterate both cells
(for instances) and the API DB (for request specs) and that's
not something we should do during a db sync call.
But we want to eventually drop the online data migration and
the accompanying compat code found in the api and conductor
services.
This adds a nova-status upgrade check for missing request specs
and fails if any existing non-deleted instances are found which
don't have a matching request spec.
Related to blueprint request-spec-use-by-compute
Change-Id: I1fb63765f0b0e8f35d6a66dccf9d12cc20e9c661
With the exception of a single spot in the code, every request we were
making through keystoneauth1 Adapter/Session was using raise_exc=False -
i.e. don't raise exceptions on >=400; we process the Response ourselves.
As of keystoneauth1 3.9.0 it is possible to pass a raise_exc kwarg into
the Adapter upon creation, setting that as the default for all requests
on that Adapter. With this patch, we set raise_exc=False on all Adapter
creations and only override it in the one spot where we were actually
expecting and processing failures as exceptions.
Change-Id: I477afdfce746d405adf77134f57670563fd575e1
The placement database connection (which can use the same connection
string as the api database) needs to be managed from its own module so
that the nova db files are not imported as this eventually leads to
importing object-related and other code which the placement service does
not need.
The wsgi application is now responsible for initializing the database
configuration.
Fixtures and a number of tests are updated to reflected the new
location of the placement engine.
The original parse_args needed to import RPC and database related
modules that are not required in the placement context. A new
_parse_args method is added to wsgi.py which does the minimal required
work. Because we no longer use the central parse_args, the placement
db sync in nova/cmd/manager.py must make a manual configuration of the
context manager.
blueprint placement-extract
Change-Id: I2fff528060ec52a4a2e26a6484bdf18359b95f77
We keep forgetting to bump the minimum required placement version in
nova.cmd.status (and all the related bits and pieces) whenever we
change the report client to require a new version.
This patch interposes a check in the test_report_client functional suite
any time get/put/post/delete is called from the report client. If we
see a microversion higher than the minumum specified in nova.cmd.status,
we raise an exception, which will blow up the test. This should force
the author of a new patch on SchedulerReportClient to do the necessary
paperwork in that patch.
...assuming said author happens to write a test in test_report_client.
This pattern can and should be copied into other test suites where
report client tests are likely to be written, to broaden the scope of
this enforcement.
Change-Id: I5482b92f941261ab6ee6b7cd532ce268c31fe793
If 'connection' is set in the 'placement_database' conf group use
that as the connection URL for the placement database. Otherwise if
it is None, the default, then use the entire api_database conf group
to configure a database connection.
When placement_database.connection is not None a replica of the
structure of the API database is used, using the same migrations
used for the API database.
A placement_context_manager is added and used by the OVO objects in
nova.api.openstack.placement.objects.*. If there is no separate
placement database, this is still used, but points to the API
database.
nova.test and nova.test.fixtures are adjusted to add awareness of
the placement database.
This functionality is being provided to allow deployers to choose
between establishing a new database now or requiring a migration
later. The default is migration later. A reno is added to explain
the existence of the configuration setting.
This change returns the behavior removed by the revert in commit
39fb302fd9 but done in a more
appropriate way.
Note that with the advent of the nova-status command, which checks
to see if placement is "ready" the tests here had to be adjusted.
If we do allow a separate database the code will now check the
separate database (if configured), but nothing is done with regard
to migrating from the api to placement database or checking that.
blueprint placement-extract
Change-Id: I7e1e89cd66397883453935dcf7172d977bf82e84
Implements: blueprint optional-placement-database
Co-Authored-By: Roman Podoliaka <rpodolyaka@mirantis.com>
Change I496e8d64907fdcb0e2da255725aed1fc529725f2 made nova-scheduler
require placement >= 1.25 so this change updates the minimum required
version checked in the nova-status upgrade check command along with the
upgrade docs.
related to blueprint: granular-resource-requests
Change-Id: I0a17ee362461a8ae2113804687799bb9d9216110
Change Id7eecbfe53f3a973d828122cf0149b2e10b8833f made
nova-scheduler require placement >= 1.24 so this change
updates the minimum required version checked in the
nova-status upgrade check command along with the upgrade
docs.
Change-Id: I4369f7fb1453e896864222fa407437982be8f6b5
Dan Smith