A central place to gather all our fixtures.
Change-Id: Iee6b8f809a38fafaf5230bac1f0d01b5f0004c3c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Previously, the way Neutron fixture's update_port method was coded
would cause the binding:profile and binding:vid_details fields to get
clobbered if they were not passed to the update. This was because if
nothing was passed, a default of {} was used. This is not how the real
Neutron API behaves. If nothing is passed, the previous values remain
and are not replaced with {}. This patches fixes this in the Neutron
fixture.
Change-Id: Ia7ad1322b5a15d1407140c77fe0edb179f66ec7a
We have a load of tests to enable specific behavior depending on the
guest architecture, but in many cases we're actually testing the host
architecture. This is wrong and breaks our ability to create guests with
non-host architectures plus our ability to rely on mocking of these APIs
by 'FakeLibvirt'. We fix this by switching checks against the CPU
architecture reported by libvirt's host capabilities, to checks against
the guest image metadata (via 'nova.virt.libvirt.utils.get_arch') with a
fallback to the host architecture (via 'os.uname'). This seemingly
simple change subsequently highlights *a lot* of gaps in the mocking
done by 'FakeLibvirtFixture'. Pretty much all of these stem from the
fact that by previously extracting the CPU host architecture from host
capabilities, we were in practice almost always checking against x86-64.
Now that we're doing checks against image metadata with a fallback to
'os.uname()', we suddenly need to be able to provide host and domain
capabilities for other architectures and doing additional mocks for some
of these architecture-specific code paths.
Note that this change effectively removes test coverage for the
'LibvirtDriver._has_uefi_support' helper because we now mock it in the
'FakeLibvirtFixture' to avoid checks in AArch64 tests (AArch64 defaults
to a UEFI bootloader). This is okay since we're going to remove this
helper entirely in a future change.
Also note that this _still_ isn't as complete as I'd like it to be,
owing to time constraints and the sheer depth of the problem here. The
remaining issues will be addressed in follow-ups, potentially alongside
patches to drop support for legacy architectures like MIPS and I686 (the
former has been EOL'd in favour of RISC-V as of March 2021 and both
Intel and AMD stopped manufacturing products using the latter ISA for
anything other than embedded use cases many moons ago).
Change-Id: I158dd70076ecdbef422ef08c11ce6a140c991537
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We shouldn't be looking up the architecture of the test node during
tests to ensure tests work across nodes with varying architectures.
Change-Id: I458b1db091e33c0b835e44b5a86de6c0a08f99a3
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Co-authored-by: Lee Yarwood <lyarwood@redhat.com>
As explained in the spec, in order to support routed networks, we need
to add a new scheduler pre-filter with a specific conf option
so we can then look at the nova aggregates related to the segments.
Since this pre-filter is called every time we verify the scheduler,
that means that when we move instances, we should be also be able
to only support hosts that are in the same related aggregate.
NOTE(sbauza) : We're just setting admin_api variable in the
integrated helpers as it's used by _get_provider_uuid_by_host()
Implements: blueprint routed-networks-scheduling
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Co-Authored-By: Balazs Gibizer <balazs.gibizer@est.tech>
Change-Id: I667f56612d7f63834863476694cb1f4c71a58302
This change extends the conductor manager to append the cyborg
resource request to the request spec when performing an unshelve.
On shelve offload an instance will be deleted the instance's ARQs
binding info to free up the bound ARQs in Cyborg service.
And this change passes the ARQs to spawn during unshelve an instance.
This change extends the ``shelve_instance``, ``shelve_offload_instance``
and ``unshelve_instance`` rpcapi function to carry the arq_uuids.
Co-Authored-By: Wenping Song <songwenping@inspur.com>
Implements: blueprint cyborg-shelve-and-unshelve
Change-Id: I258df4d77f6d86df1d867a8fe27360731c21d237
This rather beefy (but also quite simple) patch replaces the
'stub_out_image_service' call and associated cleanup in all functional
tests with a new 'GlanceFixture', based on the old 'FakeImageService'.
The use of a fixture means we don't have to worry about teardown and
allows us to stub Glance in the same manners as Cinder, Neutron,
Placement etc.
Unit test cleanup is handled in a later patch.
Change-Id: I6daea47988181dfa6dde3d9c42004c0ecf6ae87a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Enable the policy fixture by default, which should yield more realistic
functional tests. We need to update some tests to use admin APIs where
policy dictates they are necessary. Note that we're currently testing
the legacy policy - not the updated, scoped policy - since the legacy
policy is the default one currently.
Note that we also need to modify the 'SingleCellSimple' fixture in this
change to use the same project ID as the 'OSAPIFixture'.
Change-Id: Ia3dea78f16cb3c7081714c4db36e20d5ee76ed7d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve some TODOs. This is necessary to prove out SR-IOV live
migration.
Change-Id: I8a040f148427cdee7b8a4983efe2b586d73be88d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
setattr kills discoverability, making it hard to figure out who's
setting various fields. Don't do it.
While we're here, we drop legacy compat handlers for pre-Train
compute nodes.
Change-Id: Ie694a80e89f99c8d3e326eebb4590d93c0ebf671
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
For attach:
* Generates InstancePciRequest for SRIOV interfaces attach requests
* Claims and allocates a PciDevice for such request
For detach:
* Frees PciDevice and deletes the InstancePciRequests
On the libvirt driver side the following small fixes was necessar:
* Fixes PCI address generation to avoid double 0x prefixes in LibvirtConfigGuestHostdevPCI
* Adds support for comparing LibvirtConfigGuestHostdevPCI objects
* Extends the comparison of LibvirtConfigGuestInterface to support
macvtap interfaces where target_dev is only known by libvirt but not
nova
* generalize guest.get_interface_by_cfg() to work with both
LibvirtConfigGuest[Inteface|HostdevPCI] objects
Implements: blueprint sriov-interface-attach-detach
Change-Id: I67504a37b0fe2ae5da3cba2f3122d9d0e18b9481
When _poll_unconfirmed_resizes runs or a user tries to confirm
a resize in the API, if the source compute service is down the
migration status will be stuck in "confirming" status if it never
reached the source compute. Subsequent runs of
_poll_unconfirmed_resizes will not be able to auto-confirm the
resize nor will the user be able to manually confirm the resize.
An admin could reset the status on the server to ACTIVE or ERROR
but that means the source compute never gets cleaned up since you
can only confirm or revert a resize on a server with VERIFY_RESIZE
status.
This adds a check in the API before updating the migration record
such that if the source compute service is down the API returns a
409 response as an indication to try again later.
SingleCellSimple._fake_target_cell is updated so that tests using
it can assert when a context was targeted without having to stub
nova.context.target_cell. As a result some HostManager unit tests
needed to be updated.
Change-Id: I33aa5e32cb321e5a16da51e227af2f67ed9e6713
Closes-Bug: #1855927
The nested context warning was raised from contextlib.nested, which is a
Python 2.7-only construct, while the 'pkg_resources' issues are not
present in Python 3.6 and later.
Change-Id: I795f96ab02c6aa0ebbbe211af37707d91dcc9e93
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
DatabaseAtVersion causes the transaction context manager of its
database to be started, but doesn't set SESSION_CONFIGURED as used by
the Database fixture. Consequently, a subsequent test using the
Database fixture will attempt to call configure on a started context
manager, and fail.
We fix this by rolling the functionality of DatabaseAtVersion into
Database.
Change-Id: Ia6b63e0d26df53e611a5a36e5a2eeb09a73e4704
Closes-bug: #1804811
While reviewing [1] it was noted that we've probably been invoking the
real netifaces.interfaces() in some test paths. This isn't good, so
introduce a fixture that blows up any test that does that.
(NB: during development, the netifaces.interfaces()-specific poison
fixture evolved into a generic fixture that can be extended in the
future to poison anything in a similar fashion.)
[1] https://review.opendev.org/#/c/671471/
Change-Id: I1fea14d5be10bb4e884f52e0ae8be722519ddd3f
These particular imports are no longer needed in a Python 3-only world.
Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This recreates the referenced bug, demonstrating that requests for
versioned discovery endpoints (/v2, /v2.1) are being piped through
authentication.
Change-Id: Iaef1229f542e4e824c6c5c73335bc601bed08c04
Related-Bug: #1845530
This patch series now works for many VM operations with libvirt:
* Creation, deletion of VM instances.
* Pause/unpause
The following works but is a no-op:
* Lock/unlock
Hard reboots are taken up in a later patch in this series.
Soft reboots work for accelerators unless some unrelated failure
forces a hard reboot in the libvirt driver.
Suspend is not supported yet. It would fail with this error:
libvirtError: Requested operation is not valid:
domain has assigned non-USB host devices
Shelve is not supported yet.
Live migration is not intended to be supported with accelerators now.
Change-Id: Icb95890d8f16cad1f7dc18487a48def2f7c9aec2
Blueprint: nova-cyborg-interaction
* Call Cyborg with device profile name to get ARQs (Accelerator Requests).
Each ARQ corresponds to a single device profile group, which
corrresponds to a single request group in request spec.
* Match each ARQ to associated request group, and thereby obtain the
corresponding RP for that ARQ.
* Call Cyborg to bind the ARQ to that host/device-RP.
* When Cyborg sends the ARQ bind notification events, wait for those
events with a timeout.
Change-Id: I0f8b6bf2b4f4510da6c84fede532533602b6af7f
Blueprint: nova-cyborg-interaction
Find the name of the device profile, if any, in flavor extra specs.
Get its profile groups (equiv to flavor request groups) from Cyborg.
Parse/validate them similar to extra_specs.
Generate RequestGroup objects and add them to the request spec
(in requested_resources field, following precedent).
Change-Id: Icd2ee9024dd4af0a7eb105eca14df8e458e9de77
Blueprint: nova-cyborg-interaction
As noted in a TODO by mriedem, we were making two trips to the DB when
retrieving 'Service' objects in order to build a list of availability
zones. Optimize things so that this is no longer the case.
Change-Id: Id0d8c52cba8b548a0e6436d94765b84f040efee5
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
At some point in the past, there was only nova-network and its code
could be found in 'nova.network'. Neutron was added and eventually found
itself (mostly!) in the 'nova.network.neutronv2' submodule. With
nova-network now gone, we can remove one layer of indirection and move
the code from 'nova.network.neutronv2' back up to 'nova.network',
mirroring what we did with the old nova-volume code way back in 2012
[1]. To ensure people don't get nova-network and 'nova.network'
confused, 'neutron' is retained in filenames.
[1] https://review.opendev.org/#/c/14731/
Change-Id: I329f0fd589a4b2e0426485f09f6782f94275cc07
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Finish the job by removing all the now-unused modules. This also allows
us to - wait for it - kill mox at long last. It's a great day in the
parish.
Partial-Implements: blueprint remove-nova-network-ussuri
Partial-Implements: blueprint mox-removal-ussuri
Change-Id: Ia33ec2604b2fc2d3b6830b596cac669cc3ad6c96
A future change will bump flake8 to 3.x. See off most of the issues that
this will introduce now, with the exception of some missing typing
imports in 'nova/virt/hardware.py' - fixing those here with the current
version of flake8 would actually raise an error about unused imports.
Change-Id: I9480ac1749d448efe4f415f5e80ff9b9837216b6
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This patch removes the high level mocking of the
neutronv2.api.API.setup_networks_on_host and the
neutronv2.api.API.migrate_instance_start calls from NeutronFixture as
the fixture now has all the necessary low level stubs in place.
This allows the fixture to track multiple port bindings properly
through the low level mocked calls as the
migrate_instance_start and the setup_networks_on_host calls also used to
manage multiple bindings.
The unit test base class change reintroduces the stub to
migrate_instance_start for the unit test only as there more work would
be needed on the network info cache handling to keep
migrate_instance_start un-stubbed.
Change-Id: Idcc0a0e827ba6d0a99edd7fbd077a2ff4afa7443
blueprint: support-move-ops-with-qos-ports-ussuri
Live migration uses multiple port bindings but the NeutronFixture used
in the functional tests does not track such bindings. Later patches will
need to assert that such bindings are properly updated with allocation
keys during the migration with ports having resource request.
To be able to do it this patch extends the NeutronFixture.
To allow easy stubbing this patch refactors
neutronv2.api.APi.activate_port_binding() and introduces
neutronv2.api.APi._get_port_binding()
blueprint: support-move-ops-with-qos-ports-ussuri
Change-Id: Id349f2f355b89445139e58e6efc38a0daabe9e91
This adds the ConfirmResizeTask task which will be called
from the API to orchestrate confirming a cross-cell resize.
This task is responsible for cleaning up the source host
and cell including destroying the guest from the source
hypervisor and hard deleting the instance-related records from
the source cell database. It also updates the information
about the instance and migration in the target cell database.
The resize.confirm.(start|end) notifications, which are
traditionally sent from the confirm_resize() method on the
source host, are sent from this task since confirming a cross
cell resize spans multiple cells.
Part of blueprint cross-cell-resize
Change-Id: I5b9d41ef34385689d8da9b3962a1eac759eddf6a
None of our functional live migration tests uses neutron ports. Simply
adding a port to the boot request shows that the live migration step of
the tests fails because the compute manager's
_do_pre_live_migration_from_source() call wait for vif-plugged events
from Neutron but our Neutron fixture does not send it.
This does not cause problem during a simple boot because during boot the
virt driver triggers the waiting for the vif-plug event and in the
functional test we have a FakeDriver that does not do it.
This patch implements the simplest solution to make it possible to test
live migration with neutron ports. It simply turns off the waiting by
setting CONF.vif_plugging_timeout to 0
A nicer solution would be to extend either the Neutron fixture or the
fake virt driver to send vif-plug event at certain triggers. But that
would require a lot more surgery.
This patch is a prerequisite to later patches covering live migration
with qos neutron ports.
blueprint: support-move-ops-with-qos-ports-ussuri
Change-Id: I63c1109dcdb9132cdbc41010654c5fdb31a4fe31
For an unbound port real neutron return {} as binding:profile. In the
NeutronFixture the binding:profile key was missing in the predefined
ports. This made the tests asserting the binding:profile inconsistent.
This patch removes the inconsistency by make NeutronFixture behave like
real neutron.
Change-Id: I5a41dbd5c2d28b28bd8e464b6aec3b28ca3b1163
blueprint: support-move-ops-with-qos-ports-ussuri
With the impending removal of nova-network, there's nothing that should
be using this. Remove it.
Change-Id: I352b71b5976d008c2b8fab8a6d6939c0e0b305be
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This addresses a NOTE/TODO from change
I3cf6eb4654663865d9258c38f05cd05974ffcf9d where the NeutronFixture is
storing "admin-ness" from the last get_client call and using it to
control the behavior of the list_ports method.
This adds a wrapper class that represents a Neutron client object and
stores a reference to a NeutronFixture in order to track admin-ness for
separate get_client calls.
While making this change, unused methods:
fake_get_instance_security_group_bindings and _get_first_id_match were
noticed in NeutronFixture, so they are removed.
Change-Id: I7aa05e857599db820d0fc4daeb730f2a40b3291c
This adds the beginning of functional tests for
cross-cell resize which get the instance to VERIFY_RESIZE
state and assert various aspects of the instance both in
the source cell and target cell.
The fake virt driver is updated to support creating a
server with device tags and also to have a host-specific
volume connector dict to aid in debugging test failures.
Confirm/revert resize flows will build on these tests.
Part of blueprint cross-cell-resize
Change-Id: I398aeb73da6bf09e177e5f1fdc668f59f06de8dd
Driver tests were unable to run on Windows due to some import
errors.
Specifically, the 'setUp' method of the 'PoisonFunctions'
class would import 'nova.virt.libvirt' which in turn would cause
an error when running the Windows tests.
This change fixes this issue.
Partial-Bug: #1736395
Change-Id: Id21a24257ec1e00a1eb942aec725ae76e9f6e688
The nova-api checks at each move* operation if the instance has qos port
attached as not all the move operations are supported for such servers.
Nova uses the request context to initialize the neutron client for the
port query. However neutron does not return the value of the
resource_request of the port if it is queried with a non admin client.
This causes that if the move operation is initiated by a non admin
then nova thinks that the ports do not have resource request.
This patch creates an admin context for this neutron query.
The new functional tests are not added before this patch in a regression
test like way as existing functional tests are reused with different
setup and doing that without the fix causes a lot of different failure
scenarios.
Note that neutron fixture is changed to simulate the different behavior
in case of different request context are used to initialize the client.
*: Note that Id5f2f4f22b856c989e2eef8ed56b9829d1bcefb6 removed the check
for evacuate in Ussuri but exists in Train and Stein.
Change-Id: I3cf6eb4654663865d9258c38f05cd05974ffcf9d
Closes-Bug: #1850280
Stephen Finucane