Make the host class look under '/sys/fs/cgroup/cgroup.controllers' for support of the cpu controller. The host will try searching through cgroupsv1 first, just like up until now, and in the case that fails, it will try cgroupsv2 then. The host will not support the feature if both checks fail.
This new check needs to be mocked by all tests that focus on this piece of code, as it touches a system file that requires privileges. For such thing, the CGroupsFixture is defined to easily add suck mocking to all test cases that require so.
I also removed old mocking at test_driver.py in favor of the fixture from above.
Partial-Bug: #2008102
Change-Id: I99b57c27c8a4425389bec2b7f05af660bab85610
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib. Most of this
is autogenerated, as described below, but there is one manual change
necessary:
nova/tests/functional/regressions/test_bug_1781286.py
We need to avoid using 'fixtures.MockPatch' since fixtures is using
'mock' (the library) under the hood and a call to 'mock.patch.stop'
found in that test will now "stop" mocks from the wrong library. We
have discussed making this configurable but the option proposed isn't
that pretty [1] so this is better.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('nova/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Note that we cannot remove mock from our requirements files yet due to
importing pypowervm unit test code in nova unit tests. This library
still uses the mock lib, and since we are importing test code and that
lib (correctly) only declares mock in its test-requirements.txt, mock
would not otherwise be installed and would cause errors while loading
nova unit test code.
[1] https://github.com/testing-cabal/fixtures/pull/49
Change-Id: Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
These were left to last since there's a bit of cleanup necessary to move
everything across.
Change-Id: I921c812ac03f7d32eec31200772020c17f292851
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Move these to the central place. There's a large amount of test damage
but it's pretty trivial.
Change-Id: If581eb7aa463c9dde13714f34f0f1b41549a7130
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This was replaced by the '[DEFAULT] pointer_model' config option was
back in the 14.0.0 (Newton) release.
Change-Id: Ia39c0bad4c1c03b3ffb4a162c2afddb44ebaf6a1
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This rather beefy (but also quite simple) patch replaces the
'stub_out_image_service' call and associated cleanup in all functional
tests with a new 'GlanceFixture', based on the old 'FakeImageService'.
The use of a fixture means we don't have to worry about teardown and
allows us to stub Glance in the same manners as Cinder, Neutron,
Placement etc.
Unit test cleanup is handled in a later patch.
Change-Id: I6daea47988181dfa6dde3d9c42004c0ecf6ae87a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
When evacuating an instance between compute hosts on shared storage,
during the rebuild operation we call spawn() on the destination
compute. spawn() currently assumes that it should cleanup all
resources on failure, which results in user data being deleted in the
evacuate case.
This change modifies spawn in the libvirt driver such that it only
cleans up resources it created.
Co-Authored-By: Lee Yarwood <lyarwood@redhat.com>
Closes-Bug: #1550919
Change-Id: I764481966c96a67d993da6e902dc9fc3ad29ee36
(cherry picked from commit 083df01a4d)
1. Claim allocations from placement first, then claim specific
resources in Resource Tracker on destination to populate
migration_context.new_resources
3. cleanup specific resources when live migration succeeds/fails
Because we store specific resources in migration_context during
live migration, to ensure cleanup correctly we can't drop
migration_context before cleanup is complete:
a) when post live migration, we move source host cleanup before
destination cleanup(post_live_migration_at_destination will
apply migration_context and drop it)
b) when rollback live migration, we drop migration_context after
rollback operations are complete
For different specific resource, we might need driver specific support,
such as vpmem. This change just ensures that new claimed specific
resources are populated to migration_context and migration_context is not
droped before cleanup is complete.
Change-Id: I44ad826f0edb39d770bb3201c675dff78154cbf3
Implements: blueprint support-live-migration-with-virtual-persistent-memory
Also update some outdated URLs at the same time, e.g. defcore is now
interop.
Unfortunately unlike GitHub, gitea doesn't yet support URLs with
shortened SHA1s; however this is being worked on:
https://github.com/go-gitea/gitea/issues/6450
Change-Id: I6e6b63619f1138cc961b61be548453361d01f73c
This hasn't been needed since we moved console authentication to the
database in Rocky.
Part of blueprint remove-consoleauth
Change-Id: Ie5e7d70f9d6af77edd22756128937f06d0b961a9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
The recently updated minimum required libvirt version (1.3.1; in commit
403320b -- libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Rocky") brings
in the newer libvirt migration API, migrateToURI3(). The newer API was
explicitly designed[*] to be backward compatible with the older variant.
So remove the usage of the older variants:
migrateToURI()
migrateToURI2()
And just stick to the newer API -- migrateToURI3().
Clean up the following:
- Add the 'migrate_disks' and 'destination_xml' paramters, and remove
the no longer needed 'domain_xml' from the Nova migrate() method.
- Remove or fix various unit tests to use migrateToURI3().
- Stub nova.virt.libvirt.guest.Guest.migrate() correctly in
nova/tests/unit/virt/test_virt_drivers.py.
[*] https://libvirt.org/git/?p=libvirt.git;a=commit;h=4bf62f4 --
Extensible migration APIs
Change-Id: Id9ee1feeadf612fa79c3d280cee3a614a74a00a7
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
The last time we incremented versions for libvirt and QEMU was during
the "Pike" release (commit: b980df0, 11-Feb-2017); for "Queens" we
didn't do any.
The advertized NEXT_MIN_{LIBVIRT,QEMU} versions during
"Pike" release were set to: libvirt 1.3.1 and QEMU 2.5.0 -- but they
weren't actually bumped for "Queens". So let's actually bump them for
the "Rocky" release.
Fix the affected functional tests. And where applicable, use the
FAKE_{LIBVIRT, QEMU}_VERSION constants in tests, instead of hard-coding
versions.
Also fix the following live migration unit tests in the libvirt driver:
- test_live_migration_update_graphics_xml
- test_live_migration_update_serial_console_xml
- test_live_migration_with_valid_target_connect_addr
- test_live_migration_raises_exception
In above tests the following, two main changes were made:
- "Mock" the migrateToURI3() API instead of migrateToURI2() -- because
with the libvirt version bump, Nova's guest.migrate() method will
now call libvirt's newer API migrateToURI3() with 'params' dict. So
adjust the tests accordingly.
- Replace the usage of "Mox" with "Mock" in relevant tests. (I know
we should try to stick to the sensible rule of "one logical change
per patch". But the tests were already failing and we are
systematically removing the usage of Mox in Nova source tree anyway.
On that grounds, this seems reasonable.)
Also drop the following version constants, which are no longer needed
and adjust / remove corresponding unit tests:
- MIN_{LIBVIRT_KVM,QEMU}_S390_VERSION
- MIN_LIBVIRT_KVM_PPC64_VERSION
- Drop MIN_LIBVIRT_VIRTUOZZO_VERSION
Note that subsequent patches will handle clean up of remaining (now no
longer needed) version constants and compatibility code.
Change-Id: If0a091a7441f2c3269148e40ececc3696d69684c
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
In 'start_service' of nova.test.TestCase,
there is the following statement.
self.useFixture(nova_fixtures.ConfPatcher(host=host))
So it is not necessary to set the 'host' parameter
in configuration in the outside of 'start_service'.
TrivialFix
Change-Id: I01e20309df39b24edef91675d2812201d063f66e
FilterScheduler is the only scheduler driver that properly integrated
with placement api so this patch changes the functional test environment
to use the FilterScheduler instead of the ChanceScheduler.
The amount of vcpu the SmallFakeDriver reports need to be bumped to 2
as during resize to same host the filter scheduler doubles the
allocation on the host. Simply relying on the 16.0 default allocation
ratio does not work as the max_unit of the vcpu does not use the
allocation ratio to avoid that a single server overallocates the cpu.
The only change in unit test is
test_create_instance_with_oversubscribed_cpu which also loads the
SmallFakeDriver and asserts available resources on it.
Change-Id: I12de2e195022593ea2a3e2894f2c3b5226930d4f
Replace mox stubs with stub_out
in nova/tests/unit/cast_as_call.py.
Change-Id: I416835a2fa0ad807f3c35c3cb14b0f463a5d1145
Implements: blueprint remove-mox-pike
This patch replaces the scheduler's use of GET
/resource_providers?resources=XXX with GET
/allocation_candidates?resources=XXX.
In doing so, we move the interaction with the placement API out of the
scheduler driver interface and up into the scheduler manager. This
allows us to make fewer changes to the underlying HostManager and
SchedulerDriver interfaces and isolate communication with the placement
API in a single place.
The provider_summaries part of the response from GET
/allocation_candidates is used to generate the UUIDs that winnow the
number of compute nodes retrieved by the filter scheduler during
scheduling. Following patches will add in support for actually doing
the claim from the scheduler against one or more resources providers by
examining the allocation_requests part of the HTTP response and picking
one that contains the host the scheduler picked during its _schedule()
loop.
Change-Id: I1c0bd2987dcbc38f23b71db2bc8e3267f85168c8
blueprint: placement-allocation-requests
This changes the use_neutron configuration option default
value to True. nova-network has been deprecated since Newton
and only runs in a cells v1 configuration now, so it's time
to make Neutron the default configuration for new Nova
deployments.
As far as testing, the unit tests have already been taken
care of for use_neutron=True by default. There are several
functional tests which are converted over to using Neutron
via the fixture in this change. However, the legacy API
samples functional tests were written with nova-network stubs
so it will take some time to convert over to using Neutron
stubs. This patch does not attempt to clean all of those up
and just configures nova-network by default. The remaining
tests will need to be converted to using Neutron in a later
series of changes.
Depends-On: I82721b5d10711401b9b0ebc2b0ed07cc8287bbf7
Implements blueprint use-neutron-by-default
Change-Id: I8388c29ad310cd8800084b4d5c026013158bfbed
compute_manager config options were deprecated in
13.0.0, we can remove it now.
Depends-On: I93b62372e357db4f7c39992c244b2ec9e594ad9c
Implements: bp centralize-config-options-ocata
Change-Id: I84503655bd9675ff441073e51059202142db22ab
The following service manager config options were deprecated in
13.0.0 and remove them now:
- metadata_manager
- console_manager
- consoleauth_manager
- cert_manager
- scheduler_manager
- conductor.manager
Implements: bp centralize-config-options-ocata
Co-Authored-By: Stephen Finucane <sfinucan@redhat.com>
Change-Id: Ia50d18c350dbc4ca88da9543e1236e3398ea28ee
Move all scheduler options into their one of two groups. Many of the
options are simply renamed to remove their 'scheduler_' prefix, with
some exceptions:
* scheduler_default_filters -> enabled_filters
* scheduler_baremetal_default_filters -> baremetal_enabled_filters
* scheduler_driver_task_period -> periodic_task_interval
* scheduler_tracks_instance_changes -> track_instance_changes
Change-Id: I3f48e52815e80c99612bcd10cb53331a8c995fc3
Co-Authored-By: Stephen Finucane <sfinucan@redhat.com>
Implements: blueprint centralize-config-options-ocata
If the graphical consoles VNC and SPICE are disabled, the live-migration
of an instance will result in an ERROR state. VNC and SPICE are usually
disabled on IBM z systems platforms where graphical consoles are not
available. The serial console is then enabled and VNC + SPICE are
disabled. The logged error message will be:
"Migration error: Your libvirt version does not support the
VIR_DOMAIN_XML_MIGRATABLE flag or your destination node does
not support retrieving listen addresses. In order for live
migration to work properly you must either disable serial
console or upgrade your libvirt version."
This functional test executes the live migration flow with enabled serial
console and disabled VNC + SPICE and reproduces bug 1595962.
Related-Bug: 1595962
Change-Id: Ib1b22ce77d8a36e55944acc1fbca042906aa41ef
Jorge San Emeterio