With Libvirt v8.7.0+, the <maxphysaddr> sub-element
of the <cpu> element specifies the number of vCPU
physical address bits [1].
[1] https://libvirt.org/news.html#v8-7-0-2022-09-01
New flavor extra_specs and image properties are added to
control the physical address bits of vCPUs in Libvirt guests.
The nova-scheduler requests COMPUTE_ADDRESS_SPACE_* traits
based on them. The traits are already defined in os-traits
v2.10.0. Also numerical comparisons are performed at
both compute capabilities filter and image props filter.
blueprint: libvirt-maxphysaddr-support-caracal
Change-Id: I98968f6ef1621c9fb4f682c119038e26d62ce381
Signed-off-by: Nobuhiro MIKI <nmiki@yahoo-corp.jp>
This change adds a new hw:locked_memory extra spec and hw_locked_memory
image property to contol preventing guest memory from swapping.
This change adds docs and extend the flavor
validators for the new extra spec.
Also add new image property.
Blueprint: libvirt-viommu-device
Change-Id: Id3779594f0078a5045031aded2ed68ee4301abbd
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib. Most of this
is autogenerated, as described below, but there is one manual change
necessary:
nova/tests/functional/regressions/test_bug_1781286.py
We need to avoid using 'fixtures.MockPatch' since fixtures is using
'mock' (the library) under the hood and a call to 'mock.patch.stop'
found in that test will now "stop" mocks from the wrong library. We
have discussed making this configurable but the option proposed isn't
that pretty [1] so this is better.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('nova/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Note that we cannot remove mock from our requirements files yet due to
importing pypowervm unit test code in nova unit tests. This library
still uses the mock lib, and since we are importing test code and that
lib (correctly) only declares mock in its test-requirements.txt, mock
would not otherwise be installed and would cause errors while loading
nova unit test code.
[1] https://github.com/testing-cabal/fixtures/pull/49
Change-Id: Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
arch in libvirt driver support
This is split 1 of 3 for the architecture emulation feature.
This adds the 'hw_emulation_architecture' property to the image meta
properties, allowing for operator to define whether they will use
emulation or not.
This adds the capability as a feature to ensure no impact to normal
operations or functionality.
Account for object versioning has been added to raise exceptions and
handle proper
Implements: blueprint pick-guest-arch-based-on-host-arch-in-libvirt-driver
Signed-off-by: Jonathan Race <jrace@augusta.edu>
Change-Id: If4f598c0d3f9e64617beb54450faa04e7d20dd20
Finish up removing these entries from the versioned instance
notifications. They're useless since we dropped support for the XenAPI
virt driver. The underlying model is retained for now: that will be
handled separately.
Change-Id: I774c50fca99bc655ca5010e3b9d8247b739293b3
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The 'bochs' display device is a recommended[1] safe option for UEFI
guests. This is supported in libvirt from version 5.6.0; Nova's current
MIN_LIBVIRT_VERSION (6.0.0) satisfies this requirement.
Partially quoting the "bochs display device" section from a post written
by a QEMU graphics maintainer[1]:
[...] Main advantage over standard VGA is that this device is alot
simpler. The code size and complexity needed to emulate this device
is an order of magnitude smaller, resulting in a reduced attack
surface. Another nice feature is that you can place this device in a
PCI Express slot.
For UEFI guests it is safe to use the bochs display device instead
of the standard VGA device. The firmware will setup a linear
framebuffer as GOP anyway and never use any legacy VGA features.
For BIOS guests this device might be useable as well, depending on
whenever they depend on direct VGA hardware access or not. There is
a vgabios which supports text rendering on a linear framebuffer, so
software which uses the vgabios services for text output will
continue to work. Linux bootloaders typically fall into this
category. The linux text mode console (vgacon) uses direct hardware
access and does not work. The framebuffer console (fbcon running on
vesafb or bochs-drm) works.
[1] https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/
Implements: blueprint add-bochs-display-device
Change-Id: Id194028f5d2cbaac1c5e699b71fd9751f87f5da5
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.
The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.
Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The USB-based tablet is often the only USB device in an x86 instance,
while the USB-based keyboard is often the only such device in an AArch64
instance (x86 have PS2 keyboards and mice). Replacing these with
virtio-based devices can eliminate the need to have a USB host adapter
in the instance. Enable just that possibility by adding a new value
image metadata property, 'hw_input_bus'. This allows us to specify not
only virtio-based pointer and keyboard input devices but also USB
equivalents.
Note that this also fixes one instance of a particular class of bugs,
whereby we have checks for *guest* architecture-specific behavior that
are being toggled based on the *host* architecture. In this instance,
we were attempting to add a keyboard device on AArch64 guests since they
don't have one by default, but we were determining the architecture by
looking at the CPU architecture reported in the host capabilities. By
replacing this check of the host capabilities with a call to the
'nova.virt.libvirt.utils.get_arch' helper, we correctly handle requests
to create non-host architecture guests via the 'hw_architecture' image
metadata property. There are many other instances of this bug and those
can be resolved separately.
Change-Id: If9f3ede3e8449f9a6c8d1da927974c0a73923d51
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This patch adds the 'socket' value to the allowed PCI NUMA affinity
policies, both to the 'hw:pci_numa_affinity_policy' flavor extra spec,
and the 'hw_pci_numa_affinity_policy' image property.
For now the new value is a no-op and remains undocumented. It will be
wired-in in a subsequent patch.
Implements: blueprint pci-socket-affinity
Change-Id: I0680d4e21f3e317ac702b55afef4c87e8acbfc3a
Add support for the 'hw:tpm_version' and 'hw:tpm_model' flavor extra
specs along with the equivalent image metadata properties. These are
picked up by the scheduler and transformed into trait requests. This is
effectively a no-op for now since we don't yet have a driver that
reports these traits.
Part of blueprint add-emulated-virtual-tpm
Change-Id: I8645c31b4ecb18afea592b2a5b360b0165626009
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Introduce a 'mixed' instance CPU allocation policy and
will be worked with upcoming patches, for purpose of
creating an instance combined shared CPUs with dedicated
or realtime CPUs.
In an instance mixed with different type of CPUs, the shared CPU
shared CPU time slots with other instances, and also might be a
CPU with less or un-guaranteed hardware resources, which implies
to have no guarantee for the behavior of the workload running on
it. If we call the shared CPU as 'low priority' CPU, then the
realtime or dedicated CPU could be called as 'high priority' CPU,
user could assign more hardware CPU resources or place some
guaranteed resource to it to let the workload to entail high
performance or stable service quality.
Based on https://review.opendev.org/714704
Part of blueprint use-pcpu-and-vcpu-in-one-instance
Change-Id: I99cfee14bb105a8792651129426c0c5a3749796d
Signed-off-by: Wang Huaqiang <huaqiang.wang@intel.com>
Update the versioned object notification for 'InstanceNUMACell'
object to reflect the change of a new 'pcpuset' field.
Part of blueprint use-pcpu-and-vcpu-in-one-instance
Change-Id: Ifeb21b9eff828952f67002172c8500f738e9984c
Signed-off-by: Wang Huaqiang <huaqiang.wang@intel.com>
ImageMetaPropsPayload (for notifications) was supposed to contain
exactly the same fields as ImageMetaProps itself; and the SCHEMA member
was those same keys in a trivially different shape. And we were having
trouble remembering to keep these two objects in sync before. So this
commit eliminates the repetition and just generates
ImageMetaPropsPayload's `fields` and `SCHEMA` from
ImageMetaProps.fields.
Change-Id: I7dc38f7f5a62098fb0c366736f27e23bd7b7ee86
This change adds missing fields to the image metadata
notification object and adds a unit tests to assert
that the notification object and nova object stay in sync.
This change also adds unit tests to assert the notification
schema and fields are in sync and that the schema is
valid.
Change-Id: I36ea5d5e677ab3e6c88223b20f5377e9471c55db
Closes-Bug: #1856100
This addresses bug #1795920 by adding support for
defining a pci numa affinity policy via the flavor
extra specs or image metadata properties enabling
the policies to be applied to neutron sriov port
including hardware offloaded ovs.
Closes-Bug: #1795920
Related-Bug: #1805891
Implements: blueprint vm-scoped-sriov-numa-affinity
Change-Id: Ibd62b24c2bd2dd208d0f804378d4e4f2bbfdaed6
This adds AggregateCacheNotification, related payload, and code in
conductor to emit this per-compute with progress information. This
also adds a "progress" phase to NotificationPhase, which allows for
start..progress..progress..end information for a single operation
(cache_images in this case).
Related to blueprint image-precache-support
Change-Id: I69ae26d4caf4b56ab2c4864455bfe9b5b736dbf3
This adds the bulk of the image pre-caching logic to the conductor
task manager. It takes an aggregate and list of image ids from the
API service and handles the process of calling to the relevant compute
nodes to initiate the image downloads, honoring the (new) config knob
for overall task parallelism.
Related to blueprint image-precache-support
Change-Id: Id7c0ab7ae0586d49d88ff2afae149e25e59a3489
- This change extends the VideoModel field object to allow 3 new values
(virtio, gop, none)
- This change makes the libvirt driver use ALL tuple from the
nova.fields.VideoModel object instead of declaring a second
tuple inline for validation.
- This change allows the virtio video model to now be used
for all architectures when explicitly requested via the
hw_video_model image metadata property
- This change introduces unit tests and a release note
for the new capablities.
Change-Id: I2830ccfc81cfa9654cfeac7ad5effc294f523552
Implements: blueprint libvirt-video-device-models
This patch adds a new parameter ``locked_reason`` to
``POST /servers/{server_id}/action`` request where the
action is lock. It enables the user to specify a reason when locking
a server.
The locked_reason will be exposed through ``GET servers/{server_id}``,
``GET /servers/detail``, ``POST /servers/{server_id}/action`` where
the action is rebuild and ``PUT servers/{server_id}`` requests' responses.
The InstanceActionNotification will emit the locked_reason
along with the other instance details. This patch hence changes the
payload object to include the "locked_reason" field.
Note that "locked" will be allowed as a valid filtering/sorting parameter
for ``GET /servers/detail`` and ``GET /servers`` from this new microversion.
Implements blueprint add-locked-reason
Change-Id: I46edd595e7417c584106487123774a73c6dbe65e
This brings in a couple of new checks which must be addressed, many of
which involve a rather large amount of changes, so these are ignored for
now. A series of follow-up changes will resolved these.
'pycodestyle' is added as a dependency rather than it being pulled in
transitively. This is necessary since we're using it in tests.
Change-Id: I35c654bd39f343417e0a1124263ff31dcd0b05c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Change I019e88fabd1d386c0d6395a7b1969315873485fd in Stein, which
is not yet officially released, exposes the unencrypted
database_connection URL and MQ transport_url to a CellMapping in
the select_destinations versioned notification CellMappingPayload.
While notifications are not meant to be consumed by end users of
the cloud but only internal services of the deployment, it still
seems like a bad idea to give the keys to the nova cell DB and MQ
to an external-to-nova service like ceilometer.
This change removes the fields from the CellMappingPayload and
bumps the major version to 2.0 to signal the change to consumers,
although I don't expect anything is consuming this yet but we should
follow standard versioning procedure anyway.
Note that notification consumers do not request a specific payload
version nor do they get a schema to perform their own backporting,
they just get what they get, so after this there should be no worry
about needing to support the 1.0 format for this payload.
Change-Id: Ib5edea32d15db01000e6730aebceaf119daf8c5c
Closes-Bug: #1823104
Transform the scheduler.select_destinations.start and
scheduler.select_destinations.end notifications to the
versioned notification framework.
Change-Id: I019e88fabd1d386c0d6395a7b1969315873485fd
Implements: bp versioned-notification-transformation-stein
The following notifications have been transformed to
the versioned notification framework.
* compute_task.build_instances
* compute_task.migrate_server
* compute_task.rebuild_server
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: Ibfb0a6db5920d921c4fc7cabf3f4d2838ea7f421
Implements: bp versioned-notification-transformation-stein
The volume.usage notification has been transformed to
the versioned notification framework.
Change-Id: Ica45a95d26b602f9a149d42516baf4b84fc01cec
Implements: bp versioned-notification-transformation-stein
The libvirt.error notification has been transformed to
the versioned notification framework.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: I7d2287ce06d77c0afdef0ea8bdfb70f6c52d3c50
Implements: bp versioned-notification-transformation-stein
The metrics.update notification has been transformed to
the versioned notification framework.
Change-Id: I2f6aba0c032baf181b103d2a16b8e768815170c9
Implements: bp versioned-notification-transformation-rocky
The instance.live_migration_force_complete.start and
instance.live_migration_force_complete.end notifications have been
transformed to the versioned notification framework.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: I2e9a474457a7597a9febc111b13a67ddcf45f29e
Implements: bp versioned-notification-transformation-rocky
The aggregate.update_prop.start and aggregate.update_prop.end
notifications has been transformed to the versioned notification
framework.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: I37b19573b6d0e1131c446fcec361f01fa2560f82
Implements: bp versioned-notification-transformation-rocky
In this patch, the ServerGroupPayload is updated to include
the new ``policy`` field; the ``policies`` field is deprecated
for removal but still put into the notification payload for
backward compatibility.
Related to blueprint complex-anti-affinity-policies
Change-Id: Ie739ee8dec4685cd70e735ff83f7f30bc7e95a57
nova/db/__init__.py was importing * from nova.db.api. This meant that
any time any code anywhere within the nova.db package was imported
then nova.db.api was too, leading to a cascade of imports that may
not have been desired. Also, in general, code in __init__.py is a pain.
Therefore, this change adjusts code that so that either:
* nova.db.api is used directly
* nova.db.api is imported as 'db'
In either case, the functionality remains the same.
The primary goal of this change was to make it possible to import the
model files without having to import the db api. Moving the model files
to a different place in the directory hierarchy was considered, but
given that "code in __init__.py is a pain" this mode was chosen.
This looks like a very large change, but it is essentially adjusting
package names, many in mocks.
Change-Id: Ic1fd7c87ceda05eeb96735da2a415ef37060bb1a
The instance.rebuild_scheduled notification has been transformed
to the versioned notification framework.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: I640562b8dd049ff5b6fea9286dc3082b9d837ed3
Implements: bp versioned-notification-transformation-rocky
The instance action notifications contain the user id and the
project id of the owner of the instance. However an instance
action might be initiated by another user. It could be another
user from the same project or can be an admin from the admin project.
To be able to distinguish between the user who initiated the instance
action from the user owning the instance we need to add two new
fields to the instance action notifications, action_initiator_user
and action_initiator_project
Change-Id: I649d8a27baa8840bc1bb567fef027c749c663432
Closes-bug: #1744658
Blueprint: add-action-initiator-to-instance-action-notifications
The instance.unlock versioned notification is introduced in this
patch.
The unlock operation just changes the instance.locked to False in
API, we send the notification after db operation.
Change-Id: Ic750c33b4f88ba9c62ea8cba86915c6010f2cd6f
blueprint: trigger-notifications-when-lock-unlock-instances
The instance.lock versioned notification is introduced in this
patch.
The lock operation just changes the instance.locked to True in
API, we send the notification after db operation.
Change-Id: I03a13afa91c29f4fee25336cb82ea5c887eae2f8
blueprint: trigger-notifications-when-lock-unlock-instances
This patch adds full traceback to ExceptionPayload in versioned
notifications.
The instance fault field and instance-action REST API has already
provide the traceback to the admin users (controlable through policy)
and the notifications are also admin only things as they are emitted
to the message bus by default. So it is assumed that security is not
a bigger concern for the notification than for the REST API.
On the ML [1] post there was no objection to add new string field to the
ExceptionPayload that will hold the serialized traceback object.
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128105.html
Implements: blueprint add-full-traceback-to-error-notifications
Change-Id: Id587967ea4f9980c292492e2f659bf55fb037b28
Add the 'trusted_image_certificates' field to InstanceCreatePayload
and InstanceActionRebuildPayload notifications.
Change-Id: Ib5b50a3889ab15d5aac992f92e9be372a915eeff
This patch contains the instance.exists related notification
transformation. The functional test for instance_usage_audit
periodic task case is added in a subsequent patch.
Change-Id: I1a0afa0e8740c229db77c18b932e316196880de5
Implements: bp versioned-notification-transformation-rocky
This moves the tests which are specific to payloads defined in
nova/objects/notifications/objects/instance.py from the generic
nova.objects.unit.notifications.objects.test_notification into
the more path-appropriate test_instance module. The test_notification
module should probably be renamed to test_base but that's left for
another day.
Change-Id: I1eda20634bd788bc892aaa312bafe87461eb4b4d
As it was agreed on the Rocky PTG [1] it is useful to have the
request_id of in the payload of every instance action versioned
notification. For example it could help the deployer connect
the state change described in the notification with the user
action, the request, on the REST API.
So this patch proposes to extend the InstanceActionPayload
versioned object with a new request_id field and populate
the request_id from the context object used for emitting
the instance action notifications.
[1] https://etherpad.openstack.org/p/nova-ptg-rocky L391
Implements: bp add-request-id-to-instance-action-notifications
Change-Id: I7243b60938d6e9c7c2bc2aacdba5c667cca8ec9b
The aggregate.update_metadata.start and aggregate.update_metadata.end
notifications has been transformed to the versioned notification framework.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: Iab73b1351a50869620334f692aaefb566ba4ddf8
Implements: bp versioned-notification-transformation-rocky
The servergroup.addmember notification has been transformed
to the versioned notification framework.
Change-Id: I5291e453ba06f0f26c4928e4f7482c40e725a781
Implements: bp versioned-notification-transformation-rocky
The instance.resize_prep.start and instance.resize_prep.end
notifications are transformed to the versioned framework.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: If33d092e8688ed4d4864b2f7130104ab85ce1ea4
Implements: bp versioned-notification-transformation-queens
Nobuhiro MIKI