Hacking has bumped the version of flake8 that it's using to 5.0 in its
6.0.1 release. This turns up quite a few pep8 errors lurking in our
code. Fix them.
Needed-by: https://review.opendev.org/c/openstack/hacking/+/874516
Change-Id: I3b9c7f9f5de757f818ec358c992ffb0e5f3e310f
Adding IOError in list of catching exceptions in order to
fix behavior when nova-compute wouldn't retry image download
when got "Corrupt image download" error from glanceclient
and had num_retries config option set.
Closes-Bug: #1950657
Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0
If downloding of Glance image failed we should
close iterator of image body.
Otherwise Glance is unable to delete the image.
Change-Id: I193df2fcbf2588c10be953eb4e9eef4609b6286f
Closes-Bug: 1948706
This option was deprecated during Queens and can be removed now.
A single test is fully removed as it is no longer possible to hit the
usecase while an additional two tests are modified to test the recently
introduced rbd direct download usecase.
Change-Id: I5e8bbc078266d81d64c2073a828b6ff394e4f0e1
Replace six.text_type with str.
A subsequent patch will replace other six.text_type.
Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
Remove references to the feature from various comments scattered
throughout the code.
Change-Id: Ic353a2489389c557859c249218eaf6060974e1a9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This change allows compute hosts to quickly download and cache images on the
local compute host directly from Ceph rather than slow dowloads from the
Glance API.
New '[glance]/enable_rbd_download' option is introduced to enable this
behavior. This is slight change compared to the original idea described in
the relevant blueprint where it was discussed to use (now obsolete)
'[glance]/allowed_direct_url_schemes' option.
Additionally, when an image signature verification is requested, it should
be done also for the image fetched by the new download handler. This was
completely missing so far.
New '[glance]/rbd_{user,pool,ceph_conf,connect_timeout}' configurables
are introduced to allow operators to configure access to the cluster
hosting Glance without the need to use the existing '[libvirt]'
specific configurables. nova.storage.rbd_utils.RBDDriver has also been
modified to accept these but continues to default to the '[libvirt]'
specific configurables for now.
Change-Id: I3032bbe6bd2d6acc9ba0f0cac4d00ed4b4464ceb
Implements: blueprint nova-image-download-via-rbd
This should help the readability and also make it possible
to call new _verify_and_write method with different arguments.
Change-Id: I20913201cf945a7fde1f9e6264c415e1235db7b9
Execute glance upload in a native thread as it may block the current
coroutine until it completes.
Despite the fact we use eventlet monkey_patching [1] to achieve cooperative
yielding for network IO, file IO on busy file system may still get
nova-compute hanging.
Stick those IO in a native thread using eventlet tpool.execute() [2]
avoid this issue.
[1] https://eventlet.net/doc/patching.html
[2] https://eventlet.net/doc/threading.html
Closes-Bug: #1874032
Change-Id: I8dbc579e0037969aab4f2bb500fccfbde4190726
In Queens we deprecated the nova.image.download hook, which provided a
mechanism to inject custom code into the download path of our glance
module. This can now be removed.
The one known potential user of this is was a direct-rbd download
proposal, which needs further modification of the base glance client
in order to work. Thus, this removes the hook point, but not the
config option to allow specifying location schemes that should be
considered direct by the glance module. This provides a path forward
to integrate the direct-rbd code as a proper feature to our glance
module, which will mean un-deprecating that config option, but also
providing a stable list of potential options for it.
Change-Id: I7463af2ba9b74a73ffbb0a6b5fa12dff3fa5cac6
This just provides minimal support for calling the import API in Glance.
That API can do more things, but it is unlikely Nova would ever need to
call them, so this is rather opinionated and could be extended later if
needed.
Related to blueprint rbd-glance-multistore
Change-Id: Icf78fcabad8b966b6b5c289e1b660c01c928272d
These particular imports are no longer needed in a Python 3-only world.
Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This doesn't exist for 'nova.volume' and no longer exists for
'nova.network'. There's only one image backend we support, so do like
we've done elsewhere and just use 'nova.image.glance'.
Change-Id: I7ca7d8a92dfbc7c8d0ee2f9e660eabaa7e220e2a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
If snapshot creation via glance fails due to lack of space or over
quota, we want to have a clearer error message.
Change-Id: Ic9133f6bc14d4fe766d37a438bf52c33e89da768
Closes-Bug: #1613770
When an admin creates a snapshot of another project owners
instance, either via the createImage API directly, or via the
shelve or createBackup APIs, the admin project is the owner
of the image and the owner of the instance (in another project)
cannot "see" the image. This is a problem, for example, if an
admin shelves a tenant user's server and then the user tries to
unshelve the server because the user will not have access to
get the shelved snapshot image.
This change fixes the problem by leveraging the sharing feature [1]
in the v2 image API. When a snapshot is created where the request
context project_id does not match the owner of the instance project_id,
the instance owner project_id is granted sharing access to the image.
By default, this means the instance owner (tenant user) can get the
image directly via the image ID if they know it, but otherwise the image
is not listed for the user to avoid spamming their image listing. In the
case of unshelve, the end user does not need to know the image ID since
it is stored in the instance system_metadata. Regardless, the user could
accept the pending image membership if they want to see the snapshot
show up when listing available images.
Note that while the non-admin project has access to the snapshot
image, they cannot delete it. For example, if the user tries to
delete or unshelve a shelved offloaded server, nova will try to
delete the snapshot image which will fail and log a warning since
the user does not own the image (the admin does). However, the
delete/unshelve operations will not fail because the image cannot
be deleted, which is an acceptable trade-off.
Due to some very old legacy virt driver code which started in the
libvirt driver and was copied to several other drivers, several virt
drivers had to be modified to not overwrite the "visibility=shared"
image property by passing "is_public=False" when uploading the image
data. There was no point in the virt drivers setting is_public=False
since the API already controls that. It does mean, however, that
the bug fix is not really in effect until both the API and compute
service code has this fix.
A functional test is added which depends on tracking the owner/member
values in the _FakeImageService fixture. Impacted unit tests are
updated accordingly.
[1] https://developer.openstack.org/api-ref/image/v2/index.html#sharing
Change-Id: If53bc8fa8ab4a8a9072061af7afed53fc12c97a5
Closes-Bug: #1675791
openstack-dev was decomissioned this night in https://review.openstack.org/621258
Update openstack-dev to openstack-discuss
Change-Id: If51f5d5eb710e06216f6d6981a70d70b6b5783cc
In [0] the way parameters are passed to the glance client was changed.
Sadly one required argument was dropped during this, we need to insert
it again in order to fix e.g. rbd backend usage.
[0] https://review.openstack.org/614351
Change-Id: I5a4cfb3c9b8125eca4f6c9561d3023537e606a93
Closes-Bug: 1803717
When creating a snapshot of a server using the nova API, failure occurs
if the image contains the metadata property "version". This was due to
the way that the GlanceClientWrapper.call() function signature was
structured.
This patch forces all client positional args to be passed as a named
"args" argument to the call() function and all client named args to be
pass as a named "kwargs" argument to the call() function. This
eliminates any argument name-shadowing that previously caused issues.
Closes-bug: #1717547
Change-Id: I3ed3303309fe2a25c0043fd206f36bada4b3b8f9
Implement the certificate_utils module. The module's verify_certificate
method can be applied to the creation or rebuild of an instance. It is
triggered by one of two ways:
1) The enable_certificate_validation configuration option is set to
True in Nova's glance configuration (alongside the
verify_glance_signatures option also set to True)
2) A list of trusted certificate IDs is provided
Change-Id: I0ae2dbf66241207a425bf7d0fc02a4d2e2dea409
Implements: blueprint nova-validate-certificates
In certain configurations, like when setting [service_user]
config, and not setting [glance]/api_servers, the KSA adapter
get endpoint code (new in Queens) will return a versioned URL
which glanceclient doesn't handle (due to bug 1707995) so we
need to workaround that by parsing the URL to strip the version
from the endpoint URL we got from KSA.
This is validated in the nova-next CI job which configures a
service user token for glance.
Change-Id: I363182e916480c734cc37f279e8e89c8f3ec653c
Closes-Bug: #1747511
Related-Bug: #1707995
Glance image backends may return HTTPConflict in many cases
during deleting image, for example, an rbd image is used for
booting a new instance, or rbd image has snapshot(s).
If user have already backed up instance to few numbers and then
execute backup api with rotation 1, then nova will delete the
previously created images exceeding rotation limit.
During deleting these images, if the first one of the backup
images are deleted failed with HTTPConflict or other exceptions,
all images exceeding rotation limit will be left over.
This patch handles ImageDeleteConflict and all other exceptions
during deleting backup images, logs a message and continues
deleting all of the remaining images.
Closes-Bug: #1734838
Change-Id: Ie8091fe3e0e4275717ddc50166345f1c9df4b889
we abstract image (as well as volume etc) to image api and
volume api instead of glance and cinder direct usage, so
in most nova code we should use image.generate_image_url instead of
glance.generate_image_url. Thus we can utlize generic api of
image API instead of call glance client wrapper function directly.
Change-Id: I6ca9aeba78f7d939a3921a368d36612f8ede44da
refactor compute code to remove glance direct usage,
1) use image_api.get instead of get glance code then show the image
detail information
2) according to TODO, create a new api in image.api and use it
Change-Id: I0c8dee5d0d18adeaa83183da81c85f378ae5f8fd
Change the Adapter loading for glance to use the auth from the user
context instead of exposing and requiring it in the conf. With this
change, it is possible to leave the [glance] conf section empty and
still be able to discover the image API endpoint from the service
catalog.
Note that, when we do this, we often end up with the user auth being a
_ContextAuthPlugin, which doesn't conform to the characteristics of
keystoneauth1.identity.base.BaseIdentityPlugin as augmented in
keystoneauth1 3.1.0. This requires a series of workarounds until bug
1709118 is fixed. These, along with workarounds for bugs 1707993 and
1707995, are subsumed with this change set in a (hopefully temporary)
helper method nova.utils.get_endpoint.
This lays the foundation for other services that should use user
context for authentication - those via which Nova is acting on behalf
of the user, i.e. cinder, keystone, and (sometimes) neutron[1].
(Services such as placement and ironic (and sometimes neutron) should
continue to use admin auth context loaded from the conf.)
[1] bb4faf40df/nova/network/neutronv2/api.py (L149-L160)
Co-Authored-By: Eric Fried <efried@us.ibm.com>
Partial-Implements: bp use-ksa-adapter-for-endpoints
Change-Id: I4e755b9c66ec8bc3af0393e81cffd91c56064717
Operators omitting the protocol (http(s)) from their [glance]api_servers
value(s) have had long enough to fix it. Use it as is; and if the
protocol is omitted, let it fail hard in the request.
Change-Id: Ifebc9192e1e754180c97a3e40806c1c496a8b715
Provide a new method:
nova.utils.get_ksa_adapter(service_type, ks_auth=None, ks_session=None,
min_version=None, max_version=None))
...to configure a keystoneauth1 Adapter for a service. The Adapter, and
its component keystoneauth1 artifacts not passed into the method, are
loaded based on options in the conf group corresponding to the specified
service_type.
The ultimate goal is to replace the various disparate mechanisms used by
different services to do endpoint URL and version discovery. In Queens,
the original mechanisms will still take precedence, but (other than
[glance]api_servers - see the spec) will be deprecated. In Rocky, the
deprecated options will be removed.
This change incorporates the above utility into endpoint discovery for
glance and ironic. Future change sets will do the same for other
services (cinder, neutron, placement).
Change-Id: If625411f40be0ba642baeb02950f568f43673655
Partial-Implements: bp use-ksa-adapter-for-endpoints
Closes-Bug: #1707860
The only in-tree implementation of the nova.image.download.modules
extension point was nova.image.download.file which was removed in
Pike: I7687cc89545a7a8b295dd6535b4ccebc913a2e0e
At the time of that removal, there was an operators ML thread
asking if anyone was using this code, or the extension point,
and the answer was no (or no answer at all).
Since we have no in-tree implementation of this extension point
and the extension point itself is not maintained or documented,
and even the TransferBase base class was removed in Pike, we
should deprecate the extension point and the configuration option
associated with its use so that we can simplify our internal
glance API client code.
Note that the libvirt Rbd image backend which does support
direct_url / image locations configuration for fast clones is
still supported and unrelated to this code.
Change-Id: I13162ebc9050dd2c468d0f8b969b96409f60afa8
Recent changes [1][2] added fsync to the data file in
GlanceImageServiceV2.download. This raises EINVAL if the file is a
pipe/FIFO or socket [3].
This change set adds a static _safe_fsync method to GlanceImageServiceV2
which conditions the fsync call not to run if the file handle represents
a pipe/FIFO or socket, and uses that call from the download method.
[1] https://review.openstack.org/#/c/441246/
[2] https://review.openstack.org/#/c/443583/
[3] http://man7.org/linux/man-pages/man2/fsync.2.html#ERRORS
Change-Id: Ied5788deadcf3d1336a48288cf49d8571db23659
Closes-Bug: #1677047
In the last few cycles oslo.context has picked up a standard way of
representing most of the information available from a token context.
There's no more need for nova to manage these properties. Remove
properties that shadow the base oslo.context properties and helpers that
we don't need.
Change-Id: I1b11e405232b1acee053cb3bd30c18202d3b7c8f
The i18n team has decided not to translate the logs because it
seems like it not very useful; operators prefer to have them in
English so that they can search for those strings on the internet.
Partially fix on nova/conductor, nova/console, nova/consoleauth,
nova/db and nova/image other paths will be fixed on next commits
Change-Id: I132f60cb4743f85c96ec5231d6f860cfb0815758
Service token will be passed along with user token to communicate with
services when dealing with long running tasks like snapshots.
Change-Id: Ib61e045742fc98f5ff86f6aaab23ad7505c3ee07
Implements: blueprint use-service-tokens-pike
Migrate Glance client to Keystonesuth to make it
consistent with Cinder and Neutron clients.
Note: We still use the api_servers variable, rather than
fetching endpoints from the service catalog.
This change is needed to enable the use of service token
with Glance client.
Related to blueprint use-service-tokens-pike
Change-Id: I02be31bbe2de54c69210934b234ed380daca4fda
Co-Authored-By: Pushkar Umaranikar <pushkar.umaranikar@intel.com>
This changes the constructor so that glance calls will carry forward
the request_id to the glance service. ``global_id`` is a magic
property on new oslo.context which is either set to the
global_request_id sent into Nova, or the local request id if it's not
set.
Fix some unit tests to handle this new parameter when mocking
glanceclient calls.
oslo spec I65de8261746b25d45e105394f4eeb95b9cb3bd42
Change-Id: I16f9dda3c904c4a2578fa6a691fed646a41f6793
This change removes the signature_utils module
from Nova and uses the cursive library, which
contains an identical module.
Change-Id: I8179282a9d19f829aca0b5bd2775d855b3364c86
Depends-On: I7e5797661fee258bc0270b5f109704b591633519
Implements: blueprint signature-code-cleanup
Partial-Bug: #1528349
fdatasync is not available on Windows, for which reason this
operation now fails.
This change switches to fsync, which is platform independent.
Closes-Bug: #1671435
Change-Id: I71ac19160f198a97bb1834c5c81684758f129bcc
Image download closes the filehandle of the downloaded image before
use, but doesn't fdatasync it. This means that in the event of a host
crash we may be left with only the file metadata when the host comes
back up: i.e. an empty file of the correct size. Nova cannot detect
this data corruption, so operator intervention is required.
By adding an fdatasync() before closing the file handle we ensure that
the downloaded file is either entirely present, or entirely not
present.
See also change I33bd99b0, which fixes this issue for downloads
requiring a subsequent conversion step.
Partial-Bug: #1669844
Change-Id: Id9905a87f16f66530623800e33e2581c555ae81d
This class isn't used, referenced, or tested anywhere.
GlanceImageServiceV2 is now used everywhere instead.
Change-Id: Ib502b0b2ecbbdad9d34b9f827b4b21228d4fd79a
Artom Lifshitz