When updating legacy images, currenly we use the user's
context and elevate priviledges. However, we do not
require admin priviledges for the cinder API calls.
This patch removes the special case where we elevate
priviledges as it wasn't doing anything rather avoiding
us to use right credentials and failing to fetch volume
in the right location because of wrong credentials.
The correct credentials are either the service ones
set in glance-api.conf file or the user context
credentials, using which the Image-Volume was created.
NOTE: When using cinder as glance backend and we want
to perform optimized volume upload to image, one thing
we should make sure is either using the context or the
cinder credentials set in glance-api.conf file, it should
match the following details on the cinder side (if we are
using internal context to create clone of image-volumes):
cinder_store_user_name = context.user_id/cinder_internal_tenant_user_id
cinder_store_project_name = context.project_id/cinder_internal_tenant_project_id
The cinder_internal_tenant_user_id and
cinder_internal_tenant_project_id are set in the
[DEFAULT] section of cinder.conf.
This issue was first discovered when testing the new
location APIs[1] where tempest creates a volume with
cinder's internal context and glance uses wrong (user context)
credentials to access it and failing with 404 not found.
[1] https://review.opendev.org/c/openstack/cinder/+/909847
Closes-Bug: #2056179
Change-Id: I4f27a9800f239da8dbf29f4c028678df1f867664
The previous attempt a5ba027922 was
incomplete and there are still a few more logs where access keys are
logged. This fixes these to avoid leaking access keys to log.
Related-Bug: #2047688
Change-Id: I8dc564bed33d6fc71965f4f573ae9109b410b1d4
This imports the existing rados_connect_timeout option handling in
cinder so that users can use a different timeouts for Glance
specifically, instead of relying on the global ceph.conf options.
This parameter was initially deprecated in Zed release by [1] and has
had no effect since then, but this change restores the parameter with
the logic to override the timeout value in RADOS client.
[1] b1d0feeba4
Related-Bug: #1983499
Change-Id: Ib370f527c06dc85bcfd9df6ca1efb2a4e8cb5e7d
Resolves a bug introduced in
c43f19e845
This issue is only in evidence when glance is behind a proxy where the
client buffer size can be lower (for haproxy: bufsize = 16384) which
can cause unaligned reads
(https://github.com/openstack/glance/blob/master/glance/common/wsgi.py#L1028).
The response length can be bigger than the store_chunk_size for the
first time, so at the end the RBD write will fail because it wants
to write more data than the actual RBD image size after the first
resize.
Thanks to Robert Varjasi for investigating this issue!
Fixes-Bug: 1916482
Change-Id: Ie03693c2cb8b096978fb156231c3b1cab695470f
Cinder store requires a few additional dependencies.
- cinderclient
- os.brick
- oslo.provsep
Currently the driver handles missing cinderclient, but it does not
properly treat in case one of the other two libraries is unavailable.
Note that oslo.rootwrap is also part of the additional dependencies but
this library is not directly imported. So we do not implement check
at store initialization.
Closes-Bug: #2043583
Change-Id: I4ea0a9749fdd821f3f958f77a2923b0fafefa471
The vmwareapi virt driver in nova was marked as experimental some
cycles ago due to lack of CI and maintainers[1]. This deprecates
the VMWare Datastore so that we can remove it when nova removes
the virt driver. The store driver is used along with the vmwareapi virt
driver and we don't expect any usage of this store driver with
the other virt drivers such as libvirt driver.
[1] 713ef7839670bd5d73af52c211a14932adc933ba
Change-Id: I57fc8cbd6308c4a82a638a2d1e997605736aedc4
RBD snapshots automatically end up in the trash
when deleted when using RBD image format v2.
However, they prevent images from being deleted.
Instead, move RBD images to the trash in this
situation. This trash must be purged by a scheduled
rbd trash purge operation outside of Glance.
This passes new cinder<->image dependency tests in
I5fee2395195 when using ceph require-min-compat-client
of "mimic", which enables clone v2.
Closes-Bug: #1959186
Change-Id: I34dcd90a09d43127ff2e8b477750c70f3cc01113
librbd methods call lower level C code which runs in native code
and isn't aware about the eventlet threads hence hangs the eventlet
loop until the native code is finished.
This could cause problems when we are creating multiple images
with large size where one call to librados can cause the process to
hang and other operations can starve for execution and error out.
This patch wraps each RBD call in it's own native thread that won't
affect other RBD call from executing.
Note that since glance can run under eventlet *or* native threads
with something like uwsgi, this only delegates to an eventlet worker
when we are running in that environment.
Note: This was originally merged as commit
27ab8a6aeb, but without the handling
for non-eventlet cases.
Change-Id: I87dd4a129113e6309bee1b35a7a45b2a04353b04
Co-Authored-By: Dan Smith <dms@danplanet.com>
This reverts commit 27ab8a6aeb.
Reason for revert: The change introduced a blocking library call for
the thread running import jobs.
Closes-Bug: #2020122
Change-Id: I83e3a0da9fdf7167cf4adf7257d5960338901158
In order to be sure that devices are being removed from the host,
we should be using the 'force' parameter with os-brick's
disconnect_volume() method.
Closes-bug: #2004555
Change-Id: I63d09ad9ef465bc154c85a9ea125449c039d1b90
librbd methods call lower level C code which runs in native thread
and isn't aware about the eventlet threads hence hangs the eventlet loop
until the native thread is executed.
This could cause problems when we are creating multiple images
with large size where one call to librados can cause the process to
hang and other operations can starve for execution and error out.
This patch wraps each RBD call in it's own native thread that won't
affect other RBD call from executing.
Change-Id: I8efb0460df9fcba050b5ce949eb10caea325c851
Currently boto3 is not part of requirements but stevedore always tries
to import it and shows error in case boto3 is missing. This is not
a real error unless users actually enable s3 backends and can be quite
confusing.
This makes the driver code ignore ImportError and actually fail only
if users try to enable s3 backend without installing boto3.
Closes-Bug: #2007924
Change-Id: Ia94dd1d12a3d723f6263bdfb0966d416dfbae1af
This ensures glance_store checks whether the dependent libraries (rbd
and rados) when adding rbd store and disables all rados stores in case
these libraries are not available.
This avoid ugly error (eg. AttributeError) caused by missing libraries.
Change-Id: I673473aa8b540c084ce3a4934e8be7069f13e77f
Change Iec6ac7020f66f9f9728d6cf4452127b8d55a8b2d made cinderclient
optional but it did not add proper validation while initializing
a cinder store. This can cause ugly errors in case users tries to
enable cinder stores without installing cinderclient.
This ensures glance_store checks whether cinderclient is available and
disables all cinder stores in case cinderclient is not available.
Change-Id: Idb8abfdab9639b5b70ddddc3a76b1f1d24dd035d
move attachment_update to try block, so when there is some error calling
attachment_update the finally block will be called, then the attachment created could be deleted.
Closes-bug: #1983238
Change-Id: I2dc3888a56d802424c6d62a656b2e5fef9dabb3e
While creating an image, if we want to extend the volume
(to accomodate the image), we need to first detach the volume,
perform the extend and attach it again. This is inefficient
for backends that support extending attached volumes since we
are performing 3 cinder operations (the attachment call
includes 3 API calls which sum to a total of 5 API calls for 3
operations) instead of directly extending it which requires only
1 API call to cinder.
The support for extending attached volumes was added in cinder
microversion 3.42.
This patch adds a new boolean config option
``cinder_do_extend_attached`` which allows operators to specify
if the cinder backend they are using supports extending
attached (in-use) volumes. By default this will be ``false``.
Based on the parameter, we will perform the extend operation,
online (if cinder_do_extend_attached is true) and offline otherwise.
Spec: https://review.opendev.org/c/openstack/glance-specs/+/868901
Depends-On: https://review.opendev.org/c/openstack/glance/+/869021
Depends-On: https://review.opendev.org/c/openstack/cinder/+/869051
Change-Id: I5e70824e9abc5277ea25ba95704b358fe3686037
This is an effort to decouple some of the cinder backend specific
code (like nfs, scaleio) from the generic logic.
The purpose is to make the code modular and any change
for a particular cinder backend should not affect the code path
of other backends thereby reducing regression.
This is also required for another use case of supporting extend
of attached volumes added in [1].
Following are the major changes done in this patch:
1) Move cinder store to a new directory 'cinder' and rename
'cinder.py' to 'store.py' (similar to swift)
2) Create new files for nfs and scaleio backends for moving code
specific to these backends into their own separate file.
This also fixes one bug when using sparse files in nfs and
we wait for file size to be equal to volume size (initially done
for scaleio/powerflex backend) but this will never happen for
nfs sparse files. See bug: 2000584
3) Move cinder tests to 'tests/unit/cinder' directory and add tests
for base, nfs and scaleio files.
4) Modify/fix existing tests
Closes-Bug: #2000584
[1] https://review.opendev.org/c/openstack/glance_store/+/868742
Depends-On: https://review.opendev.org/c/openstack/glance/+/869021
Change-Id: I26c272b6c503e98fbbafca411d3eec47283bd6fc
Add an option to let the operator choose a proper region_name.
Some operator are not using S3 from amazon but from other provider.
In such case, the get_s3_location cannot give good region_name.
Adding an option from config seems a better option that relying on
hardcoded stuff in the code.
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I71352e75f6415494709d8fb62c8b13f3ea7c6016
Change I59b1579dc9877668b82d4195431c1 added tests that mock an
input buffer, but the mocked return_value for __len__ exceeds the
allowable maximum value for an index in the CPython implementation
on 32-bit systems. Fix this by using half the value and allowing
the buffer to be called twice during the tests.
Change-Id: Iaba20c01fb4d994cd738a570197f8f6f04f22606
Closes-bug: #1991406
The *_domain_id parmaeters should not have any default. Otherwise
keystoneauth ignores the *_domain_name parameters and it requires
only *_domain_id parameters are used.
Closes-Bug: #1620999
Change-Id: I1f8c9184761313f9fc5fda2f257e52233e0196d1
when fetching images with http driver, the redirect URL can have
mandatory query in it, which must be kept intact to successfully
fetch the image.
Change-Id: I2a9d4d026b935ea6c5e5a3a46c86f70ce1e39ae7
Closes-Bug: #1633860
When debugging issues related to glance cinder store, there are
several calls to cinder and it becomes hard to determine which step
we are currently executing without going through cinder logs.
This patch adds some useful debug logs for the new attachment's code
to understand which stage of attachment we are on.
Change-Id: I491b7292a511c47c1d6148dab69ae04269e50c85
As a new feature relevant for the Cinder store, os-brick now supports
setting the location of file locks in a different location from the
locks of the service.
The functionality is intended for HCI deployments and hosts that are
running Cinder and Glance using Cinder backend. In those scenarios the
service can use a service specific location for its file locks while
only sharing the location of os-brick with the other services.
To leverage this functionality the new os-brick code is needed and
method ``os_brick.setup`` needs to be called once the service
configuration options have been loaded.
The default value of the os-brick ``lock_path`` is the one set in
``oslo_concurrency``.
This patch adds support for this new feature in a backward compatible
way so code works even if using an os-brick version that doesn't have
this feature. That is the case for many CI jobs that install os-brick
from PyPi instead of the code present in master.
Change-Id: Ib11d63e3c388e12f145f40247d17030a566b8c5e
Python 2 has been deprecated for almost two years, and has not been
guaranteed to work with glance_store for a while. This patch removes all
traces of six, unicode strings and Python 2 tweaks.
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: Ifa78924d7ecf4f2d9a54c677888ab2926530c487
Rajat Dhasmana