Python 2 has been deprecated for almost two years, and has not been
guaranteed to work with glance_store for a while. This patch removes all
traces of six, unicode strings and Python 2 tweaks.
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: Ifa78924d7ecf4f2d9a54c677888ab2926530c487
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() has been added to
oslo_utils. See https://review.opendev.org/#/c/750031/
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
It looks like the uses of the md5 are primarily for checksums and
generation of etags.
With this patch, all the unit and functional tests appear to pass
on a FIPS enabled system.
Change-Id: I0603ba217d6dc19f5c9f73c60c7b365efd28d30b
Depends-On: https://review.opendev.org/#/c/760160
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I99b8b24f714858f6b289e5e7b5976e519bb81c11
pass configured cacert for Swift to the keystoneauth's Session
as well so that the swift endpoint can be resolved from the catalog
when a custom CA bundle is used.
Change-Id: I439f6b5af34c685f72c9b4933c7eb0c77cc92e14
Closes-Bug: #1820817
There is a small problem in _option_get function, it has problem
dealing with boolean parameters with False value, such as 0, False.
It will rasie exception for missing options.
Fix the issue by changing the condition from "not result" to
"result is None" which is true indication for missing options.
Change-Id: I313aa9f8d583c9cf44b052685b0f26824d2b6ade
As a part of vocabulary correction, changed the location
metadata key name from 'backend' to 'store'. Modified
corresponding tests as well.
Change-Id: I1a8056a316fcfae8b4b32f74fb62f90eaceb9467
bp:multi-store-vocabulary-correction
As a part of vocabulary correction, changed the location
metadata key name from 'backend' to 'store'. Modified
corresponding tests as well.
bp:multi-store-vocabulary-correction
Change-Id: Ida27628c54607b43e76389029f1f6f78ccc49f93
For updating location metadata with store information to the images
which are existed prior to enabling multiple stores, added new
public attribute 'url_prefix' to each of the store instance.
Implements: blueprint location-uri-prefix
Change-Id: Icd760d30e947867c2b5b87f86bbe4b1a4240d214
While testing glance with Ceph Rados Gateway using latest Ceph release
(Nautilus), i've found that glance fails to upload the manifest using
dynamic large objects mode because of the value used in ETag request.
This issue has been reported to Ceph as it seems related to some recent
change in radosgw code [1].
However, checking at the upload workflow used by glance and comparing
to Swift documentation [2], I wonder if adding the etag is actually
providing any value. In the Swift the ETag header is used to validate
integrity when uploading chunks, not the manifest while glance is doing
exactly the oposite, not sending the etag in the chunks (I guess to
avoid checksuming big images, which makes sense to me) and sending it
when puting the manifest.
This patch is removing the etag header when sending the PUT request for
the manifest in chunked uploads.
[1] https://tracker.ceph.com/issues/39160
[2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects
Closes-bug: #1824533
Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808
During large file uploads under py3, we are occasionally seeing a
"unicode objects must be encoded before hashing" error even though
we are reading from a byte stream. From what I can tell, it looks
like it's happening when a zero-byte read is requested, so we handle
that case explicitly. This is a band-aid fix; we still need to track
down the source.
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Related-bug: #1805332
Change-Id: Ia7653f9fcbe902abc203c10c80ab44a641a4d8f9
Config option help is used to generate sample config files. This
is done statically and only in English. Translation is done
dynamically at runtime by loading the translation catalogs. So
putting the _() translation around config option help creates
unnecessary work for the translators as this will never be loaded
and translated.
This removes translation markers from config option definitions.
Change-Id: I0597a5b69ef4e179324885cb35bbb72fceb2b830
Adds the ability to compute a "multihash" (see the Glance spec
for what this is exactly). To maintain backward compatability,
a new store_add_to_backend_with_multihash function is added.
Backward compatability for each store's add() method is achieved
by a back_compat_add wrapper.
Co-Authored-by: Scott McClymont <scott.mcclymont@verizonwireless.com>
Co-Authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: I063d0900b7dc7e0d94dfb685971eb9b17ed67c7b
Partially-implements: blueprint multihash
Added multi store support for http, swift, sheepdog and vmware driver.
The default behavior is maintained for backward compatibility.
DocImpact
Partial-Implements: bp multi-store
Change-Id: I93ccdafc6e740065ff4ca3adc6b49eb82e8afa10
Currently we use asserts for uri validation, which is not good
practice because assert will be deleted from optimized code.
We must use exceptions in such cases.
Co-authored-by: kairat_kushaev <kkushaev@mirantis.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: I89c5f1b74be89c759d5754d6cab54dc86c946be1
During segmented (chunked) uploads of large files to Swift, if a
single segment fails to upload, Glance will abort the entire
upload, which could have been in progress for many hours.
However, if seek() and tell() methods are provided on the input
file stream, swiftclient will attempt to reset the filestream
back to the beginning of the segment and retry.
This patch adds glance_store._drivers.swift.buffered.BufferedReader to
provide this reset capability. It works by buffering the segment/chunk
to disk in case an upload error occurs and a reset is required. This
could potentially use much larger disk space, so a CONF setting,
CONF.glance_store.swift_store_reader_class is required to
enable the new reader. (The default is to use the existing
glance_store._drivers.swift.store.ChunkReader.)
This patch does not address the automatic revert to ChunkReader if
we run out of space for buffering.
Co-Authored-By: Brian Elliott <bdelliott@gmail.com>
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
DocImpact
Partially Implements: blueprint buffered-reader-for-swift-driver
Change-Id: I7d7337cee930fd2fb451fa5c7093c5fa9f985dcb
The swift backend did not make use of the insecure option in
the config when creating a Keystone session, enable or disable
verification based on it.
Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Change-Id: Ic783afde7ae8af522480996fdf91ed54e02e72d2
Closes-Bug: #1606268
When Swift multi-tenant store is used along with the
scheme ``swift+config`` (the scheme that identifies the
need to use the swift config), a reference to use swift
objects from the swift config file set using the option
``swift_store_config_file`` is made resulting in the
storage url for an accessible object being created from
a swift auth_address. So is the case when the scheme is
``swift``. This behavior is suitable for single-tenant
swift store and not multi-tenant store.
The solution is to ensure that if deploying multi-tenant swift
store, prohibit setting of swift_store_config_file.
This patch does this by ensuring that multi-tenant and
swift_store_config_file are not configured to work together.
TODO: When somebody changes to multi-tenant from single-tenant store,
and they were using the swift config for single-tenant, the operator will
not be able to access those images any longer. Logic to ensure that
multi-tenant uses the swift conf file if there is a need to access
an image that was created using single-tenant and swift+config could be
introduced.
UpgradeImpact
DocImpact
Closes-Bug: 1625075
Change-Id: I8b5c31541d3c501ad7c2520b463f881599f4a28e
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
When using swift multi-tenant store, using a user's service
catalog to obtain the storage url for an object does not work
if the image was not created by the same user. This is because
the user's context, if used to establish a connection, would only
be looking at a location url formed with the wrong owner information
resulting in an object GET failure (404).
This patch solves this issue by a 'revert in principle' to the old code
logic where the storage_url for an image when using multi-tenant store
was fetched from the database via 'location.swift_url'. The change
that had introduced the fetch of storage_url from user's context is [1].
[1] I7bc23dfc11900b55f45fe98144d14f883c381c9f
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
Partial-Bug: #1625075
Change-Id: I8c22a0ab566cf1ec99cbee78ea43ef28abcd8dc0
A direct conversion of keystoneclient usage to the newer supported
keystoneauth library. The libraries are largely compatible and there
should be no issues swapping between them.
This doesn't fix any problems of the way auth is used, it just changes
out the library.
Change-Id: Ibe212e17150a3c750e9c2536a4c869d87e9d4e13
Some enhancement for debug message that do not
confuse people who debugging glance.
Closes-bug: #1631995
Change-Id: I425cb28cdc95c6a62bc7e65dc91105c5d2418952
get_manager_for_store is a weird function that switches based on the
class type of the first parameter. This is an odd throw back to pre
object orientated days where the object defines what it wants. Refactor
it to put the class on the object.
Change-Id: I0bca2607267aef3bda720cdfbbbe0e5a8093a20d
Adding improved help text for swift_store_region, swift_store_endpoint
and swift_store_service_type
Partial-Bug: #1570946
Change-Id: I81cf3502c88dcaddb91ca8c20bd43e9c7790a72f
In the first few improvements to the help text of configuration
options, the section "Services which consume this" was included
to clearly indicate which services consume a particular
configuration option. However, this is redundant information as
all glance services have their configuration files. And, the very
fact that a configuration option appears in a certain file
indicates that the corresponding service potentially consumes it.
Change-Id: I1dbaf620d5d9e7025a491b28546be66bfa4442c3
Partial-Bug: #1570946
Enable re-authentication when downloading or uploading images.
If single tenant store is used then request the new token
for service user.
If multi tenant store is used then request the new token with
trusts.
Note: Both features are available for Keystone V3 API only.
If store.auth_version is not '3' then use old approach to
receive Swift Connections.
DocImpact: Describe how to enable/disable re-authentication
and add notes about Keystone v3 support only.
Implements bp prevention-of-401-in-swift-driver
Change-Id: Id4e479e29ae8f71ff93f769246989b4b180f5c68
The patch defines implementation of swift connection manager for
swift driver. It allows to receive swift connections and
update them if user token is going to expire soon.
Connection manager for single tenant store uses swift service
user credentials to receive new token.
Connection manager for multi-tenant store uses trusts to
receive new token and initialize a connection.
Please note that this is first part of bp implementation that
defines framework and helpers for re-authentication.
Implementation of keystoneclient initialization and enabling of
re-authentication for swift store available in next patch.
Implements bp prevention-of-401-in-swift-driver
Change-Id: I61b0fcfe284bdfbf4c0558178318c69617ec6127
liyou01