The *_domain_id parmaeters should not have any default. Otherwise
keystoneauth ignores the *_domain_name parameters and it requires
only *_domain_id parameters are used.
Closes-Bug: #1620999
Change-Id: I1f8c9184761313f9fc5fda2f257e52233e0196d1
Python 2 has been deprecated for almost two years, and has not been
guaranteed to work with glance_store for a while. This patch removes all
traces of six, unicode strings and Python 2 tweaks.
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: Ifa78924d7ecf4f2d9a54c677888ab2926530c487
In SingleTenant authv3 context, connection manager does not evaluate
swift_store_endpoint and always takes endpoint from catalog.
The change ensures CONF.glance_store.swift_store_endpoint
will take over catalog value also in that case.
Closes-Bug: #1885651
Change-Id: Ib18ff19cd539e0117909f849672036b8c9e5f049
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() has been added to
oslo_utils. See https://review.opendev.org/#/c/750031/
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
It looks like the uses of the md5 are primarily for checksums and
generation of etags.
With this patch, all the unit and functional tests appear to pass
on a FIPS enabled system.
Change-Id: I0603ba217d6dc19f5c9f73c60c7b365efd28d30b
Depends-On: https://review.opendev.org/#/c/760160
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I3e92b23ab2a335b378f156c0456fb1d52706ed12
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I99b8b24f714858f6b289e5e7b5976e519bb81c11
When the image is size larger than chunk size, swift store
uploads 0 bytes for swift_buffered_upload enabled.
Closes-Bug: #1863691
Change-Id: I272d380d8fc093e946ecd6bd5ba99b5bbe2409b2
pass configured cacert for Swift to the keystoneauth's Session
as well so that the swift endpoint can be resolved from the catalog
when a custom CA bundle is used.
Change-Id: I439f6b5af34c685f72c9b4933c7eb0c77cc92e14
Closes-Bug: #1820817
While testing glance with Ceph Rados Gateway using latest Ceph release
(Nautilus), i've found that glance fails to upload the manifest using
dynamic large objects mode because of the value used in ETag request.
This issue has been reported to Ceph as it seems related to some recent
change in radosgw code [1].
However, checking at the upload workflow used by glance and comparing
to Swift documentation [2], I wonder if adding the etag is actually
providing any value. In the Swift the ETag header is used to validate
integrity when uploading chunks, not the manifest while glance is doing
exactly the oposite, not sending the etag in the chunks (I guess to
avoid checksuming big images, which makes sense to me) and sending it
when puting the manifest.
This patch is removing the etag header when sending the PUT request for
the manifest in chunked uploads.
[1] https://tracker.ceph.com/issues/39160
[2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects
Closes-bug: #1824533
Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808
As of version 3.5.0 moxstub will be deprecated, so remove it where it has
been used.
Change-Id: I2622c457871815311241d2eea562d7a3c70b0795
Signed-off-by: Chuck Short <chucks@redhat.com>
As of 3.7, the configparser module will not allow defaults values to be None.
This patch replaces such values with "default".
Change-Id: Id5a414412cd66d479fb8f8784cba5deddc628dfd
Closes-Bug: #1785641
Adds the ability to compute a "multihash" (see the Glance spec
for what this is exactly). To maintain backward compatability,
a new store_add_to_backend_with_multihash function is added.
Backward compatability for each store's add() method is achieved
by a back_compat_add wrapper.
Co-Authored-by: Scott McClymont <scott.mcclymont@verizonwireless.com>
Co-Authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: I063d0900b7dc7e0d94dfb685971eb9b17ed67c7b
Partially-implements: blueprint multihash
During segmented (chunked) uploads of large files to Swift, if a
single segment fails to upload, Glance will abort the entire
upload, which could have been in progress for many hours.
However, if seek() and tell() methods are provided on the input
file stream, swiftclient will attempt to reset the filestream
back to the beginning of the segment and retry.
This patch adds glance_store._drivers.swift.buffered.BufferedReader to
provide this reset capability. It works by buffering the segment/chunk
to disk in case an upload error occurs and a reset is required. This
could potentially use much larger disk space, so a CONF setting,
CONF.glance_store.swift_store_reader_class is required to
enable the new reader. (The default is to use the existing
glance_store._drivers.swift.store.ChunkReader.)
This patch does not address the automatic revert to ChunkReader if
we run out of space for buffering.
Co-Authored-By: Brian Elliott <bdelliott@gmail.com>
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
DocImpact
Partially Implements: blueprint buffered-reader-for-swift-driver
Change-Id: I7d7337cee930fd2fb451fa5c7093c5fa9f985dcb
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I225da4f95187387007df11c91047497a2a93e862
The swift backend did not make use of the insecure option in
the config when creating a Keystone session, enable or disable
verification based on it.
Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Change-Id: Ic783afde7ae8af522480996fdf91ed54e02e72d2
Closes-Bug: #1606268
When Swift multi-tenant store is used along with the
scheme ``swift+config`` (the scheme that identifies the
need to use the swift config), a reference to use swift
objects from the swift config file set using the option
``swift_store_config_file`` is made resulting in the
storage url for an accessible object being created from
a swift auth_address. So is the case when the scheme is
``swift``. This behavior is suitable for single-tenant
swift store and not multi-tenant store.
The solution is to ensure that if deploying multi-tenant swift
store, prohibit setting of swift_store_config_file.
This patch does this by ensuring that multi-tenant and
swift_store_config_file are not configured to work together.
TODO: When somebody changes to multi-tenant from single-tenant store,
and they were using the swift config for single-tenant, the operator will
not be able to access those images any longer. Logic to ensure that
multi-tenant uses the swift conf file if there is a need to access
an image that was created using single-tenant and swift+config could be
introduced.
UpgradeImpact
DocImpact
Closes-Bug: 1625075
Change-Id: I8b5c31541d3c501ad7c2520b463f881599f4a28e
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
When using swift multi-tenant store, using a user's service
catalog to obtain the storage url for an object does not work
if the image was not created by the same user. This is because
the user's context, if used to establish a connection, would only
be looking at a location url formed with the wrong owner information
resulting in an object GET failure (404).
This patch solves this issue by a 'revert in principle' to the old code
logic where the storage_url for an image when using multi-tenant store
was fetched from the database via 'location.swift_url'. The change
that had introduced the fetch of storage_url from user's context is [1].
[1] I7bc23dfc11900b55f45fe98144d14f883c381c9f
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
Partial-Bug: #1625075
Change-Id: I8c22a0ab566cf1ec99cbee78ea43ef28abcd8dc0
A direct conversion of keystoneclient usage to the newer supported
keystoneauth library. The libraries are largely compatible and there
should be no issues swapping between them.
This doesn't fix any problems of the way auth is used, it just changes
out the library.
Change-Id: Ibe212e17150a3c750e9c2536a4c869d87e9d4e13
get_manager_for_store is a weird function that switches based on the
class type of the first parameter. This is an odd throw back to pre
object orientated days where the object defines what it wants. Refactor
it to put the class on the object.
Change-Id: I0bca2607267aef3bda720cdfbbbe0e5a8093a20d
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
http://docs.openstack.org/developer/hacking/#imports
Change-Id: I7525672d64a3bc5df2dc6bdbf19da5b1d6c9dc1e
assertEqual expects that the arguments provided to it should be
(expected, observed). If a particluar order is kept as a convention,
then it helps to provide a cleaner message to the developer if Unit
Tests fail. The following patch fixes this issue.
TrivialFix
Change-Id: I07b78383ef38731140143c91ae3a902bea55eebb
Closes-Bug: #1259292
In particular, stop mocking out some internal details like
swiftclient.client.Connection._retry
Change-Id: If118b9656ae1e87139c0d48dcc97c1daee948a85
Related-Change: I62cc037493af87373a75e37e2d9f33b8aedf9889
Related-Bug: 1556059
Previously we passed all user configurations to swift.
Swift uses keystoneclient when authenticating and some
parameters were defaulting by swiftclient. Swiftclient
did not use sessions and auth plugins. When we implemented
re-authentication we started to use sessions and auth plugins
but it turned out that some parameters are not defaulting here:
namely, project_domain_name and project_domain_id.
glance_store is also not defaulting them (we need to introduce
configurations params for that in glance_store) so as a result
we have error from keystone service.
The patch introduces default domains (as keystoneclient
without sessions does). The long term fix (introduce
configuration parameters) will be proposed later.
Change-Id: I90665dabaab13a9c9d5f00770496f8ccfccba024
Closes-Bug: #1561947
swiftclient changed their code so if the chuck size is greater than
zero it will return a generator otherwise it will return the response.
Which would be the object you are looking for. This causes our tests
to fail since they are always expecting the response instead of a
generator.
Closes-bug: 1556059
Co-Authored-by: avarner <avarner@us.ibm.com>
Change-Id: I62cc037493af87373a75e37e2d9f33b8aedf9889
Previously, we introduced a re-auth approach for swift store.
In case of single-tenant store we stated that it works with
Keystone v3 only when re-authenticating but we also need to make
the same restriction when receiving swift endpoint from keystone
becase it breaks some installations where auth_url contains
trailing slash.
So now if auth_version is not v3 then we don't search endpoint url
from Keystone in this case and simply return None. It is safe
because we don't do any re-authentication for v1 or v2 and use
old methods for that version.
Change-Id: Id8dab9ed74eef56ffa4937bf29f96888b673ad64
Closes-Bug: #1552132
Enable re-authentication when downloading or uploading images.
If single tenant store is used then request the new token
for service user.
If multi tenant store is used then request the new token with
trusts.
Note: Both features are available for Keystone V3 API only.
If store.auth_version is not '3' then use old approach to
receive Swift Connections.
DocImpact: Describe how to enable/disable re-authentication
and add notes about Keystone v3 support only.
Implements bp prevention-of-401-in-swift-driver
Change-Id: Id4e479e29ae8f71ff93f769246989b4b180f5c68
The patch defines implementation of swift connection manager for
swift driver. It allows to receive swift connections and
update them if user token is going to expire soon.
Connection manager for single tenant store uses swift service
user credentials to receive new token.
Connection manager for multi-tenant store uses trusts to
receive new token and initialize a connection.
Please note that this is first part of bp implementation that
defines framework and helpers for re-authentication.
Implementation of keystoneclient initialization and enabling of
re-authentication for swift store available in next patch.
Implements bp prevention-of-401-in-swift-driver
Change-Id: I61b0fcfe284bdfbf4c0558178318c69617ec6127
In the parent patch, the swift driver is supported for images that
are greater than the 'large_object_size.' This patch adds support
for images that are less than the 'large_object_size.'
Change-Id: Ifaa15b794a7213c1c393288631d071ac66dcbc81
Partial-Bug: #1516031
Multi-tenant store doesn't need to fetch swift storage_url from
store location because it can fetch it from service catalog
in user context.
That allows to avoid subtle errors when uploading/downloading
with scheme = 'swift+config'. In the current implementation
multi-tenant store uses location.swift_url to request swift
endpoint but swift_url is keystone endpoint in that case.
The change provides possibility to get rid of this dependency
and consider context as primary source for swift store url.
Closes-Bug: #1511025
Change-Id: I7bc23dfc11900b55f45fe98144d14f883c381c9f
glance_store passes some unused parameters such as tenant, auth_version
and others to Multi-tenant store when initializing Swift connection.
The patch removes these parameters from the Connection __init__
method because glance_store doesn't need to pass them - it passes
a valid auth_token to Swift connection (so it doesn't need to
initialize Keystone client and request the token).
Change-Id: I3805f2bb31b7c75d222867aaefefa0f5c0be3847
Takashi Natsume