test_cooperative_reader_of_iterator_stop_iteration_err passes an empty
list to utils.CooperativeReader since "[l * 3 for l in '']" evaluates to
an empty list. The
test_cooperative_reader_unbounded_read_on_empty_iterator also
initializes utils.CooperativeReader this way.
The function's docstring is a copy/paste of
test_cooperative_reader_of_iterator's. Judging by the method's name, it
seems its goal was to make sure the StopIteration exception was properly
handled in CooperativeReader.read(), which is already tested by the
following methods:
- test_cooperative_reader_of_iterator
- test_cooperative_reader_on_iterator_with_buffer
- test_cooperative_reader_unbounded_read_on_iterator
- test_cooperative_reader_preserves_size_chunk_equals_read
- test_cooperative_reader_preserves_size_chunk_less_then_read
- test_cooperative_reader_preserves_size_chunk_more_then_read
- test_cooperative_reader_unbounded_read_on_empty_iterator
The test_cooperative_reader_of_iterator_stop_iteration_err therefore
seems useless and is removed in this commit.
Change-Id: I28834aab2602f59cbfa3ba061ab245af7ac56c40
This patch updates the location URL of the legacy images while
upgrading from single cinder store to multiple stores.
It does that with the help of lazy loading logic i.e. while
GET images call, it checks the location URL and metadata
of the image against the configured store ids and updates
images to respective stores on the basis of volume type (comparing
image-volume's type with the configured cinder_volume_type).
Legacy image URL:
cinder://<volume-id>
New image URL:
cinder://<store-id>/<volume-id>
NOTE: bumping lower-constraints/requirements of glance-store to 2.3.0 as
it includes changes[1] that are a hard requirement for cinder multiple
stores to work with glance
[1] https://review.opendev.org/#/c/746556/
Change-Id: I087a89c20813378fea8ff22ddf81d7a10c220db3
Implements: blueprint multiple-cinder-backend-support
Glance has a facility lazy loading for legacy images which will be called
on get/list api calls to add store information in image's location metadata
based on location URL of image. Even if admin decides to change the store
names in glance-api.conf same will also be updated in location metadata
for all images related to that particular store. Current implementation of
legacy image performs this operation on each get/list call as location metadata
is not getting updated in database or it doesn't handle to perform store name
check in glance-api.conf.
Improvements done:
1. Save updated location metadata information in database permenantly
2. Add logic to perform lazy loading only if store information is not present
in location metadata or store present in location metadata is not defined in
glance's enbaled_backends configuration option.
Change-Id: I789fa7adfb459e7861c90a51f418a635c0c22244
Closes-Bug: #1886374
If operator has configured read-only 'http' store in multiple stores and while
importing or copying existing image in multiple stores if user has specified
--all-stores true then depending on `allow-failure`; if it is True then image
is imported/copied to other stores than http or if it is false then image can
not be imported/copied to other stores as well. Aslo, new image property
`os_glance_failed_imports` shows `http` as failed store.
Excluded `http` store from the list of stores for import/copying workflow
if user specifies --all-stores as True.
Change-Id: I2ad41022aa709491881e78129708da0ccc25f4f6
Closes-Bug: #1881958
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Newly added multiple import fails if user passes "all_stores" as "true"
in the request. The reason is if all_stores is specified then we are getting
stores using CONF.enabled_backends and unfortunately we are injecting
reserved stores 'os_glance_staging_store' and 'os_glance_tasks_store'
runtime. As a result import job tries to import the image in the
'os_glance_staging_store' store as well where it fails as that image
is already staged in this staging store.
Made a provision to exclude available reserved stores if "all_stores"
is specified.
Change-Id: If2616c275a969cdad5649e2cb0851275e5d0c7d2
Closes-Bug: #1863879
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.
Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
As eventlet ssl termination is broken with python 3 and
we won't be supporting python 2.7 anymore we will just
remove ssl termination to glance-api and expect the
termination being handled by something else, like HAProxy.
This patch also removes the broken ssl test job as the
non-existing feature is not broken anymore.
Change-Id: Iaf16dfcfdb3a2c93312dcad1ea1229e6b3c8caaa
In case of Multiple store while adding location to image if backend store is
not provided it returns 400 BadRequest saying 'Invalid location'.
Made changes to loaction API to findout the store from location URI
and add it as a backend store to the location metadata.
DocImpact
Closes-Bug: #1802587
Co-Authored-By: Victor Coutellier <victor.coutellier@gmail.com>
Change-Id: If6d0348346d2086a2500b0012a0e81e80cea7395
This should always be bytes; previously there was a path that returned an empty
native string.
Change-Id: I1cb4997cd3e766f9a7aea41ebe41a73971496bc3
Closes-Bug: #1818919
Replace raise of StopIteration with return.
PEP 0479, https://www.python.org/dev/peps/pep-0479/, makes the
following change: "when StopIteration is raised inside a generator,
it is replaced it with RuntimeError".
And states: "If raise StopIteration occurs directly in a generator,
simply replace it with return."
Change-Id: Ia7d6af7165077aa93c76185c28c355c06a202088
Closes-Bug: #1781627
Web-download tests introduced in
e54622ca72 did have some ignored
settings that did not get tested. Cleaned them out of the tests
that did not verify them and added test doing so. Also flattening
the path by one layer.
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Co-authored-by: Erno Kuvaja <jokke@usr.fi>
Change-Id: I01a73d41ae293e8204b0ea87e42922ba6a1bf9d2
In many places, there are more specific asserts
which can be used. I replaced the generic assert
with more specific ones, where it was possible.
This change enhances readibility, and on fail, more useful
message is displayed
Change-Id: I76dca80b6bd8886860fb383ee8d35e16cd99762e
When listing images there are several optional parameters that
can be used to filter the list of images retrieved by the API. The
following two parameters are not working properly: the created_at and
the updated_at. Before the Mitaka release it was possible to use these
filters just using a datetime in the format ISO 8601, starting on
Mitaka, you need to add an operator along with the datetime stamp or
the API call fails. This commit adds backwards compatibility so it is
possible to filter the images list using only a datetime stamp without
also specifing an operator. If no operator is used an eq operator is
assumed.
Change-Id: Id5d5455e77637e0dc7baec25c8163b21634d72c4
Partial-Bug: 1584415
Added support filtering images based on lists using the 'in' operator.
Filters:
*id
*name
*container_format
*disk_format
*status
DocImpact
ApiImpact
Implements bp: in-filtering-operator
Change-Id: I9cac81b9d5cbec979e88cf2dd0e3b710ed45630c
Provide support in Glance API for querying the image list for
created_at and updated_at times using guidance from the API Working
Group recommendations for filtering. Filtering is applied at the DB
layer.
DocImpact
ApiImpact
MitakaPriority
Change-Id: Ie94295bb82779ec17ab773928c71ae4a9ee8fbcc
Implements bp: v2-additional-filtering
* CooperativeReader: buffer is a byte string, use b'' syntax
and use bytes() instead of str().
* Don't compare None to int, it raises a TypeError on Python 3.
* Use str[i:i+1] to get a substring from a byte string. On Python 3,
str[i] now returns an integer for byte strings.
* tox.ini: add glance.tests.unit.common.test_utils to Python 3.4
Change-Id: Ie678555ffd8ac48913f9bf08ac674d1c643bec29
Incorrect variable name is declared in
test_validate_key_cert_key_cant_read() method.
In the second 'with' statement, changed variable name
from keyf to certf.
Closes-Bug: #1482144
Change-Id: Ibf3168702d13d1737f2a1978fdea6c357eae2467
* Replace glance.common.utils.exception_to_str() with
oslo_utils.encodeutils.exception_to_unicode().
* Remove exception_to_str() from glance.common.utils
* Add "from oslo_utils import encodeutils" and remove "from
glance.common import utils" (when utils is no more used)
* glance/api/v2/images.py: replace utils.exception_to_str() with
six.text_type() to cast the oslo_i18n message to Unicode
Glance exception_to_str() function was not compatible with Python 3,
whereas exception_to_unicode() is compatible with Python 3, it's in Oslo
common libraries and it's well tested.
This patch was first generated by a tool (modified version of sixer),
and then fixed to respect the PEP8 (especially the constraint of 80
columns).
Change-Id: I86008c8adc0c5664f96573c1015cc15e2d06e3e2
Utilizing assertIn and assertNotIn in a consistent way rather than
assertTrue( x <not> in y) and assertFalse(x in y).
Change-Id: Ic43d107c40a5a50b852ee476b1167c21e5ad86e8
CooperativeReader, being an eventlet-friendly wrapper around the
generator- based reader of image data, actually transforms
chunk-by-chunk iteration into the readable stream. It is used when the
image is being copied from the remote source: some generator-based
image data representing the remote source acts as its underlying
object, and the instance of CooperativeReader is passed as a data
stream to the backend client which uses it to read the data.
Before this patch, the CooperativeReader was ignoring the "length"
parameter of the read method, always returning the whole chunk returned
by the underlying generator (in case of HTTP source the size of this
chunk is 16 M). This was causing problems for the clients attempting to
read data from it, and - under some circumstances - the loss of data.
For chunked storage of files in Swift a special class (ChunkReader,
declared in the swift store driver) is used to reduce the requested
read length so no extra data is read and transferred. However, this was
not working as the CooperativeReader (which was the underlying stream
for the ChunkReader) was ignoring the requested size. This was causing
the data to be lost when reading behind the boundaries of the Chunks.
This patchset introduces a buffer in the CooperativeReader to store the
most recently fetched iterator chunk. The reads are independent from
requests to iterator, so the CooperativeReader is able to return the
exact requested amount of bytes and no data is lost due to extra-reads.
SecurityImpact
Change-Id: Ief37d1e29487bb03e612320f5cc06910cfd1c23a
Closes-bug: #1412802
It would be great to enhance Glance to use minimum of SHA2
to do digital signature for FIPS compliance. Since in
FIPS(FEDERAL INFORMATION PROCESSING STANDARDS) says the
SHA-1 is not suitable for general-purpose digital signature
applications (as specified in FIPS 186-3) that require 112
bits of security. In the case of digital signatures, SHA-1
does not provide the 112 bits of collision resistance needed
to achieve the security strength.
Now we're using hardcode 'sha1'. So this patch will make it
configurable firstly and set the default value as sha1 in
Kilo for smooth upgrade, which will be changed with sha256
in next release(L).
DocImpact
UpgradeImapact
SecurityImpact
Closes-Bug: #1288545
Change-Id: I9236cc85f4e9881ac1aa35d69bc6761a59c1b6c8
The only reason netaddr was pulled in was for ip validation routines
which exist in oslo.utils. Use those instead.
Change-Id: Ic3beee7e8bfb2b2d16ecac9be6322fe2704c70bd
Args of assertEquals method in glance.tests are arranged
in wrong order. In result when test fails it shows incorrect
information about observed and actual data. It's found more
than 2000 times. Right order of arguments is "expected, actual".
Change-Id: Ifea4809f5a696b608a19e36a9ed9e5eec46c8a21
Co-Authored-By: Li Yingjun <liyingjun1988@gmail.com>
Closes-Bug: 1277104
When trying to use "glance-replicator livecopy" to replicate the
images from one OpenStack glance to another, it encounters "Bad
header" error for "x-image-meta-virtual-size" header. This is
because this header is not in the whitelist of v1 API headers of
form "x-image-meta-xxx".
Another problem after adding this "x-image-meta-virtual-size" to
header whitelist is that the sqlalchemy fail due to virual_size is
"None", which cannot be converted to int. To fix this problem, we
should avoid converting the None value attribute in GET
/v1/images/detail response to the header of POST /v1/images. I
added the None value check, following what the way in
glance/common/utils.py image_meta_to_http_headers() method.
Closes-Bug: #1373333
Change-Id: I13182c4e3212024a9d5663cefd2951c576961f4f
Added new rule in policy.json and applied that rule to
'download_image' policy.
For example,
"restricted": "not ('test_key':(test_key)s and role:_member_)"
"download_image": "role:admin or rule:restricted"
So if 'download_image' policy is enforced then in above case only admin or
user who satisfies rule 'restricted' will able to download image. Other users
will not be able to download the image and will get 403 Forbidden response.
In addition, delete property access should be restricted for other users
so that they will not be able to delete the property of the image.
[test_key]
create = admin,member
read = admin,member,_member_
update = admin,member
delete = admin,member
Added new method to create dictionary-like mashup of image core and custom
properties.
Modified v1 and v2 api to add download restriction.
Modified logic of caching to restrict download for v1 and v2 api.
DocImpact:
Need to add new rule in policy.json
"restricted": "not ('test_key':%(test_key)s and role:_member_)"
blueprint: restrict-downloading-images-protected-properties
Change-Id: I05bad0441952150bd15b831ac1b1a0bb9ae79c74
To support Python3, unicode() calls has been replaced by
six.text_type.
Added utils.exception_to_str(): is the proper way to convert
an exception to string, since it manages logic related to
encoding.
Change-Id: I27101390e4f95e5c7690b1b445b7e75b8bcb9a08
Closes-Bug: #1284677
Due to a bad regex, the glance-replicator tool rejects any source or
target host:port combination that contains a period. As a result, only
hostnames without periods can be used.
This fix expands the host/port checks to:
- allow periods in hostnames
- allow and verify IPv6 addr/port pairs (i.e., [fe80::f00d:face]:1234)
- allow and verify IPv4 addr/port pairs (i.e., 172.17.17.2:1234)
- sanity-check port numbers
This also includes extensive tests for each component of the parsing,
and adds network_utils to openstack-common.conf.
Change-Id: I94fdd7a57a4cb0aa5d79f66d68be159d1f1266d1
Closes-Bug: #1216247
o glance/common/utils.py:
def add_features_to_http_headers(features, headers):
This method was removed in the following commit.
commit 76c3620c7e
Author: Brian Waldon <bcwaldon@gmail.com>
Date: Fri Sep 7 17:21:34 2012 -0700
Drop Glance Client
In this case, it has been used only in the following test program.
- glance/tests/unit/common/test_utils.py
o glance/tests/functional/__init__.py: def reload_server(self,
This method was removed in the following commit.
commit 866cb846f5
Author: John Bresnahan <jbresnah@redhat.com>
Date: Fri May 3 10:50:42 2013 -1000
Remove glance-control from the test suite
The following methods are not used(w/o tests), however, it can not be
deleted, added the comment.
o glance/common/location_strategy/__init__.py:
def verify_location_strategy(conf=None,
strategies=_available_strategies):
Change-Id: I50e3fdb6e929337b946e3534c8e7dbff77ac7998
Oslo is removing uuidutils during the icehouse cycle. We had removed
uuidutils.generate_uuid() in https://review.openstack.org/#/c/60643/
It seems the UUID test was not executed, for example, there is no method
called generate_uuid() in common/utils, the Jekins still passed. This
patch move the testcase and remove the test method for generate_uuid.
Change-Id: If4f508129f0b44d63fe77ed1251c0e0737e7e897
Partial-Bug: #1253497
to keep Python 3.x compatibility, use six.StringIO/BytesIO to
replace StringIO.StringIO
StringIO works for unicode
BytesIO works for bytes
Change-Id: I93e043c633e1de9e4dedcb0a313032403b6a70fb
Closes-Bug: #1280100
Add a whitelist of allowed 'x-image-meta-xxx' style
headers. Attempts to supply other, unexpected headers
will return 400.
This prevents cases such as 'x-image-meta-locations'
being processed, which were not being handled correctly.
Addresses bug 1205018.
Change-Id: I771bb6ae2a4f9cbd7726f952c7a71da99162b490
The only change of behaviour here (according to the former glance
unit tests) is that 'T' now means True not False. I suspect that's
ok.
Change-Id: Icf98f913a3f78a1fe2f38a82a25a071c2da03739
Abhishek Kekane