This adds a little more caution around the format_inspector,
specifically when calling virtual_size in case something very
unexpected goes wrong to ensure we don't interrupt the upload
process.
Change-Id: I14648e7b724b771755cd10fc5f5362a826780dd7
Related-Bug: #1983279
If we are processing a disk_format that we know about, wrap the
data pipeline with the format inspector and set virtual_size
after upload is complete.
Related to blueprint calculate-virtual-size
Change-Id: I25cd3cde94fefaa5d8ac72f10a075fe34a5df7bf
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.
Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
While creating an image, image data stays in backend if image
signature verification fails.
After raising SignatureVerificationError exception, image status is
being set to 'killed' in DB but the image data remains as it is in
the backend.
Adding delete_from_backend() call to cleanup the data from backend when
Singature Verification fails.
Closes-Bug: #1736336
Change-Id: I2a1a7addd33050cc8845aec24479aa4d1bc26ca0
This change removes the signature_utils module
from Glance and uses the cursive library, which
contains an identical module.
Change-Id: I80fcafa528b87a83b90ed7c0e4c0db9228852bc2
Depends-On: Ic3ffb6b318dc2ac6c9d3a60bed5198fd4d40e318
Partial-Bug: #1528349
This change removes the "sign-the-hash" signature
verification code in the signature_utils module and
the ImageProxy class. This code was deprecated in
Mitaka and scheduled for removal in Newton.
Change-Id: I8862f6c94538dd818c7360ba287e14c1264ff20f
Closes-Bug: #1516031
Per discussion on the mailing list [1] and the related nova
specification [2] it has been decided that the signature should be of
the image data directly, rather than of the glance MD5 "checksum" hash
of the image data.
This patch adds the ability to verify a signature of the image data
directly, using a verifier that is passed to the glance_store backend.
It is dependent on a glance_store patch which updates this verifier
object with the image data as it is creating the checksum (see
Depends-On below).
[1] http://bit.ly/1Q0M0C7
[2] https://review.openstack.org/#/c/188874/19
Depends-On: I43799e6a4a6643a23769af8d839a2beb4e0ff9bf
Partial-Bug: #1516031
Change-Id: If0c06b3094cecef6c8ca8a65753038b6b5a9d8fe
TestStoreAddToBackend contains a set of tests that should belong to
the glance_store. Recently, these tests were about to break
glance as they mocked internal methods. Those mocks were
removed in commit eab1567d48.
The tests are added back to glance_store in
change https://review.openstack.org/#/c/273786.
Depends-On: I688629d37e9146aef33fbc9bb6fd8780521e06ab
Change-Id: Ic67826a4d02cb30cc429a3d4abc46d6ca96f31ad
The glance_store library now accepts a verifier object for use with
verifying signatures, as a result of a recently merged patch [1].
There are some glance unit tests that do not expect this verifier
object to be passed to the add method of glance_store, and will
therefore fail with a new release of the glance_store library.
This patch removes the assert lines for the tests that will fail,
since it is not necessary for glance to test glance_store in that
way at the glance level.
[1] https://review.openstack.org/#/c/183110/
Change-Id: I404fbc40dafa159a63a6dfee563f05ee80c073dc
Partial-Bug: #1516031
Image members CRUD doesn't generate notifications which
is impacting searchlight service by not having latest
changes to Image memberships.
If you create an image and later change its members,
the members are not updated via notifications.
You have to run the index sync again to get the updated
member list.
See: https://bugs.launchpad.net/searchlight/+bug/1490697
Membership information is critical for horizon filtering.
Typically, a person is allowed to view an image under the
following conditions:
1) The image is owned by the project I am currently logged into.
2) The image is public
3) The image is owned by another project which has added me
as a member and I have accepted membership to it.
Without current membership information, 3) above is not possible.
See: https://bugs.launchpad.net/searchlight/+bug/1491085
Change-Id: Ia56e42d3d8da36cfa419d5c3c7d69c9ccf8974fd
Closes-Bug: #1441453
This patch adds support for the image signing feature by adding the
ability for glance to verify a signature.
This patch still needs:
* castellan added to global requirements (see Depends-On below)
Depends-On: I2283d2853d4ccd6d41d706db5b02cf6c74c5ba93
Change-Id: I0b0592a3526a4e4f5b39ae6ce8b4dedd0ccc31d9
Implements: blueprint image-signing-and-verification-support
* StoreLocations: add a __eq__() method, Python 3 doesn't use __cmp__()
anymore
* Fix StoreLocations.__delitem__(): Python 3 now calls it with a slice
for "del locations[a:b]" instead of calling __delslice__().
* Fix test_store_location: mark byte strinsg with b'...' prefix.
* tox.ini: add test_store_location to Python 3.4
Change-Id: Ibe8dac3d442ee08ae6b347e256947b6b9c5224ae
Currently image data cannot be removed synchronously for an image that
is in saving state. And when, the upload operation for such an image is
completed the operator configured quota can be exceeded.
This patch fixes the issue of left over chunks for an image which was
deleted from saving status. However, by the limitation of the design we
cannot enforce a global quota check for the image in saving status.
This change introduces a inconsonance between http response codes of
v1 and v2 APIs. The status codes which we will now see after the upload
process completes on an image which was deleted mid way are:
v1: 412 Precondition Failed
v2: 410 Gone
SecurityImpact
UpgradeImpact
APIImpact
Closes-Bug: 1383973
Closes-Bug: 1398830
Closes-Bug: 1188532
Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
The change will be used to restrict client to download and delete any
file in glance-api server. The same resone and logic as what we did in
v1:
https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L429
Closes-Bug: bug/1400966
DocImpact
Note: Even this change could fully resolve the problem for Glance, but
we still need to fix this issue from glance_store perspective
separatelly due to other projects can use the lib directly.
Change-Id: I72dbead3cb2dcb87f52658ddb880e26880cc229b
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Args of assertEquals method in glance.tests are arranged
in wrong order. In result when test fails it shows incorrect
information about observed and actual data. It's found more
than 2000 times. Right order of arguments is "expected, actual".
Change-Id: Ifea4809f5a696b608a19e36a9ed9e5eec46c8a21
Co-Authored-By: Li Yingjun <liyingjun1988@gmail.com>
Closes-Bug: 1277104
Latest glanec_store release added a new `context` keyword to some
functions. This broke backwards compatibility for some of the tests in
master and juno.
This change adds the corresponding keyword to the mock calls.
Closes-bug: #1391437
Change-Id: I5e58462e4c076b3ea56e27361eb6c3b0aafa2493
This patch adds support for random access to images to the API v2. It's
possible to partially access the image data by passing the offset where
the image should be fetched from and the size of the chunk expected.
Note that not all stores support random access to image data. In such
cases, the API will return an error mentioning the selected store
doesn't support such a feature.
DocImpact:
Content-Range header is now parsed to support random access to image
data.
Implements blueprint: restartable-image-download
Change-Id: Iade692a8197a9a1d9532711a5c572c505e5e7d20
This commits removes the old `store` package from glance and adopts the
usage of the new glance.store library. The library was designed to
preserve backwards compatibility as much as possible. In fact, most of
the changes in this patch are related to function args ordering and not
function renames or workflow changes.
Some changes that are worth mentioning:
1. Glance store doesn't rely on a global config object. All config
options must be explicitly registered.
2. All store operations now accepted an optional context. This is a
fallout from the context not being required in the `Store` constructor
anymore.
3. Store drivers are behind a private package called `_drivers` and
they're not suppose to be accessed directly. Instead, functions like
`get_store_from_scheme` should be used.
4. Stores are disabled by default
5. All the store specific options are under the `glance_store` group.
DocImpact:
The old store related configuration options have been moved under the
`glance_store` section. However, the old options will go through a
deprecation path. That is, they'll still be read from the `DEFAULT`
section to give deployers enough time to update their config files.
In k-2, the deprecated options will be completely obsolete.
Closes-bug: #1291848
Implements-blueprint: create-store-package
Change-Id: Iaacc70993ad5da292b93de42bbecda73d53b19fd
image_size_cap should be checked and enforced on upload
Enforcement is in two places:
- on image metadata save
- during image save to backend store
Closes-Bug: 1315321
Change-Id: I45bfb360703617bc394e9e27fe17adf43b09c0e1
Co-Author: Manuel Desbonnet <manuel.desbonnet@hp.com>
Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.
Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.
This is second part of this change which covered DB API, domain and REST
API.
Partially-Implements BP: image-location-status
Change-Id: I744679e2dadbaec099aef33d8c5a3fe4ecf96865
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
This patch moves the location domain's specific code out of
glance.store. Since this code is Glance specific, it's necessary to move
it out of the store package so that we can then remove it all together
and use the `glance.store` library.
Partially-Implements blueprint: create-store-package
Change-Id: I7dc3d5d211054cabc2adc2b9b689cda11bce2190
The following replacements were done in unit tests to have
clearer messages in case of failure:
- assertTrue(* is None) with assertIsNone
- assertTrue(* is not None) with assertIsNotNone
- assertTrue(* in *) with assertIn
- assertTrue(* not in *) with assertNotIn
- assertFalse(* in *) with assertNotIn
- assertTrue(* == *) with assertEqual
- assertTrue(* != *) with assertNotEqual
Change-Id: I0c47f991c3974e441335e71c9d26fab8a127f2ca
Glance currently enables all stores by default. This patch changes that
by removing all stores that require manual configuration and leaving
those that work right out of the box.
Current behavior causes a lot of confusion to users since most of those
stores print errors when they're not configured correctly. All extra
stores should be enabled explicitly by users.
This fix makes tests use http locations. All other locations besides the
default ones should be tested in their own test suites.
DocImpact
Closes-bug: #1255556
Change-Id: I82073352641d3eb2ab3d6e9a6b64afc99a30dcc7
Glance should not allow already uploaded images to be modified by
another upload. Currently, when configured with the local filesystem
backend, Glance prevents duplicate upload of an already uploaded image
BUT it changes the image status to "saving".
This commit adds a status transition state machine to the domain.Image
class and modifies the domain.Image.status setter function to verify
status transitions based on this state machine - only target states that
can be reached from the image's current state are permitted.
Tests have also been agumented to verify that the original image (meta)
data does not change in case of a conflicting upload.
Closes-Bug: #1241379
Change-Id: I62c5acae4c29abf0691d8279b51c59008f9c0047
Based on current implement, the image size won't be updated if user
update/add locations against a queued image. This fix will set the
image size based on given location.
Closes-Bug: #1261624
Change-Id: I265c6a92274ab2f94e005a1ab50e01d2f0c2143e
This commit makes the glance code base E125 and E126 compliant :
* E125 continuation line does not distinguish itself from next logical line
* E126 continuation line over-indented for hanging indent
Change-Id: I7120149bedb665fb66320498fe98948602a6cd52
Closes-bug: #1263437
We currently have a requirement for mox >= 0.5.3.
Confusingly there are two flavours of "mox 0.5.3"
in the wild:
1) pypi's mox-0.5.3.tar.gz (https://pypi.python.org/pypi/mox)
2) Google's mox-0.5.3.tar.gz (https://code.google.com/p/pymox/)
These are not exactly the same; tests which can pass on (1) may
fail on (2).
Fedora packages are based on (1) while Debian/Ubuntu packages
are based on (2).
This is a known issue: https://code.google.com/p/pymox/issues/detail?id=40
When running the tests using a virtual env (1) is used, but
on Debian/Ubuntu when not running using a virtual env (2)
will be used: this leads to 3 of the 1740 tests failing.
This patch allows all 1743 tests to run whether you are running
in a virtual env or using Debian/Ubuntu/Fedora packages.
Fix for bug 1212625.
Change-Id: Ib0b11ec18aafbfb7f3d243bf149b94f93a1c13c9
Enable image locaton proxy checking metadata when the location changing.
Implemented bp: location-proxy-metadata-checking
Change-Id: If411a454cd2c3e277ea9840cc3d83ec5125bb372
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
This patch modifies the PATCH /v2/images/{id} API call. Clients can add,
remove and replace locations from the set of multiple locations
associated with a given image ID in the following way:
PATCH /images/1234
[{"op": "add", "path": "/locations/-",
"value": {"url": "scheme3://path3", "metadata": {}}}]
PATCH /images/1234
[{"op": "add", "path": "/locations/1",
"value": {"url": "scheme4://path4", "metadata": {}}}]
PATCH /images/1234
[{"op": "remove", "path": "/locations/2"}]
PATCH /images/1234
[{"op": "replace", "path": "/locations", "value": []}]
PATCH /images/5678
[{"op": "replace", "path": "/locations",
"value": [{"url": "scheme5://path5", "metadata": {}},
{"url": "scheme6://path6", "metadata": {}}]}]
Glance will check location correctness when client adding, and will
remove the image content from the store when client remove a location.
Implement bp: multiple-image-locations
docimpact
Change-Id: I845646fde22e18be27929b5ec70ef8041b6fa733
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Enable image domain object fetch data from multiple locations, allow API
client consume image from multiple backend store.
Implement bp: multiple-locations-downloading
Change-Id: I512e2b517ac0222339d61c490d620fc414dd8e7a
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
This patches allows a storage system to add metadata to a location.
For example, if a location is a file:// URL the storage system may
wish to add information about it like the following:
{'FS': Gluster, 'mountpoint': '/usr/local/', 'namespace': 'abc-efg-xyz'}
Such information can be useful to a client with access to the direct
URL (often times the URL alone cannot encode enough information).
With this change when new data is added to the store a dictionary
is returned along with URL. That dictionary is then stored in the
DB with the URL. It is up to each store to determine what is in this
metadata dictionary. Possible way to do so are with a configuration
file. This patch does not included storage systems (other than tests)
that set this information.
If the API service is configured with the follow options:
show_multiple_locations = True
then the location and the location information is returned with the
image information in the field 'locations'. locations is
a list with the following format:
{'url': '<url>', 'metadata': <a storage specific dict>}
With this patch it will always be a list of one, but future patches
relating to the blueprint multiple-image-locations will allow for more.
blueprint: direct-url-meta-data
blueprint: multiple-image-locations
docimpact: show_multiple_locations
Change-Id: Ia832b8a8366bb06bfbaa53871af39a6a10b5721d
Adding a proxy layer to image to take care of it's locations for the
store:
1. Image location correctness checking.
2. Remove image data from the store when a location is removed from an
image.
Implement bp: multiple-image-locations
Change-Id: Ifa8b19050b48d1099aa37e77d144d10e6587b99c
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
This change brings the domain model up to speed with the v1 stack by
issuing calls to store.set_acls whenever
- an image is saved into the database (and locations are set)
- a member is added to an image (and locations are set)
related blueprint glance-api-v2-image-sharing
Change-Id: I45f10752f6c04646b65f86e44b59bdeea534f024
Prior to this change, it was common for a domain layer that was wrapping
Image objects returned by methods to neglect to unwrap the objects on
the way in to other methods.
This change fixes that by making proxy base classes less leaky and
inheriting from them in the various domain layers.
Change-Id: I22866cdd800c4c95b43c92762ead3775cffeaa80
* The DB API now exposes a 'locations' image attribute rather than
'location'. The new field is guaranteed to be a list of zero of
more items
* The v1 and v2 APIs only look for the first item in the list of
locations.
* Related to bp multiple-image-locations
Change-Id: I830b383d8a8e50a01e461658fb9abe384de1a353
Fixes bug 1034787
The recent change I made to image_data and the domain model caused this
bug to regress. This change adds assertions to the functional tests in
the hopes of preventing future regressions.
Change-Id: I02f9a5c51f54c0c778e032500079aa13ab073e3e
Patchset addresses reviewer's comments, rebase against master,
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Change-Id: I94367461505778973528eb8835d991f4fb796dab
When you delete an image, it should delete (or submit a delayed delete
for) the data in the store.
Part of implementing bp:glance-domain-logic-layer
Change-Id: I1ef4d6915a5fed02316b6945c99599511ea0392b
Dan Smith