In Xena we have mangaed to move all policy checks to API layer,
now removing the dead code from policy and authorization layer
NOTE: Some of the code is still being used from policy layer,
hence keeping it there only at this moment.
Change-Id: Ibee749cde20687d8c243cf84ae80b4de67d8ef3d
This is a rather beefy change due to the number of usages of this
import. The changes are trivial though.
Change-Id: I7badeeaca438b0291f4ed86670e7f217e6372c61
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The image state diagram shows that for v2, any image upload failure
results in the image going back to 'queued' state. The old v1 path
used the 'killed' state, which is no longer value in image list
operations for v2. The signature verification error was sending the
image back to killed state still, which results in the user losing
visibility to the image entirely.
This removes the one last way we can get an image into killed state,
but there may still be images in the DB that have been orphaned in
this way, so some cleanup routine will be necesssary.
Change-Id: I9330cc42900dd019e9132c1f4e49eb9eff4f3fc9
Related-Bug: #1924612
Note that this adds a clause to the upload exception handler which
returns 404 if the user can't upload the image via policy, but leaves
the existing internal forbidden->403 handler because of how image
property protections work.
Partially-Implements: blueprint policy-refactor
Change-Id: I1353aacf595aa36c8c4823fbe7c6d0600a532f73
When we stage an image, we know the image size, so we should set it.
Since the user may be streaming an image and did not declare the
expected size, this lets them confirm before they import. It also
provides us a value to count for the staging quota coming in a later
patch.
The bulk of this change is adjusting existing tests to validate the
size for all of our stage/import tests.
This follows the change to set the image size during conversion,
ensuring that we can set it during stage for non-conversion jobs,
and that the conversion code that re-sets it after changing the
image data and size continues to work.
Related to blueprint glance-unified-quotas
Change-Id: I93a9145df27594a0cc59828619a7d0573e58d4fc
This commit updates the policies for image actions to use default roles
available from keystone. Specifically, we're updating the defaults to
user project-member and project-reader personas. The project-admin
persona is still reserved for administrative APIs access for system
administrators/operators. This will remain the case until we can
refactor portions of glance to make it easier to implement system-scope.
NOTE:
Glance is implementing Secure RBAC as EXPERIMENTAL in Wallaby, so to
enable it operator needs to set ``glance-api.conf [oslo_policy]
enforce_new_defaults=True`` and ``glance-api.conf
enforce_secure_rbac=True``
Implements: blueprint secure-rbac
Change-Id: If0c456617a9e17c006a6ffe2a83f4a73b53da3d0
This implements distributed image import support, which addresses
the problem when one API worker has staged the image and another
receives the import request.
The general approach is that when a worker stages the image, it
records its self-reference URL in the image's extra_properties. When
the import request comes in, any other host will proxy that HTTP
request direct to the original host instead of trying to do the import
itself.
Implements: blueprint distributed-image-import
Change-Id: I12daccb43c535b579c22f9d0742039b2ab42e929
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.
Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.
NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.
Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
RequestContext.tenant and user fields are deprecated in favor
of project_id and user_id respectively.
This change modifies the glance.context.RequestContext constructor
to transition usage of tenant/user to project_id/user_id until
all tests are moved over to the new attributes. Runtime usage of
the old fiels is updated.
To prevent new code from using the deprecated fields, a warnings
filter is added which will make tests fail if they hit code using
the old fields.
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Change-Id: I351380840308a24769ece93abc6d1a9a6d6aa06f
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
Added some unit tests for coverage purpose.
Added functional tests for create and import scenarios.
Note:
For functional tests I have considered file store with two
different image directories.
Related to blueprint multi-store
Change-Id: I59e28ab822fb5f6940f48ddbf6dfba4cb7d4c509
Made provision for multi-store support. Added new config option
'enabled_backends' which will be a comma separated Key:Value pair
of store identifier and store type.
DocImpact
Depends-On: https://review.openstack.org/573648
Implements: blueprint multi-store
Change-Id: I9cfa066bdce51619a78ce86a8b1f1f8d05e5bfb6
There are two methods to create images:-
Method A)
POST /v2/images
PUT /v2/images/{image_id}/file
Method B)
POST /v2/images
PUT /v2/images/{image_id}/stage
POST /v2/images/{image_id}/import
The traditional image upload API (PUT /v2/images/{image_id}/file)
uses 'upload_image' policy which is same for
Method B (POST /v2/images/{image_id}/import)
image-create-via-import(new API for image create) API.
The long term goal is to make users use method B to create images
and cross services to use Method A until changes are made to
use Method B.
To restrict normal users from using Method A to create images both
these APIs (/v2/images/{image_id}/file and /v2/images/{image_id}/import)
should have a distinct policy.
This patch move the 'upload_image' policy check from imge.set_data()
to the controller and not introduce any new policies at this point
for import API call (POST /v2/images/{image_id}/import)
on the theory that an operator can stop import by restricting the
'image_create' policy. And also this fix will not change the semantics
of the 'upload_image' policy from the operator perspective.
Closes-Bug: #1732141
Change-Id: Icc62add5f8d48549aac94c8058d66d6b77b56d41
1. When user calls stage command again then it fails with 500 internal
server error. This is because after stage call image status changes
to 'uploading' and for second call it again tries to set status to
'uploading' and fails with 'InvalidImageStatusTransition: Image
status transition from uploading to uploading is not allowed'
exception which is not caught at the controller side.
2. If image upload (/file call) is in progress image is in saving
state at that time. If user tries to make a /stage call on same image
then it returns 500 internal server error as Image transition from
saving to uploading is not allowed.
Caught 'InvalidImageStatusTransition' and return 409 HTTPConflict
resonse status.
Change-Id: Ie66d3e3474b38b2f6c8d31f5c1fb252d45cbd3c9
Closes-Bug: #1733274
Closes-Bug: #1733512
ImageSizeLimitExceeded exception block is unreachable in stage
call because glance is using sotre_backend.add (filesystem.add
in this case) and it will raise 'exceptions.StorageFull()' [1]
if image file is too large (errno.EFBIG).
Used LimitingReader to enforce the image size limitation which
will raise ImageSizeLimitExceeded if image size is larger that
config option 'image_size_cap'.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/filesystem.py#L691
Change-Id: I50821111ac4c782966ed1ccd48e92f8dd4100e62
Closes-Bug: #1732830
If user tries to stage data to unexisting image then it fails with 500
internal server error.
Caught NotFound exception and raised HTTPNotFound to return 404
response to the user.
Change-Id: I78d252fceb9a5537135f0c238c4ad2ba52fdda7c
Closes-Bug: #1733551
If store_backend denies writing to store then glance_store will raise
'StorageWriteDenied' error. After catching this exception at glance
side, unstage call fails with TypeError as it requires 3 arguments
and only two were provided.
Passed 'staging_store' argument to _unstage call for correcting the
same.
Change-Id: I6d3d1b304e84fbfc1d653b3640ed92082f5b11cd
Closes-Bug: #1732835
Currently glance v2 API incorrectly accepts ‘Content-Range’ header
for random image access and does not set response headers.
As per rfc7233, ‘Range’ requests should be accepted and ‘Content-Range’
must be returned in the response headers.
This patch enables Glance v2 API to accept the more appropriate ‘Range’
requests and sets ‘Content-Range’ response header.
For backward compatibility with pre-Pike Glance clients, the incorrect
'Content-Range' header will be accepted silently in perpetuity.
Thus this patch contains tests for 'Content-Range' in requests to
prevent regressions.
DocImpact
Implements lite-spec I5bdadde682a0c50836bd95e2a6651d6e7e18f172
Closes-Bug: #1677391
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ib7ebc792c32995751744be3f36cbc9a0c1eead2a
Currently Glance does not send Partial response codes while
handling HTTP range requests. Also, content length is not
appropriately set.
This patch is to send partial response code and to set the correct
content length based on the range request for image download.
Upon success status code 206 is sent and the content length is set to the
requested range.
Upon failure, there can be 2 cases:
* If the HTTP range request for the image download is bad (For example,
requesting download of range of bytes 10 to 50 bytes when there are only 48
bytes), status code is set to 416 and HTTPRequestRangeNotSatisfiable is
raised.
* If the content range is valid, but the request is not satisfiable due to
glance_store side erros or privacy issues, appropriate exceptions are
raised.
APIImpact
DocImpact
Closes-Bug: #1417069
Closes-Bug: #1624508
Closes-Bug: #1399851
Closes-Bug: #1618928
Change-Id: I3cd47b998be79604511b3cd4879209820cf776b7
There are several places in the source code where
HTTP response codes are used as numeric values.
These values are used from six.moves and the
numeric values are replaced by constants.
All of the used status codes were replaced with symbolic constants
from six.moves.http_client.
More about six.moves.http_client can be found at [2],
under the table "Supported renames:".
Also, this change improves code readibility.
This patchset does not extract numeric values from
the code itself, but it can be found at [1].
[1]: Ib9e26dcea927e96e65c626c18421621d3a29a64d
[2]: https://pythonhosted.org/six/#module-six.moves
Change-Id: Idfc7b043552f428f01ac3e47b270ee0639a8f5bc
Closes-Bug: #1520159
This change removes the signature_utils module
from Glance and uses the cursive library, which
contains an identical module.
Change-Id: I80fcafa528b87a83b90ed7c0e4c0db9228852bc2
Depends-On: Ic3ffb6b318dc2ac6c9d3a60bed5198fd4d40e318
Partial-Bug: #1528349
verbose option is going to be deleted soon (see
https://review.openstack.org/#/c/206437/ for more info
why it deprecated in oslo.log) because it confuses users
(we already have debug option). So we need to delete it from glance
tests.
Change-Id: Iade46a6097d153f2df0da73554bbfd3e2b1704f5
To eliminate potential race conditions when image status
is changed it's suggested to use 'from_state' parameter
for 'save' methods everywhere where it's possible.
Also this code prevents image update when status hasn't
been changed in deactivate/reativate methods.
Closes-Bug: #1557495
Change-Id: Ic79224a8686bea6ca79976a7f30e3c87bba4e6ec
Instead of using assertTrue(A * B), developers should
use assertGreater(A, B) or assertLess(A, B) or
assertGreaterEqual(A, B) or assertLessEqual(A, B)
or assertEqual(A, B).
The * operator: >=, <=, ==
Change-Id: I85e0087c530c2f1132061036fd5427368dfa2316
As per http://developer.openstack.org/api-ref-image-v2.html:
"If no image data exists, the call returns the HTTP 204 status code. "
This commit changed that to 403:
d4d94b290c
We should revert to the juno/kilo/liberty behaviour.
APIImpact
Closes-bug: 1549869
Change-Id: Ie9353bc254d11870abc102a7b9b4c7db3917abb4
Implement trust support for images api v2 when uploading images
with registry.
The algorithm is the following:
1. If 'registry' is set as data_api backend create a trust
2. Upload an image
3. Try to update the image status to 'active' in registry
4. If trust has been created succcessfully and 401 occured during
updating the image status then renew the token and try to update
the image record in registry again
Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>
Co-Authored-By: Kairat Kushaev <kkushaev@mirantis.com>
Implements bp trust-authentication
Change-Id: Ia3b82782b14f5dfc93457620633c1039c38fc366
Add catches exception NotFound from glance_store when saving to
uploaded image.
Change-Id: Ib352af844610a8d5794372e9a0016d36fb30213e
Closes-bug: #1512416
When using v2 an attempt to download an image would return a 403 if the
get_image_location policy was set.
Note: We had been returning both 404 and 204 when no data was
available. There was no way to detect the 404 case without trying to
access the image locations so I've standardized on 204.
Change-Id: I658b08a35d3a8cb8a7096baf716ccb3d6e7d9abf
Closes-bug: 1501672
In patch I47229b366c25367ec1bd48aec684e0880f3dfe60 it was
introduced the logic that if image was deleted during file
upload when we want to update image status from 'saving'
to 'active' it's expected to get Duplicate error and delete
stale chunks after that. But if user's token is expired
there will be Unathorized exception and chunks will stay
in store and clog it.
And when, the upload operation for such an image is
completed the operator configured quota can be exceeded.
This patch fixes the issue of left over chunks for an image
which was deleted from saving status, by correctly handle
auth exceptions from registry server.
Partial-bug: #1498163
Change-Id: I17a66eca55bfb83107046910e69c4da01415deec
Some tests used incorrect order assertEqual(observed, expected).
The correct order expected by testtools is
assertEqual(expected, observed).
At some places, corrected argument order for assertNotEqual method as well
and replaced assertEqual with assertTrue/assertFalse wherever required.
Closes-Bug: #1259292
Change-Id: I6fff8a3eedd053d0de8e261245a32f9a9fdab213
This patch adds support for the image signing feature by adding the
ability for glance to verify a signature.
This patch still needs:
* castellan added to global requirements (see Depends-On below)
Depends-On: I2283d2853d4ccd6d41d706db5b02cf6c74c5ba93
Change-Id: I0b0592a3526a4e4f5b39ae6ce8b4dedd0ccc31d9
Implements: blueprint image-signing-and-verification-support
Pranali Deore