In Xena we have mangaed to move all policy checks to API layer,
now removing the dead code from policy and authorization layer
NOTE: Some of the code is still being used from policy layer,
hence keeping it there only at this moment.
Change-Id: Ibee749cde20687d8c243cf84ae80b4de67d8ef3d
The image state diagram shows that for v2, any image upload failure
results in the image going back to 'queued' state. The old v1 path
used the 'killed' state, which is no longer value in image list
operations for v2. The signature verification error was sending the
image back to killed state still, which results in the user losing
visibility to the image entirely.
This removes the one last way we can get an image into killed state,
but there may still be images in the DB that have been orphaned in
this way, so some cleanup routine will be necesssary.
Change-Id: I9330cc42900dd019e9132c1f4e49eb9eff4f3fc9
Related-Bug: #1924612
This patch enforces policy checks required for staging image data
in API layer. Image mutability check is maintained if secure RBAC
is not enabled in deployments.
Partially-Implements: blueprint policy-refactor
Change-Id: Iba990e59c454a2e9f37a0c659d8634583d3a8bf7
Note that this adds a clause to the download exception handler which
returns 404 if the user can't download the image via policy, but leaves
the existing internal forbidden->403 handler because of how image
property protections work.
Also Cache middleware do not have any influence of policy refactoring.
Partially-Implements: blueprint policy-refactor
Change-Id: I060f22145d2915bfd0b0b3efc2912c24e89b9a30
Note that this adds a clause to the upload exception handler which
returns 404 if the user can't upload the image via policy, but leaves
the existing internal forbidden->403 handler because of how image
property protections work.
Partially-Implements: blueprint policy-refactor
Change-Id: I1353aacf595aa36c8c4823fbe7c6d0600a532f73
This makes us enforce a quota on the total number of upload-related
image operations owned by a user.
Partially-implements: blueprint glance-unified-quotas
Change-Id: I2a28750aaf968e6a6324eb194d4280a640bfa5aa
This makes us enforce a quota on the amount of data a user has in
staging.
Partially-implements: blueprint glance-unified-quotas
Change-Id: I3cca4e589adc0aec138e5933c311aefd69ccee51
This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.
Note that this disables checking of the global size quota if keystone
quotas are enabled.
Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.
Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
When we stage an image, we know the image size, so we should set it.
Since the user may be streaming an image and did not declare the
expected size, this lets them confirm before they import. It also
provides us a value to count for the staging quota coming in a later
patch.
The bulk of this change is adjusting existing tests to validate the
size for all of our stage/import tests.
This follows the change to set the image size during conversion,
ensuring that we can set it during stage for non-conversion jobs,
and that the conversion code that re-sets it after changing the
image data and size continues to work.
Related to blueprint glance-unified-quotas
Change-Id: I93a9145df27594a0cc59828619a7d0573e58d4fc
This commit updates the policies for image actions to use default roles
available from keystone. Specifically, we're updating the defaults to
user project-member and project-reader personas. The project-admin
persona is still reserved for administrative APIs access for system
administrators/operators. This will remain the case until we can
refactor portions of glance to make it easier to implement system-scope.
NOTE:
Glance is implementing Secure RBAC as EXPERIMENTAL in Wallaby, so to
enable it operator needs to set ``glance-api.conf [oslo_policy]
enforce_new_defaults=True`` and ``glance-api.conf
enforce_secure_rbac=True``
Implements: blueprint secure-rbac
Change-Id: If0c456617a9e17c006a6ffe2a83f4a73b53da3d0
This implements distributed image import support, which addresses
the problem when one API worker has staged the image and another
receives the import request.
The general approach is that when a worker stages the image, it
records its self-reference URL in the image's extra_properties. When
the import request comes in, any other host will proxy that HTTP
request direct to the original host instead of trying to do the import
itself.
Implements: blueprint distributed-image-import
Change-Id: I12daccb43c535b579c22f9d0742039b2ab42e929
This patch removes majority of the registry and it's related
endpoints and config options that has been deprecated for
removal in various releases.
Change-Id: I75014bd50bf382efebe56bd89c20ffefbdde25f5
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.
This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.
Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
In Rocky multiple backend support is added as experimental feature. In
order to take advantage of this feature it is decided to deprecate
work_dir and node_staging_uri configuration options
and reserve two filesystem stores 'os_glance_tasks_store' and
'os_glance_staging_store', which can be used to get rid of initializing
store via internal functions.
These internal stores are considered "reserved stores" by Glance.
For the time being, these are hard-coded as filesystem stores. The
store prefix 'os_glance_' is reserved for internal Glance use and
the glance-api service will refuse to start if a store with this
prefix is included in the enabled_backends config option in
glance-api.conf.
NOTE: Because there are no sensible default values for the location
of the datadir for each of these stores, the operator must define
'os_glance_tasks_store' and 'os_glance_staging_store' in
glance-api.conf configuration file as shown below.
[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir/
[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging/
Each filesystem store must have a unique datadir.
Depends-On: https://review.openstack.org/#/c/639765/
Implements: blueprint rethinking-filesystem-access
Change-Id: I86ec513c5fc653dbb97b79d953d8430f014e684f
When operator has not enabled 'file' store and using other stores like ceph,
swift etc. the uploading to staging area works as we explicitly
build 'file' store during this operation, while cleaning up we directly
use 'glance_store.delete_from_backend' which only works if 'file'
store is enabled.
Modified '_DeleteFromFS' task and _unstage call which will use os
module to unlink the file present in staging area explicitly to
delete the data from staging area.
Closes-Bug: #1803498
Change-Id: If0b3b0af9300301291758c67267890e0959ebb3c
RequestContext.tenant and user fields are deprecated in favor
of project_id and user_id respectively.
This change modifies the glance.context.RequestContext constructor
to transition usage of tenant/user to project_id/user_id until
all tests are moved over to the new attributes. Runtime usage of
the old fiels is updated.
To prevent new code from using the deprecated fields, a warnings
filter is added which will make tests fail if they hit code using
the old fields.
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Change-Id: I351380840308a24769ece93abc6d1a9a6d6aa06f
Made provision for multi-store support. Added new config option
'enabled_backends' which will be a comma separated Key:Value pair
of store identifier and store type.
DocImpact
Depends-On: https://review.openstack.org/573648
Implements: blueprint multi-store
Change-Id: I9cfa066bdce51619a78ce86a8b1f1f8d05e5bfb6
There are two methods to create images:-
Method A)
POST /v2/images
PUT /v2/images/{image_id}/file
Method B)
POST /v2/images
PUT /v2/images/{image_id}/stage
POST /v2/images/{image_id}/import
The traditional image upload API (PUT /v2/images/{image_id}/file)
uses 'upload_image' policy which is same for
Method B (POST /v2/images/{image_id}/import)
image-create-via-import(new API for image create) API.
The long term goal is to make users use method B to create images
and cross services to use Method A until changes are made to
use Method B.
To restrict normal users from using Method A to create images both
these APIs (/v2/images/{image_id}/file and /v2/images/{image_id}/import)
should have a distinct policy.
This patch move the 'upload_image' policy check from imge.set_data()
to the controller and not introduce any new policies at this point
for import API call (POST /v2/images/{image_id}/import)
on the theory that an operator can stop import by restricting the
'image_create' policy. And also this fix will not change the semantics
of the 'upload_image' policy from the operator perspective.
Closes-Bug: #1732141
Change-Id: Icc62add5f8d48549aac94c8058d66d6b77b56d41
1. When user calls stage command again then it fails with 500 internal
server error. This is because after stage call image status changes
to 'uploading' and for second call it again tries to set status to
'uploading' and fails with 'InvalidImageStatusTransition: Image
status transition from uploading to uploading is not allowed'
exception which is not caught at the controller side.
2. If image upload (/file call) is in progress image is in saving
state at that time. If user tries to make a /stage call on same image
then it returns 500 internal server error as Image transition from
saving to uploading is not allowed.
Caught 'InvalidImageStatusTransition' and return 409 HTTPConflict
resonse status.
Change-Id: Ie66d3e3474b38b2f6c8d31f5c1fb252d45cbd3c9
Closes-Bug: #1733274
Closes-Bug: #1733512
ImageSizeLimitExceeded exception block is unreachable in stage
call because glance is using sotre_backend.add (filesystem.add
in this case) and it will raise 'exceptions.StorageFull()' [1]
if image file is too large (errno.EFBIG).
Used LimitingReader to enforce the image size limitation which
will raise ImageSizeLimitExceeded if image size is larger that
config option 'image_size_cap'.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/filesystem.py#L691
Change-Id: I50821111ac4c782966ed1ccd48e92f8dd4100e62
Closes-Bug: #1732830
If user tries to stage data to unexisting image then it fails with 500
internal server error.
Caught NotFound exception and raised HTTPNotFound to return 404
response to the user.
Change-Id: I78d252fceb9a5537135f0c238c4ad2ba52fdda7c
Closes-Bug: #1733551
If store_backend denies writing to store then glance_store will raise
'StorageWriteDenied' error. After catching this exception at glance
side, unstage call fails with TypeError as it requires 3 arguments
and only two were provided.
Passed 'staging_store' argument to _unstage call for correcting the
same.
Change-Id: I6d3d1b304e84fbfc1d653b3640ed92082f5b11cd
Closes-Bug: #1732835
In stage() method of v2 api, _restore() is called after raising
HTTPConflict exception, which won't be reachable in any case after
the exception is raised.
_restore() changes the image_status from 'uploading' to 'queued'
and data staging would start from the beginning after new stage call
with same image-id even though staging data has already been loaded
to store.
Hence it seems there is no point in keeping that _restore() incase of
Duplicate exception is raised.
Change-Id: I317587bc1ba6544b722a11ae217a5d1212c96ce2
Closes-Bug: 1732077
Adds a RequestDeserializer for the stage and adjusts the image
status transitions so that they can handle the 'uploading' status
of an image with data in the stage.
Closes-bug: #1710958
Change-Id: I6f1cfe44a01542bc93a43cbd518956686adb366d
Change If045d76574823f29c58b39a104354abc1407cb8d removed
most instances of this, but apparently one more was added
between when that patch was done and when it merged. This
removes one more case.
Change-Id: Ia0bb871a844852f172c50724bda487379e9d44e1
'staging' endpoint and related store controls for users to upload
image data to staging are for direct-glance image import path.
Change-Id: I3f5faa2dfd4117882890f69be4089b82141d21c5
Currently glance v2 API incorrectly accepts ‘Content-Range’ header
for random image access and does not set response headers.
As per rfc7233, ‘Range’ requests should be accepted and ‘Content-Range’
must be returned in the response headers.
This patch enables Glance v2 API to accept the more appropriate ‘Range’
requests and sets ‘Content-Range’ response header.
For backward compatibility with pre-Pike Glance clients, the incorrect
'Content-Range' header will be accepted silently in perpetuity.
Thus this patch contains tests for 'Content-Range' in requests to
prevent regressions.
DocImpact
Implements lite-spec I5bdadde682a0c50836bd95e2a6651d6e7e18f172
Closes-Bug: #1677391
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ib7ebc792c32995751744be3f36cbc9a0c1eead2a
WebOb 1.7 changed [0] how request bodies are determined to be
readable. Prior to version 1.7, the following is how WebOb
determined if a request body is readable:
#1 Request method is one of POST, PUT or PATCH
#2 ``content_length`` length is set
#3 Special flag ``webob.is_body_readable`` is set
The special flag ``webob.is_body_readable`` was used to signal
WebOb to consider a request body readable despite the content length
not being set. #1 above is how ``chunked`` Transfer Encoding was
supported implicitly in WebOb < 1.7.
Now with WebOb 1.7, a request body is considered readable only if
``content_length`` is set and it's non-zero [1]. So, we are only left
with #2 and #3 now. This drops implicit support for ``chunked``
Transfer Encoding Glance relied on. Hence, to emulate #1, Glance must
set the the special flag upon checking the HTTP methods that may have
bodies. This is precisely what this patch attemps to do.
[0] https://github.com/Pylons/webob/pull/283
[1] https://github.com/Pylons/webob/pull/283/files#diff-706d71e82f473a3b61d95c2c0d833b60R894
Closes-bug: #1657459
Closes-bug: #1657452
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: I19f15165a3d664d5f3a361f29ad7000ba2465a85
Currently Glance does not send Partial response codes while
handling HTTP range requests. Also, content length is not
appropriately set.
This patch is to send partial response code and to set the correct
content length based on the range request for image download.
Upon success status code 206 is sent and the content length is set to the
requested range.
Upon failure, there can be 2 cases:
* If the HTTP range request for the image download is bad (For example,
requesting download of range of bytes 10 to 50 bytes when there are only 48
bytes), status code is set to 416 and HTTPRequestRangeNotSatisfiable is
raised.
* If the content range is valid, but the request is not satisfiable due to
glance_store side erros or privacy issues, appropriate exceptions are
raised.
APIImpact
DocImpact
Closes-Bug: #1417069
Closes-Bug: #1624508
Closes-Bug: #1399851
Closes-Bug: #1618928
Change-Id: I3cd47b998be79604511b3cd4879209820cf776b7
This change removes the signature_utils module
from Glance and uses the cursive library, which
contains an identical module.
Change-Id: I80fcafa528b87a83b90ed7c0e4c0db9228852bc2
Depends-On: Ic3ffb6b318dc2ac6c9d3a60bed5198fd4d40e318
Partial-Bug: #1528349
when an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where an exception
occured initially instead of place where the exception re-raised.
Change-Id: I76bda89eb37a309a66a68f6d3dea18d51843b83c
To eliminate potential race conditions when image status
is changed it's suggested to use 'from_state' parameter
for 'save' methods everywhere where it's possible.
Also this code prevents image update when status hasn't
been changed in deactivate/reativate methods.
Closes-Bug: #1557495
Change-Id: Ic79224a8686bea6ca79976a7f30e3c87bba4e6ec
As per http://developer.openstack.org/api-ref-image-v2.html:
"If no image data exists, the call returns the HTTP 204 status code. "
This commit changed that to 403:
d4d94b290c
We should revert to the juno/kilo/liberty behaviour.
APIImpact
Closes-bug: 1549869
Change-Id: Ie9353bc254d11870abc102a7b9b4c7db3917abb4
Implement trust support for images api v2 when uploading images
with registry.
The algorithm is the following:
1. If 'registry' is set as data_api backend create a trust
2. Upload an image
3. Try to update the image status to 'active' in registry
4. If trust has been created succcessfully and 401 occured during
updating the image status then renew the token and try to update
the image record in registry again
Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>
Co-Authored-By: Kairat Kushaev <kkushaev@mirantis.com>
Implements bp trust-authentication
Change-Id: Ia3b82782b14f5dfc93457620633c1039c38fc366
Add catches exception NotFound from glance_store when saving to
uploaded image.
Change-Id: Ib352af844610a8d5794372e9a0016d36fb30213e
Closes-bug: #1512416
Import i18n functions from module directly and do not use
global module variables like _ = i18n._. It makes code shorter
and cleaner. It also to detect cases when these functions are not
used in module.
Change-Id: Iaa593ac1f2dd15cbcad049bef6ba68f1cfa610da
Pranali Deore