The oslo.limit 2.3.0 release contains the validation to ensure
the [oslo_limit] endpoint_id option is set[1]. This fixes the test
failures caused by that validation by overriding the option in all
test cases where unified quota implementation is used.
[1] https://review.opendev.org/c/openstack/oslo.limit/+/897509
Closes-Bug: #2049064
Change-Id: I704393892799ee0d5edf45fabe07afe01473e38d
This is the same as the 'range' keyword in Python 3
Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.
Note that this disables checking of the global size quota if keystone
quotas are enabled.
Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.
Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
This adds some infrastructure to be able to query and honor limits
declared in keystone. It adds a single initial quota value for the
total size of all active images for bootstrapping the tests.
Checking these values is controlled by a new configuration option
that globally enables and disables the checking, defaulting to
False.
Related to blueprint glance-unified-quotas
Change-Id: I8d8f4aaed465486e80be85bc9a5d2c2be7f1ecad
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.
Change-Id: I5c32a049e3c8c84777ea4a01e04d1416881bfb5c
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.
Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
Infinite recursion is possible if you try to deepcopy a
QuotaImageTagsProxy object, which is something that happens when using
oslo.policy 1.43.1 and beyond.
By adding a __gettattribute__ within the __getattr__ method we can break
the loop when the name of the attribute is 'tags'.
The added test will fail (with infinite recursion) if the new code is
removed.
For completeness a test is added checking what happens when the tags
attribute is removed.
The thing that is not immediately clear from the __getattr__ code is
that it is not for getting at the tags themselves, it is for getting
at the methods on the set object at self.tags. The non-obviousness
of this led to rather a lot of confusion while trying to get this
code right. It isn't really "right" now, but rather: safe in the
context of its use.
Change-Id: Ic47c9bf8e9b97de5a5a49a35f631753c54e0e2af
Related-Bug: #1808063
In the related bug (below) we can see getting tags from a
QuotaImageTagsProxy can sometimes cause an infinite loop, presumably
because of bizarre threading circumstances. To at least provide a bit
more information when this bizarre circumstance happens, instead
of looping, resulting in a stackoverflow and the processing being
terminated without a traceback, now raise a TypeError. This should
help figure out what may be going on in the rare times this happens.
Also, it is generally a healthy idea to always have a guard like this
in a __getattr__ that uses an attr.
Change-Id: Ib4ad119a838d5b06372d9302dc6608da1772eed6
Related-Bug: #1808063
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
Made provision for multi-store support. Added new config option
'enabled_backends' which will be a comma separated Key:Value pair
of store identifier and store type.
DocImpact
Depends-On: https://review.openstack.org/573648
Implements: blueprint multi-store
Change-Id: I9cfa066bdce51619a78ce86a8b1f1f8d05e5bfb6
This patch removes the redundant tearDown() methods
in order to keep the code clean, In the subclass,
tearDown methods has no benefit from that,
the one in superclass will satisfy that
Change-Id: I0c23ceb133ebd0e1af41195a24597c3ce9e443b0
In Python 3 __ne__ by default delegates to __eq__ and inverts the
result, but in Python 2 they urge you to define __ne__ when you
define __eq__ for it to work properly [1].There are no implied
relationships among the comparison operators. The truth of x==y
does not imply that x!=y is false. Accordingly, when defining
__eq__(), one should also define __ne__() so that the operators
will behave as expected.
[1]https://docs.python.org/2/reference/datamodel.html#object.__ne__
Change-Id: I7d41128d88c79ebb5de1c38f8aaf98cc9561367c
Image members CRUD doesn't generate notifications which
is impacting searchlight service by not having latest
changes to Image memberships.
If you create an image and later change its members,
the members are not updated via notifications.
You have to run the index sync again to get the updated
member list.
See: https://bugs.launchpad.net/searchlight/+bug/1490697
Membership information is critical for horizon filtering.
Typically, a person is allowed to view an image under the
following conditions:
1) The image is owned by the project I am currently logged into.
2) The image is public
3) The image is owned by another project which has added me
as a member and I have accepted membership to it.
Without current membership information, 3) above is not possible.
See: https://bugs.launchpad.net/searchlight/+bug/1491085
Change-Id: Ia56e42d3d8da36cfa419d5c3c7d69c9ccf8974fd
Closes-Bug: #1441453
Instead of using six.text_type (which may or may not work)
use the oslo.utils helper function that does a much more
through job of trying to get a valid exception message from
a provided exception.
Change-Id: I6a58ca489a8a72f27e8f5e70b608683477745e4a
Mainly to improve consistency, use range() from six.moves
renames across glance.
Behaves consistently like py2 xrange() and py3 range().
Removes unnecessary range() from glace/api/v2/images.py
Change-Id: Id21f923d05600b902f2239e25ef01716c07e74a3
Currently image data cannot be removed synchronously for an image that
is in saving state. And when, the upload operation for such an image is
completed the operator configured quota can be exceeded.
This patch fixes the issue of left over chunks for an image which was
deleted from saving status. However, by the limitation of the design we
cannot enforce a global quota check for the image in saving status.
This change introduces a inconsonance between http response codes of
v1 and v2 APIs. The status codes which we will now see after the upload
process completes on an image which was deleted mid way are:
v1: 412 Precondition Failed
v2: 410 Gone
SecurityImpact
UpgradeImpact
APIImpact
Closes-Bug: 1383973
Closes-Bug: 1398830
Closes-Bug: 1188532
Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Function get_remaining_quota returns None when quota config
param doesn't match a pattern (i.e. it's broken). It messes up
the logic of its callers and leads to unpredictible behavior of
the entire system.
This code raises InvalidOptionValue exception if config is broken
Change-Id: Ib08d2114f460d127aacf580fc519a58d6e8eba16
Currently in Glance v2 if, for some reason, an image has more
properties than the image_propery_quota allows (e.g. the quota was
lowered after the image was created with properties) then any request
to modify or delete existing properties results in a 413 overlimit
error. Ideally a user should be able to remove properties or any other
action except for adding a property when they are over their quota for
a given image.
This commit does this by adding a new member to the quota.ImageProxy
class to "remember" what properties were already present in an image
*before* any new property operations are preformed on the image. After
the new property operations are performed the quotas are checked (before
writing the image info to DB) only if any new properties have been
added.
This commit does not use a subclass of ExtraPropertiesProxy to check
property quotas (in the __setitem__ method) because Glance does not
implement the JSON-patch RFC correctly - in Glance all operations in a
patch are applied and the quota checked only after all operations have
been applied (RFC requires that operations be applied sequentially and
fail on the first failure). Therefore it is possible for the quota to be
temporarily exceeded when a patch is being applied and therefore we
cannot check for quotas as they are being added - we have to wait until
all patch operations have been completed.
Also, as per review discussions in IRC, a new file:
glance/tests/integration/v2/test_property_quota_violations.py
has been added to perform image property quota related tests (because
the functional test framework is slow).
Change-Id: Icf1b46343463791ed3d2f3ce376f11e409e792ff
Closes-bug: #1258331
Author: David Koo <david.koo@huawei.com>
Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.
Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.
This is second part of this change which covered DB API, domain and REST
API.
Partially-Implements BP: image-location-status
Change-Id: I744679e2dadbaec099aef33d8c5a3fe4ecf96865
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
upload_utils is concatenating str and Message objects which doesn't work
on python 2, so change the str to a unicode object.
test_quota.py was casting Message objects to str which fails for the
same reason, so rather than cast to str, cast using six.text_type.
See oslo commit 2cfc1a78d8063cf20083cf7df796d730a576551c for history and
details on why the Message object doesn't support __str__ casting.
This is a small part of a larger effort that the oslo team will be
rolling out with graduation and adoption of the oslo-i18n library.
Closes-Bug: #1334774
Change-Id: I9d3f29c661f1afffabbbbb499f48d37ef5a8fcee
The following replacements were done in unit tests to have
clearer messages in case of failure:
- assertTrue(* is None) with assertIsNone
- assertTrue(* is not None) with assertIsNotNone
- assertTrue(* in *) with assertIn
- assertTrue(* not in *) with assertNotIn
- assertFalse(* in *) with assertNotIn
- assertTrue(* == *) with assertEqual
- assertTrue(* != *) with assertNotEqual
Change-Id: I0c47f991c3974e441335e71c9d26fab8a127f2ca
user_storage_quota now accept value in B, KB, MB, GB
or TB. The unit is optional. If no unit is specified
Bytes is used as default.
DocImpact
Change-Id: Icc3f672869a5947cbcae38de92993c88ce0ef4e1
Closes-Bug: #1261747
Replace mox testing library by mock one in the file
glance/tests/unit/test_quota.py
Implements: blueprint replace-mox-with-mock
Change-Id: I4577c4debe92846ba3dd0ecf3c6bef96b4ca352c
Based on current design, the image size should be updated to
None if there is no location assocaited with the image. This
patch fixes this issue and introduces some related changes.
Closes-Bug: #1263729
Change-Id: I893468f1dc320ea9434f07c3a32f978cd5941b33
This patch adds the image_location_quota config option. This allows
a deployer to limit the number of image locations allowed on an image.
The default value is 10. Users will only be able to update an image
if the result of the transaction would be under this limit. This
behavior is intended to be similar to 'quota_metadata_items' in Nova.
We have implemented the same limit for image properties, members and tags,
see:
https://review.openstack.org/#/c/56981/https://review.openstack.org/#/c/59563/https://review.openstack.org/#/c/58056/
docImpact
Closes-Bug: #1259031
Change-Id: I0e0033a1d11be267144e8e10b730bb6331263d4c
When uploading data of a newly created image to a local filesystem
store, if the image data size exceeds the quota allocated to the user
then the operation fails with a "500 Internal Server Error".
This bug is triggered only when using chunked transfers to upload the
image data (curl and glanceclient both use chunked transfers).
When using chunked transfers the 'Content-Length' header is not set and
so the quota checking code in start of quota.ImageProxy.set_data() is
skipped. Furthermore, because of the missing size info, the store
downloads the entire image data without checking for the quota (the
quota is checked only after the image data has been downloaded and the
size determined).
If after the image data has been downloaded it is found that the quota
has been exceeded, then the StorageQuotaFull exception is raised and a
call to store.safe_delete_from_backend() made to clean up the image
data. But the 'location' and 'context' parameters to this call are given
in the wrong order and so this operation raises an (uncaught) exception,
giving rise to this bug.
This bug was not caught by existing tests because existing quota tests
don't use chunked transfer.
This commit fixes the order of the parameters and adds quota tests using
chunked transfers.
Change-Id: Ib8ab11f2115e3aead98af70788eaa45c2c33219c
Closes-bug: #1265446
This commit makes the glance code base E125 and E126 compliant :
* E125 continuation line does not distinguish itself from next logical line
* E126 continuation line over-indented for hanging indent
Change-Id: I7120149bedb665fb66320498fe98948602a6cd52
Closes-bug: #1263437
Based on current implement, user will run into 500 error during
updating locations for a queued image. The root cause is we're
trying to calculate the overall size with current image size and
the number of locations. This fix will get the correct image size
from backend store if current image size is not available.
Fixes bug 1257507
Change-Id: Ia4ac73e36211d475752ee9f4995fc3768fb2981a
There were two test cases in test_quota that shared the same name, but
did almost the same thing. After a bit of digging, it looked like most
of the test cases did the almost same thing, so I refactored the common
bits and fixed the test case name duplication.
Partial-Bug: #1259036
Change-Id: I828eea570d358d98520c7201454c03ba174a35ad
This patch adds the image_member_quota config option. This allows a deployer
to limit the number of image members allowed per image. The default value
is 128, to be consistent with other quota defaults. Users will only be able
to update an image if the result of the transaction would be under this limit.
This is for both Glance v1 and v2
Fixes bug 1252459
docImpact
Change-Id: I02f5e82ca4c4acf6cd7bc94f9b99086054a616c9
This patch adds the image_tag_quota config option. This allows a deployer
to limit the number of image tags allowed on an image for v2. The default value
is 128. If an image is somehow over the limit, tags can still be removed but no
more may be added.
Fixes bug 1252337
docImpact
Change-Id: I2036e2a88601d7c5aa85fad32d90fe0ed30b84c8
This patch adds the image_property_quota config option. This allows a deployer
to limit the number of image properties allowed on an image. The default value
is 128, as is currently the limit enforced by nova. Users will only be able to
update an image if the result of the transaction would be under this limit.
This behavior is intended to be similar to 'quota_metadata_items' in nova.
This is for both Glance v1 and v2.
Fixes bug 1251518
docImpact
Change-Id: I4aa9504deae836404f11c9ada71a91f85caeba4c
Instead of globally ignoring Pyflakes and Hacking
warnings, blacklist explitly only those that trigger
numerously. Fix the rest alongway and start to gate
on those that are now passing.
Change-Id: Ia19dc708cf0ccab2da5b46d1652f16e901499c24
This patch adds a storage quota that is applied against the sum total
of a users storage consumption against all configured storage systems.
A single quota is applied to all users via the configuration option
'total_storage_quota'. Most of the patch is about enforcement so
when a separate service for quota management emerges in OpenStack the
per user value to enforce can be obtained from that service but the
enforcement code will remain the same.
blueprint glance-basic-quotas
docImpact
Change-Id: I251832f7372c70942be6f0c6aa12285145dd7c18
Takashi Kajinami