This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.
Note that this disables checking of the global size quota if keystone
quotas are enabled.
Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.
Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
Now that glance is using properties in the os_glance namespace for
internal purposes, we should exclude the counting of these from the
enforced image property quota.
Change-Id: I5fbe5eb12fd34e054137732a02c4cc5b687e7c77
Related-Bug: #1912001
This teaches glance-api how to do async threading things when it is
running in pure-WSGI mode. In order to do that, a refactoring of things
that currently depend on eventlet is required.
It adds a [wsgi]/task_pool_threads configuration knob, which is used
in the case of pure-WSGI and native threads to constrain the number
of threads in that pool (and thus the task parallelism). This will
allow tuning by the operator, but also lets us default that to just
a single thread in the backport of these fixes so that we can avoid
introducing a new larger footprint in the backport unexpectedly.
Partial-Bug: #1888713
Depends-On: https://review.opendev.org/#/c/742047/
Change-Id: Ie15028b75fb8518ec2b0c0c0386d21782166f759
String interpolation should be delayed to be handled
by the logging code, rather than being done
at the point of the logging call.
See the oslo i18n guideline.
* http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: I400a38a61e1ed6288296e8572218c7642795cbd4
Closes-Bug: #1596829
Glance tests are currently too verbose for subunit and make
them randomly fail under python 3.
Using raw strings should avoid the DeprecationWarnings around
'invalid escape sequence' being used with the re module.
Change-Id: I14c381ac4b17ca4c5755ed78c55dc44362ab37ef
RequestContext.tenant and user fields are deprecated in favor
of project_id and user_id respectively.
This change modifies the glance.context.RequestContext constructor
to transition usage of tenant/user to project_id/user_id until
all tests are moved over to the new attributes. Runtime usage of
the old fiels is updated.
To prevent new code from using the deprecated fields, a warnings
filter is added which will make tests fail if they hit code using
the old fields.
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Change-Id: I351380840308a24769ece93abc6d1a9a6d6aa06f
Import i18n functions from module directly and do not use
global module variables like _ = i18n._. It makes code shorter
and cleaner. It also to detect cases when these functions are not
used in module.
Change-Id: Iaa593ac1f2dd15cbcad049bef6ba68f1cfa610da
Replace all the ":return:" with ":returns:"
Replace all the ":returns " wtih ":returns:"
Replace all the ":returns :" with ":returns:"
Add the changes for ":raise" also, change ":raise" to ":raise:"
Change-Id: I5d0def4665d0cd69b73c3c3826ad8f208e1e970e
Add the ability to dynamically change configuration settings of a running
glance server with no impact to service.
A running glance server consists of a parent/master process and one or
more child/worker processes.
On receipt of a SIGHUP signal the master process will:
* reload the configuration
* send a SIGHUP to the original workers
* start (a potentially different number of) new workers with the new
configuration
* its listening socket will *not* be closed
On receipt of a SIGHUP signal each original worker process will:
* close the listening socket so as not to accept new requests
* complete any in-flight requests
* complete async requests (V1 create with copy-from option and V2 task api)
* exit
This approach is based on nginx's behaviour and avoids some of the
disadvantages of the current oslo Launcher reload:
* Race conditions: Launcher does not shutdown eventlet cleanly, existing
requests can fail.
* If all workers are busy (likely if #workers is small) there can be a
lengthy delay when new requests are not processed.
* Long lived pre-SIGHUP idle client connections can stall request
processing indefinitely.
* Not all parameters can be changed, eg number of workers.
* The wsgi pipeline cannot be changed, eg to enable caching.
Change-Id: I73875c654410f207e3efdb2932147cf26d75b3f1
Partially implements: bp sighup-conf-reload
oslo_config was moved out of the oslo namespace in oslo.config>=1.6.0.
Related-to: blueprint drop-namespace-packages
Change-Id: I30ecbf8f9de77496fcb643c7ad9738d79ad359f0
The builtin _ should not longer be used. Instead _
should be explicitly assigned from glance.i18n.
This patch does the following:
(1) adds these explict assignments to any
glance file that was missing them,
(2) removes the defintion of _ as a builtin from
tox.ini so that a pep8 failure will occur
if _ is not defined,
(3) removes calls to install().
This removes the last direct use of gettextutils by glance.
The indirect uses, via openstack/common, will be removed
in the next sync with oslo.
Change-Id: Ie54e8a67e747e022cc01022dfad35d89686bdfc4
Changing all Glance files to use i18n instead of gettextutils for log
translations.
Finishes bp refactoring-glance-logging
Change-Id: I90d6ab4f7e09f4e7591921f9526de6370ebdb789
This change complements glance change #106776 by raising some logging
levels and fixing some of the translations of logging.
Related to bp refactoring-glance-logging
Closes-Bug: #1336955
Change-Id: Ib547f4603025ebe4f6c055779032ab4064784490
Function get_remaining_quota returns None when quota config
param doesn't match a pattern (i.e. it's broken). It messes up
the logic of its callers and leads to unpredictible behavior of
the entire system.
This code raises InvalidOptionValue exception if config is broken
Change-Id: Ib08d2114f460d127aacf580fc519a58d6e8eba16
This change fixes translations to be using correct translation functions.
Contains few log level drops for too high logging levels.
This PS does not address logging changes for modules under glance/store/
due to the new repo request. After stores are moved to their own repo
new PS will be submitted for those.
This PS does not touch the test code.
Separate PS will be submitted for log lever raises withing codebase
as requested during the spec review.
Related to bp refactoring-glance-logging
Change-Id: I683c35face8d5b7056ef7fedcd75deb63dbbf8e2
user_storage_quota now accept value in B, KB, MB, GB
or TB. The unit is optional. If no unit is specified
Bytes is used as default.
DocImpact
Change-Id: Icc3f672869a5947cbcae38de92993c88ce0ef4e1
Closes-Bug: #1261747
In some cases the exception context can be cleared,
resulting in None being attempted to be re-raised after
an exception handler is run.
Change-Id: I16811ba8f0bfaa15199a648748bbc7d7a6a500bc
Partial-Bug:#1281453
When a create call fails due to user_storage_quota
being exceeded, log a warning message.
Closes-Bug: #1270832
Change-Id: Ic7ec266aace03cff0a1f72dc3ba6f17b89f5fd55
Per the hacking guidelines locals() should not be used when creating
messages, but additionally this is preparation for upcoming message
internationalization work.
Change-Id: I9ce616b94faf2d7df895c6bffb94baac8c82c7ce
This patch adds a storage quota that is applied against the sum total
of a users storage consumption against all configured storage systems.
A single quota is applied to all users via the configuration option
'total_storage_quota'. Most of the patch is about enforcement so
when a separate service for quota management emerges in OpenStack the
per user value to enforce can be obtained from that service but the
enforcement code will remain the same.
blueprint glance-basic-quotas
docImpact
Change-Id: I251832f7372c70942be6f0c6aa12285145dd7c18
We currently have a mix of 'except X, y:' and 'except X as y:' syntax.
Standardize on the newer 'except as' form.
This patch was generated automatically using:
find * -type f -name 'glance-*' ! -name '*\.conf' ! -name '*\.ini' -o \
-name '*\.py' | xargs 2to3 -f except | patch -p0
Addresses bug 1160331.
Change-Id: Ica8d1303973af64d0406457267ee75f608e869d0
Patchset addresses reviewer's comments, rebase against master,
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Change-Id: I94367461505778973528eb8835d991f4fb796dab
1) caching_iter doesn't handle backend exceptions:
caching_iter assumes any exception that occurs is the result of being
unable to cache. Hence the IOError raised from size_checked_iter, which
indicates a problem with the backend, means the caching_iter will
continuing trying to serve non-existent data. The exception was not
been re-raised in this case, making wsgi keep the connection open and
clients stuck forever waiting for more data.
Raising a GlanceException in size_checked_iter rather than an IOError
allows caching_iter to distinguish between a problem fetching data, and
a problem writing to the cache.
2) Checksum verification happens after cache commit rather than before:
This block was outside the context manager block which meant the
GlanceException was not caught by open_for_write and the rollback didn't
happen. This resulted in an error been logged, but the bad image still
placed in and subsequently served from the cache.
Also:
* Fix test_gate_caching_iter_bad_checksum - the loop to consume the
iterator in was in a subroutine that never got called.
* Move test_gate_caching_iter_(good|bad)_checksum into
ImageCacheTestCase to excercise both the sql and xattr drivers.
* Remove invalid registry_host/registry_port params from
TestImageCacheXattr/TestImageCacheSqlite setup which caused a failure
when testing the file on it's own using nosetests.
Fixes bug 1045792
Change-Id: I8aedec347e7f50566c44c5b6c6db424573c5ebaf
The checked_iter image streaming wrapper should be in a common
place so the v2 API code can use it as well. This moves it to
glance/api/common.py and gives it a slightly more descriptive
name 'size_checked_iter'.
Change-Id: I9fbf2c76666bfc0b9227a6a6c4d824367526e56a