Made `centralized_db` cache driver as default driver so that we
can test it using tempest jobs in gate.
Implements blueprint centralized-cache-db
Depends-On: https://review.opendev.org/c/openstack/devstack/+/907110
Change-Id: Id94e93e3ba3fc207b39c7dbff92495805aa0f6f9
If we have import methods enabled, we must have a staging directory
in order to complete those operations. This adds a startup check
for the staging directory with a warning log message if it is not
found.
Change-Id: Iac31d7f15ec6783a59b5e314883fb58f88fe2fd1
Non existing property protection file raises 500 Internal server
error. If admin/operator specifies non existing property protection
file in glance-api.conf. For example:
[Default]
property_protection_file = non_existing_file.yaml
property_protection_rule_format = policies
Then create or update image. The 500 Internal server error will be
raised. But the expected result is 400 Bad Request.
Add the except of InavlidPropertyProtectionConfiguration exception,
and return 400 Bad Request.
The previous code lacks the specific process of
PropertyProtectionConfiguration exception, so it returned the common
500 error. Now, it returns 400 Bad Request.
Closes-Bug: #1905672
Change-Id: Id77010aac04aa9a4961c5bcafbf12694a0fe17c6
Babel is not needed as requirement, remove it.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014227.html
Commit 36d46bb0b1 removed the
openstack.common.gettextutils module, which used the babel package.
Commit a8fb4e2587 removed the babel.cfg
file.
It is therefore not needed to mock babel.localedata.locale_identifiers
in _set_expected_languages.
Change-Id: I05eaaf39c706e8ef43fe06611ea825a145caf015
We also update docs since guidance has necessarily changed here.
Change-Id: I7c24a1aa3545f3499a7a2ce30b73e2656666c764
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This is a rather beefy change due to the number of usages of this
import. The changes are trivial though.
Change-Id: I7badeeaca438b0291f4ed86670e7f217e6372c61
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We also remove hacking tests for this, along with those for iterkeys and
itervalues (no usage of these).
Change-Id: If5b46580078eb756651ac6118f502eccdc693646
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.
This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.
Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
As eventlet ssl termination is broken with python 3 and
we won't be supporting python 2.7 anymore we will just
remove ssl termination to glance-api and expect the
termination being handled by something else, like HAProxy.
This patch also removes the broken ssl test job as the
non-existing feature is not broken anymore.
Change-Id: Iaf16dfcfdb3a2c93312dcad1ea1229e6b3c8caaa
In Rocky multiple backend support is added as experimental feature. In
order to take advantage of this feature it is decided to deprecate
work_dir and node_staging_uri configuration options
and reserve two filesystem stores 'os_glance_tasks_store' and
'os_glance_staging_store', which can be used to get rid of initializing
store via internal functions.
These internal stores are considered "reserved stores" by Glance.
For the time being, these are hard-coded as filesystem stores. The
store prefix 'os_glance_' is reserved for internal Glance use and
the glance-api service will refuse to start if a store with this
prefix is included in the enabled_backends config option in
glance-api.conf.
NOTE: Because there are no sensible default values for the location
of the datadir for each of these stores, the operator must define
'os_glance_tasks_store' and 'os_glance_staging_store' in
glance-api.conf configuration file as shown below.
[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir/
[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging/
Each filesystem store must have a unique datadir.
Depends-On: https://review.openstack.org/#/c/639765/
Implements: blueprint rethinking-filesystem-access
Change-Id: I86ec513c5fc653dbb97b79d953d8430f014e684f
Added new periodic job which will run as per interval set using
'cache_prefetcher_interval' configuration option and fetch images
which are queued for caching in cache directory.
DocImpact
Change-Id: If2875f7a215aca10a6ed2febc89b02219f90a10d
In order to run the unit and functional Glance tests on Windows, we
have to:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
* we'll use subprocess.Popen when spinning up http servers.
* for the really simple ones defined in the test helpers, we'll just
use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
as well
* skip log rotation test, we can't move in-use files. Log rotation can
be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
markers rely on ordering, which can be flipped if the timestamps are
identical (can happen in case of resources created one after the
other, not sure yet if this happens really fast or the clock isn't
accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
in VMs).
blueprint windows-support
Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
The requirements/upper-constraints file was modified to allow WebOb
1.8.1 by commit 88bafa11deb9bd7595983d97ffabca338f073ba3. This
patch simplifies some Glance code that could handle both WebOb
1.7.4 and 1.8.1 so that it now only handles the latter, and updates
the glance/requirements.txt and glance/lower-constraints.txt to
reflect that we support WebOb 1.8.1+ only.
Change-Id: I03e03013927cc5434aa0d97657d5e7efd8223ee5
Closes-bug: #1770410
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
This change removes option to configure Images API v1
This change removes Images API v1 endpoints from the router
This change removes all v1 tests
This change removes the v1 dependant glance-cache-manage command
This change does not remove all v1 codebase. Further cleanup and
decoupling will be needed.
Change-Id: Ia086230cc8c92f7b7dfd5b001923110d5bc55d4d
WebOb 1.8.1 removes a function (deprecated for a long time) that
Glance uses for negotiating 'Accept-Language' headers. The removed
function is replaced by a function that implements the "Lookup"
scheme defined in RFC 4647.
Glance wraps the webob.Request object and provides a convenience
method around the old function. This patch uses the new webob
function in the convenience method and wraps it in such a way that
the convenience method preserves its current behavior (at least with
respect to the Glance tests). But ... "Lookup" is compliant with RFC
7231, whereas the old function was not. This means that given
sufficiently complex Accept-Language header values (containing
q-values), the best match under webob 1.8.1 will be different from
the best match under webob 1.7.x. Given that the best match will,
however, be RFC 7231 compliant, this is an acceptable change.
One further complication is that the lookup() function is not
available in webob 1.7.x. Thus this patch handles both versions. I
suggest that as soon as WebOb===1.8.1 is merged into
lower_constraints, the code be simplified to handle 1.8.x only, and
the following release note be submitted with that patch:
Negotiation of the 'Accept-Language' header now follows the "Lookup"
matching scheme described in `RFC 4647, section 3.4
<https://tools.ietf.org/html/rfc4647.html#section-3.4>`_. The
"Lookup" scheme is one of the algorithms suggested in `RFC 7231,
section 5.3.5
<https://tools.ietf.org/html/rfc7231.html#section-5.3.5>`_. (This is
due to a change in an underlying library, which previously used a
matching scheme that did not conform to `RFC 7231
<https://tools.ietf.org/html/rfc7231.html>`_.)
Change-Id: Ib9735b394a134de9a53d577b4d8b5a9c760b7780
Closes-bug: #1765748
With modern servers it's not uncommon to see environments that has
dozens of logical processors. Glance defaults workers to number of
these logical processors seen by OS and each worker spins up pool
of 1000 threads. Having dozens of these thread pools consuming
resources is total overkill for most of the environments.
This change limits the amount of workers used by default to 8 which
should be plenty for most of the deployments. This limitation does
not affect if more workers is specifically configured in the config
file.
Change-Id: I6fcf2f8a416e34c0c719e1abd73f0525e686be5e
Closes-Bug: #1748916
Since the WebOb 1.7 release webob doesn't natively support receiving
chunked transfer encoding bodies. [1] When glance is run under the
eventlet wsgi server this was fine, because eventlet will dechunk the
input on read() (or readline()) calls, so from the webob perspective
it's just a file object. However, the effort to remove the dependence
on using eventlet as the web server and deploy glance as a traditional
wsgi script we lose this mechanism. The wsgi spec doesn't provide a
consistent mechanism
When we run glance under uwsgi the uwsgi server provides an api to read
chunked data. [2] However, we need to explicitly call that api when to
dechunk the data and pass it to glance code which expects a file object.
This commit solves this issue by creating a fake file class that will
call the chunked_read() api from uwsgi on read() calls. This object is
then used if we're running the api code under uwsgi and the body has a
chunked transfer-encoding.
In conjuction with devstack change
Iab2e2848877fa1497008d18c05b0154892941589 this closes glance bug 1703856
[1] https://docs.pylonsproject.org/projects/webob/en/stable/whatsnew-1.7.html#backwards-incompatibility
[2] http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html
Partial-bug 1703856
Co-Authored-By: Chris Dent <cdent@anticdent.org>
Change-Id: Idf6b4b891ba31cccbeb53d373b40fce5380cea64
Currently glance v2 API incorrectly accepts ‘Content-Range’ header
for random image access and does not set response headers.
As per rfc7233, ‘Range’ requests should be accepted and ‘Content-Range’
must be returned in the response headers.
This patch enables Glance v2 API to accept the more appropriate ‘Range’
requests and sets ‘Content-Range’ response header.
For backward compatibility with pre-Pike Glance clients, the incorrect
'Content-Range' header will be accepted silently in perpetuity.
Thus this patch contains tests for 'Content-Range' in requests to
prevent regressions.
DocImpact
Implements lite-spec I5bdadde682a0c50836bd95e2a6651d6e7e18f172
Closes-Bug: #1677391
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ib7ebc792c32995751744be3f36cbc9a0c1eead2a
WebOb 1.7 changed [0] how request bodies are determined to be
readable. Prior to version 1.7, the following is how WebOb
determined if a request body is readable:
#1 Request method is one of POST, PUT or PATCH
#2 ``content_length`` length is set
#3 Special flag ``webob.is_body_readable`` is set
The special flag ``webob.is_body_readable`` was used to signal
WebOb to consider a request body readable despite the content length
not being set. #1 above is how ``chunked`` Transfer Encoding was
supported implicitly in WebOb < 1.7.
Now with WebOb 1.7, a request body is considered readable only if
``content_length`` is set and it's non-zero [1]. So, we are only left
with #2 and #3 now. This drops implicit support for ``chunked``
Transfer Encoding Glance relied on. Hence, to emulate #1, Glance must
set the the special flag upon checking the HTTP methods that may have
bodies. This is precisely what this patch attemps to do.
[0] https://github.com/Pylons/webob/pull/283
[1] https://github.com/Pylons/webob/pull/283/files#diff-706d71e82f473a3b61d95c2c0d833b60R894
Closes-bug: #1657459
Closes-bug: #1657452
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: I19f15165a3d664d5f3a361f29ad7000ba2465a85
Unit test [0] tries to assert that 'None' is returned when
'Accept-Language' in request contains an unsupported language.
To test that, we used 'zh' as the test language. However, 'zh'
is a supported language and due to some missing functionality
in oslo.i18n, it didn't show up as a valid locale.
I00601dec28adbef05e9ab0ebb6d291d0eee8becb fixes the issue in
oslo.i18n and now returns 'zh' locales as well. Hence, our test
fails. So, stop using 'zh' to test unsupported language.
[0] glance.tests.unit.common.test_wsgi.RequestTest.test_request_match_language_unexpected
Change-Id: I87b38ee9a3b46a117b2cdbcb8d00f3a87e9af111
When non-ascii characters is passed as filter parameter to "image-list"
and "md-namespace-list" APIs then it returns HTTP 500 if glance service
is running using Python2. This is because query parameters passed to
the urlparse.urlencode method raises UnicodeEncodeError for non-ascii
characters.
Overridden params property of webob.request.BaseRequest in wsgi layer
Request class to avoid encoding of request params at several places in
case of Py2. Added a new class attribute 'encoded_params' which is
initialized to encoded params dict when params property is called for
the first time and in the subsequent calls this attribute is returned
without requiring you to worry whether the params value are encoded
or not.
Closes-Bug: #1521193
Change-Id: If4497d42820aa4b8070666d2a1a4413411557f12
Currently Glance does not send Partial response codes while
handling HTTP range requests. Also, content length is not
appropriately set.
This patch is to send partial response code and to set the correct
content length based on the range request for image download.
Upon success status code 206 is sent and the content length is set to the
requested range.
Upon failure, there can be 2 cases:
* If the HTTP range request for the image download is bad (For example,
requesting download of range of bytes 10 to 50 bytes when there are only 48
bytes), status code is set to 416 and HTTPRequestRangeNotSatisfiable is
raised.
* If the content range is valid, but the request is not satisfiable due to
glance_store side erros or privacy issues, appropriate exceptions are
raised.
APIImpact
DocImpact
Closes-Bug: #1417069
Closes-Bug: #1624508
Closes-Bug: #1399851
Closes-Bug: #1618928
Change-Id: I3cd47b998be79604511b3cd4879209820cf776b7
There are several places in the source code where
HTTP response codes are used as numeric values.
These values are used from six.moves and the
numeric values are replaced by constants.
All of the used status codes were replaced with symbolic constants
from six.moves.http_client.
More about six.moves.http_client can be found at [2],
under the table "Supported renames:".
Also, this change improves code readibility.
This patchset does not extract numeric values from
the code itself, but it can be found at [1].
[1]: Ib9e26dcea927e96e65c626c18421621d3a29a64d
[2]: https://pythonhosted.org/six/#module-six.moves
Change-Id: Idfc7b043552f428f01ac3e47b270ee0639a8f5bc
Closes-Bug: #1520159
Some headers in v2 are unicode string and they cannot be
recognized by mod_wsgi. So encode all headers as utf-8 to allow
users to use glance under Apache mod_wsgi or other wsgi
frameworks. v1 functionality already encoded headers as utf-8.
APIImpact
Closes-bug: #1508880
Change-Id: Ifa83ed3ab44a4b7f98e438c74d889a956703cf4a
Some issues:
- the usual str vs bytes thing;
- 'reload' no longer exists in Python 3, but is available through six;
- 'min_ram' and 'min_disk' have to be integers because Python 3 cannot
compare integers and strings.
Change-Id: I8a65da8bad7ba14b6af32348c30366d0db49212c
* Replace filter() with list-comprehension to get a list on Python 3.
* HTTP body type is bytes: use byte strings for body.
Change-Id: Ia7fba4b075cdedd4581e4e8503f5220c1e6198d7
Eventlet can use logger directly starting from v 0.17.2.
So we can delete usage of WritableLogger in Glance.
Closes-Bug: #1440773
Change-Id: I3f393bcdaafc96f098fe7fafd425c97211051815
Request-id is not getting returned if glance throws 500 error
because context middleware 'process_response' method is not
getting called.
Catching generic exception in wsgi Resource class, and returning
'webob.exc.HTTPInternalServerError' object as response.
Context middleware adds request-id header to this response object.
Closes-Bug: #1480196
Change-Id: I5ed9f64e07c5624f816088113ff67cb03cc8e3f2
This code prevents the occurrence of 501 error when
accessing the server with a non-existent method.
Change-Id: I1105ed1399f1bcde7f1f970abcad929585fd012e
closes-bug: #1470192
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 9292
real 1m0.063s
Setting 'client_socket_timeout = 0' means do not timeout.
DocImpact
Closes-bug: 1371022
Change-Id: I9e7edcbf25ece61dc16b8cd5a8bef5ed9a14e3d6
* Replace unicode with six.text_type
* Replace "raise a, b, c" with six.reraise(a, b, c)
* Replace dict.iteritems() with six.iteritems(dict)
* Replace StringIO.StringIO with six.BytesIO for image content
* Get AssertionError from builtins, don't use the exceptions module
which was removed in Python 3
* Open configuration file in text mode (not in binary mode) in
functional tests
* Use absolute imports in artifacts_sample and image_artifact plugins
instead of relative imports.
* Fix usage of __import__() function: the level parameter expects a
positive integer, use level=0, not level=-1
* Get the input function from six.moves: use raw_input() on Python 2 and
input() on Python 3. raw_input() was renamed to input() in Python 3.
* Unpacking parameters in function definition is no more possible:
unpack the tuple manually in test_glance_replicator.py
* On Python 3, socket.fromfd() now returns directly the right socket
type. It's not more needed to cast manually the result of
socket.fromfd().
For more information on Python 3, see:
https://wiki.openstack.org/wiki/Python3
Change-Id: Iaa1a392590154058eab4645fb288d1534f66e528
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
Pick up changes to socket configuration on reload.
To avoid any interruption to request handling on reload we recycle the
original listening socket rather than closing it (if at all possible).
Closes-bug: 1436272
Change-Id: Ibf0ad60a3a50e60b1bc8b32f9a2e567079a6f821
Presently, the wsgi server allows persist connections. Hence even after
the response is sent to the client, it doesn't close the client socket
connection. Because of this problem, the green thread is not released
back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
DocImpact:
Added http_keepalive option (default=True).
SecurityImpact
Closes-Bug: #1361360
Change-Id: I93aaca24935a4f3096210233097dd6b8c5440176
* Identify and handle requests with empty body
and 'transfer-encoding' request header
set to value 'chunked'
Change-Id: I54462dad76bdbf7c069ae3031237d98702a77e40
Closes-Bug: #1249501
Args of assertEquals method in glance.tests are arranged
in wrong order. In result when test fails it shows incorrect
information about observed and actual data. It's found more
than 2000 times. Right order of arguments is "expected, actual".
Change-Id: Ifea4809f5a696b608a19e36a9ed9e5eec46c8a21
Co-Authored-By: Li Yingjun <liyingjun1988@gmail.com>
Closes-Bug: 1277104
Abhishek Kekane