Made provision to migrate existing cache records from
sqlite to centralized db and delete those from sqlite
after successful migration.
Implements blueprint centralized-cache-db
Change-Id: I2a840915bb6d9d72d6379cae09ebec0e51a4a6dd
If we have import methods enabled, we must have a staging directory
in order to complete those operations. This adds a startup check
for the staging directory with a warning log message if it is not
found.
Change-Id: Iac31d7f15ec6783a59b5e314883fb58f88fe2fd1
Non existing property protection file raises 500 Internal server
error. If admin/operator specifies non existing property protection
file in glance-api.conf. For example:
[Default]
property_protection_file = non_existing_file.yaml
property_protection_rule_format = policies
Then create or update image. The 500 Internal server error will be
raised. But the expected result is 400 Bad Request.
Add the except of InavlidPropertyProtectionConfiguration exception,
and return 400 Bad Request.
The previous code lacks the specific process of
PropertyProtectionConfiguration exception, so it returned the common
500 error. Now, it returns 400 Bad Request.
Closes-Bug: #1905672
Change-Id: Id77010aac04aa9a4961c5bcafbf12694a0fe17c6
Till now glance was dependent on periodic job to get image
cached locally and also has dependency on ``cachemanage``
middleware for the same.
This patch removes the periodic job and dependency of
``cachemanage`` middleware and initiates immediate caching
of an image via ``PUT /cache/image_id`` API call.
Co-Author: Dan Smith <dms@danplanet.com>
Implements: blueprint instant-caching
Change-Id: I9ab3f1b7595e22dbb03af95168314352a44eb930
Glance uses http-proxy-to-wsgi middleware from oslo.middleware in
its pipeline in order to efficiently forward request headers in case
of load balancer style deployments. Hence, the configuration option
``secure_proxy_ssl_header`` was marked as deprecated.
This patch removes the option and the support for it within Glance
to entirely rely on oslo middleware. This will ensure that the related
headers set by oslo.middleware:HTTPProxyToWSGI is never modified in
Glance.
Change-Id: I11d41bb736bbfd90030d88245c11642823e4c400
Closes-Bug: 1673908
This was already fixed in 4889dc1814 but
we did not enforce this rule and reintroduced "bad" string
interpolations. This patch adds a hacking rule to prevent us from doing
this again in the future.
Change-Id: I96d9a157d3887286542859d67138ffdae5a589f1
We also update docs since guidance has necessarily changed here.
Change-Id: I7c24a1aa3545f3499a7a2ce30b73e2656666c764
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We also remove hacking tests for this, along with those for iterkeys and
itervalues (no usage of these).
Change-Id: If5b46580078eb756651ac6118f502eccdc693646
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
As noted in previous discussions, glance should clean its staging
directory on startup. This is important for scenarios where we
started an import operation, but failed in the middle. If, when we
recover, the image has already been deleted from the database, then
we will never remove the (potentially very large) residue from disk
in our staging directory.
This is currently a problem with web-download, but will also occur
with glance-direct once we have the non-shared distributed import
functionality merged.
Closes-Bug: #1913625
Change-Id: Ib80e9cfb58680f9e8ead5993dc206f4da882dd09
glance-cahce-manage is broken due to circilar imports.
Fixed the same by importing 'prefetcher' module right before
'Prefetcher' object is initialized.
Closes-Bug: #1888349
Change-Id: I57d473572ca0a341082bacc3883cd9f763d77fa7
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.
This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.
Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
When multiple stores are being used with rbd backend, Glance tries
to get fsid of the cluster using rados library.
If even one of the rbd backends is unavailable, glance-api wsgi fails
to start properly, but continue listening on the TCP socket and is not
responding at any request until the timeout.
This change moves socket configuration after the store initialization
to make sure that failed Glance API process will not be
considered as up by the haproxy TCP check.
Change-Id: I4298bb835ec60b79a0010e52318ada19eb24ad80
Related-Bug: #1861723
As eventlet ssl termination is broken with python 3 and
we won't be supporting python 2.7 anymore we will just
remove ssl termination to glance-api and expect the
termination being handled by something else, like HAProxy.
This patch also removes the broken ssl test job as the
non-existing feature is not broken anymore.
Change-Id: Iaf16dfcfdb3a2c93312dcad1ea1229e6b3c8caaa
In Rocky multiple backend support is added as experimental feature. In
order to take advantage of this feature it is decided to deprecate
work_dir and node_staging_uri configuration options
and reserve two filesystem stores 'os_glance_tasks_store' and
'os_glance_staging_store', which can be used to get rid of initializing
store via internal functions.
These internal stores are considered "reserved stores" by Glance.
For the time being, these are hard-coded as filesystem stores. The
store prefix 'os_glance_' is reserved for internal Glance use and
the glance-api service will refuse to start if a store with this
prefix is included in the enabled_backends config option in
glance-api.conf.
NOTE: Because there are no sensible default values for the location
of the datadir for each of these stores, the operator must define
'os_glance_tasks_store' and 'os_glance_staging_store' in
glance-api.conf configuration file as shown below.
[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir/
[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging/
Each filesystem store must have a unique datadir.
Depends-On: https://review.openstack.org/#/c/639765/
Implements: blueprint rethinking-filesystem-access
Change-Id: I86ec513c5fc653dbb97b79d953d8430f014e684f
Added new periodic job which will run as per interval set using
'cache_prefetcher_interval' configuration option and fetch images
which are queued for caching in cache directory.
DocImpact
Change-Id: If2875f7a215aca10a6ed2febc89b02219f90a10d
This change allows glance-api to use multiple workers on Windows
by not using fork, which is unavailable. Instead, we're duplicating
sockets and passing the handles through pipes. Also, instead of
using process groups, we have to go with Windows job objects.
Note that this doesn't change the workflow for other platforms.
A subsequent change will allow the tests to run on Windows.
blueprint windows-support
Change-Id: Ic786199844e1d804962172286905036d93a4ed14
This change will allow glance services to run on Windows, using
eventlet wsgi for API services.
This change will:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* avoiding unavailable signals
* avoid renaming in-use files or leaking handles
* update the check that ensures that just one scrubber process may
run at a time. We can't rely on process names as there might be
wrapper processes that have similar names (no she-bangs on Windows,
so the scripts are called a bit differently). We'll use a global
named mutex instead.
A subsequent change will leverage Windows job objects as a
replacement for process groups, also avoiding forking when spawning
workers.
At the moment, some Glance tests cannot run on Windows, which is
also covered by subsequent patches.
DocImpact
blueprint windows-support
Change-Id: I3bca69638685ceb11a1a316511ad9a298c630ad5
The requirements/upper-constraints file was modified to allow WebOb
1.8.1 by commit 88bafa11deb9bd7595983d97ffabca338f073ba3. This
patch simplifies some Glance code that could handle both WebOb
1.7.4 and 1.8.1 so that it now only handles the latter, and updates
the glance/requirements.txt and glance/lower-constraints.txt to
reflect that we support WebOb 1.8.1+ only.
Change-Id: I03e03013927cc5434aa0d97657d5e7efd8223ee5
Closes-bug: #1770410
Made provision for multi-store support. Added new config option
'enabled_backends' which will be a comma separated Key:Value pair
of store identifier and store type.
DocImpact
Depends-On: https://review.openstack.org/573648
Implements: blueprint multi-store
Change-Id: I9cfa066bdce51619a78ce86a8b1f1f8d05e5bfb6
WebOb 1.8.1 removes a function (deprecated for a long time) that
Glance uses for negotiating 'Accept-Language' headers. The removed
function is replaced by a function that implements the "Lookup"
scheme defined in RFC 4647.
Glance wraps the webob.Request object and provides a convenience
method around the old function. This patch uses the new webob
function in the convenience method and wraps it in such a way that
the convenience method preserves its current behavior (at least with
respect to the Glance tests). But ... "Lookup" is compliant with RFC
7231, whereas the old function was not. This means that given
sufficiently complex Accept-Language header values (containing
q-values), the best match under webob 1.8.1 will be different from
the best match under webob 1.7.x. Given that the best match will,
however, be RFC 7231 compliant, this is an acceptable change.
One further complication is that the lookup() function is not
available in webob 1.7.x. Thus this patch handles both versions. I
suggest that as soon as WebOb===1.8.1 is merged into
lower_constraints, the code be simplified to handle 1.8.x only, and
the following release note be submitted with that patch:
Negotiation of the 'Accept-Language' header now follows the "Lookup"
matching scheme described in `RFC 4647, section 3.4
<https://tools.ietf.org/html/rfc4647.html#section-3.4>`_. The
"Lookup" scheme is one of the algorithms suggested in `RFC 7231,
section 5.3.5
<https://tools.ietf.org/html/rfc7231.html#section-5.3.5>`_. (This is
due to a change in an underlying library, which previously used a
matching scheme that did not conform to `RFC 7231
<https://tools.ietf.org/html/rfc7231.html>`_.)
Change-Id: Ib9735b394a134de9a53d577b4d8b5a9c760b7780
Closes-bug: #1765748
With modern servers it's not uncommon to see environments that has
dozens of logical processors. Glance defaults workers to number of
these logical processors seen by OS and each worker spins up pool
of 1000 threads. Having dozens of these thread pools consuming
resources is total overkill for most of the environments.
This change limits the amount of workers used by default to 8 which
should be plenty for most of the deployments. This limitation does
not affect if more workers is specifically configured in the config
file.
Change-Id: I6fcf2f8a416e34c0c719e1abd73f0525e686be5e
Closes-Bug: #1748916
openstack/oslo.serialization made a change in commit
c1a7079c26d27a2e46cca26963d3d9aa040bdbe8 which changed how
jsonutils.to_primitive() handles Exception objects. Previously the
actual Exception object would be returned but now it returns a 'repr'
of the object. This was done because CircularReferences were occuring
in some use cases with Exceptions and also in version 3.0 of
oslo.serialization they were going to raise a ValueError() for
unhandled types and this change caused Exceptions to now be handled.
The code previously would call the serialization function and if any
unknown Exception occurred it would just return action_result. With
the change to oslo.serialization an exception was no longer being
raised and thus no Exception would be raised and caught.
Now check if action_result is an Exception and if so return it.
Closes-Bug: #1728368
Change-Id: Ic374ec891a65f603433091cdae27b0d4aac8362f
Since the WebOb 1.7 release webob doesn't natively support receiving
chunked transfer encoding bodies. [1] When glance is run under the
eventlet wsgi server this was fine, because eventlet will dechunk the
input on read() (or readline()) calls, so from the webob perspective
it's just a file object. However, the effort to remove the dependence
on using eventlet as the web server and deploy glance as a traditional
wsgi script we lose this mechanism. The wsgi spec doesn't provide a
consistent mechanism
When we run glance under uwsgi the uwsgi server provides an api to read
chunked data. [2] However, we need to explicitly call that api when to
dechunk the data and pass it to glance code which expects a file object.
This commit solves this issue by creating a fake file class that will
call the chunked_read() api from uwsgi on read() calls. This object is
then used if we're running the api code under uwsgi and the body has a
chunked transfer-encoding.
In conjuction with devstack change
Iab2e2848877fa1497008d18c05b0154892941589 this closes glance bug 1703856
[1] https://docs.pylonsproject.org/projects/webob/en/stable/whatsnew-1.7.html#backwards-incompatibility
[2] http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html
Partial-bug 1703856
Co-Authored-By: Chris Dent <cdent@anticdent.org>
Change-Id: Idf6b4b891ba31cccbeb53d373b40fce5380cea64
When sending a SIGTERM to the main glance-api process,
it was sending a SIGTERM to its children
but then also re-spawning its first dead child.
Closes-bug: #1714240
Change-Id: Ibef426c198d287bbdac4e764fd654edba4b7c2d6
In the file common/wsgi.py and common/config.py, the unused import
is still existed, This patch is to remove the redundant codes.
Change-Id: I7869121a2fc11b44f81b03adfd9b5807e8d08ce7
Currently glance v2 API incorrectly accepts ‘Content-Range’ header
for random image access and does not set response headers.
As per rfc7233, ‘Range’ requests should be accepted and ‘Content-Range’
must be returned in the response headers.
This patch enables Glance v2 API to accept the more appropriate ‘Range’
requests and sets ‘Content-Range’ response header.
For backward compatibility with pre-Pike Glance clients, the incorrect
'Content-Range' header will be accepted silently in perpetuity.
Thus this patch contains tests for 'Content-Range' in requests to
prevent regressions.
DocImpact
Implements lite-spec I5bdadde682a0c50836bd95e2a6651d6e7e18f172
Closes-Bug: #1677391
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ib7ebc792c32995751744be3f36cbc9a0c1eead2a
WebOb 1.7 changed [0] how request bodies are determined to be
readable. Prior to version 1.7, the following is how WebOb
determined if a request body is readable:
#1 Request method is one of POST, PUT or PATCH
#2 ``content_length`` length is set
#3 Special flag ``webob.is_body_readable`` is set
The special flag ``webob.is_body_readable`` was used to signal
WebOb to consider a request body readable despite the content length
not being set. #1 above is how ``chunked`` Transfer Encoding was
supported implicitly in WebOb < 1.7.
Now with WebOb 1.7, a request body is considered readable only if
``content_length`` is set and it's non-zero [1]. So, we are only left
with #2 and #3 now. This drops implicit support for ``chunked``
Transfer Encoding Glance relied on. Hence, to emulate #1, Glance must
set the the special flag upon checking the HTTP methods that may have
bodies. This is precisely what this patch attemps to do.
[0] https://github.com/Pylons/webob/pull/283
[1] https://github.com/Pylons/webob/pull/283/files#diff-706d71e82f473a3b61d95c2c0d833b60R894
Closes-bug: #1657459
Closes-bug: #1657452
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: I19f15165a3d664d5f3a361f29ad7000ba2465a85
Some configuration options were accepting both IP addresses
and hostnames. Since there was no specific OSLO opt type to
support this, we were using ``StrOpt``. The change [1] that
added support for ``HostAddressOpt`` type was merged in Ocata
and became available for use with oslo version 3.22.
This patch changes the opt type of configuration options to use
this more relevant opt type - HostAddressOpt.
[1] I77bdb64b7e6e56ce761d76696bc4448a9bd325eb
Change-Id: I06e8cff035ecfaa651e215d7b18de5abc3a273c3
When non-ascii characters is passed as filter parameter to "image-list"
and "md-namespace-list" APIs then it returns HTTP 500 if glance service
is running using Python2. This is because query parameters passed to
the urlparse.urlencode method raises UnicodeEncodeError for non-ascii
characters.
Overridden params property of webob.request.BaseRequest in wsgi layer
Request class to avoid encoding of request params at several places in
case of Py2. Added a new class attribute 'encoded_params' which is
initialized to encoded params dict when params property is called for
the first time and in the subsequent calls this attribute is returned
without requiring you to worry whether the params value are encoded
or not.
Closes-Bug: #1521193
Change-Id: If4497d42820aa4b8070666d2a1a4413411557f12
Currently Glance does not send Partial response codes while
handling HTTP range requests. Also, content length is not
appropriately set.
This patch is to send partial response code and to set the correct
content length based on the range request for image download.
Upon success status code 206 is sent and the content length is set to the
requested range.
Upon failure, there can be 2 cases:
* If the HTTP range request for the image download is bad (For example,
requesting download of range of bytes 10 to 50 bytes when there are only 48
bytes), status code is set to 416 and HTTPRequestRangeNotSatisfiable is
raised.
* If the content range is valid, but the request is not satisfiable due to
glance_store side erros or privacy issues, appropriate exceptions are
raised.
APIImpact
DocImpact
Closes-Bug: #1417069
Closes-Bug: #1624508
Closes-Bug: #1399851
Closes-Bug: #1618928
Change-Id: I3cd47b998be79604511b3cd4879209820cf776b7
Adding improved help texts for the config options to be used for
WSGI servers of the API services.
Change-Id: I8f2159f3f46a223685970548f5b6cb6f9cd34d57
Partial-Bug: #1570946
The HTTP_X_FORWARDED_PROTO handling fails to handle the case of
redirecting the /v1 request to /v1/ because it is handled purely by
routes and does not enter the glance wsgi code. This means a https
request is redirect to http and fails.
oslo.middleware has middleware for handling the X-Forwarded-Proto header
in a standard way so that services don't have to and so we should use
that instead of our own mechanism.
Leaving the existing header handling around until removal should not be
a problem as the worst that will happen is it overwrites an existing
'https' header value set by the middleware.
Closes-Bug: #1558683
Closes-Bug: #1590608
Change-Id: I481d88020b6e8420ce4b9072dd30ec82fe3fb4f7
Starting with 1.0.0 osprofiler release options needed for
its workability are consolidated inside osprofiler itself.
Let's use them.
Change-Id: Ib0266e0a6e9bfa99c4bacbdca623ab1211a822eb
Some config options in codes use double quotes
but most of the options use single quotes and
that need to be normalized
DocImpact
Closes-Bug: #1584558
Change-Id: I828326d9bb22b6cb98885255ebc0dfcc1efc0497
Return correct scheme in version URLs if service
behind an SSL termination proxy.
This is done by adding a new configuration option,
secure_proxy_ssl_header, which, when defined, makes
the wsgi application take the host_url scheme from
that header. By default, when this option is not
specified, there is no difference in behavior.
The intention is to configure any ssl-decrypting
proxy to set that header, so that glance-api knows
which protocol to use in the URLs in response.
This patch is largely based on the equivalent
nova patch: https://review.openstack.org/#/c/206479.
Partial-bug: 1558683
Change-Id: I9a9c0e42a6ad3c18d197f10095958b48d5cb879a
The default values needed for glance's implementation of cors
middleware have been moved from paste.ini into the configuration
hooks provided by oslo.config. Furthermore, these values have been
added to glance's default configuration parsing. This ensures
that if a value remains unset in glance-api.conf, it will be set to use
sane defaults, and that an operator modifying the configuration
file will be presented with a default set of necessary sane headers.
Change-Id: I3c9d267b6224d6c7e5cc2c41cb51fb7e363c4955
Closes-Bug: 1551836
Abhishek Kekane