The run_sql_cmd method in functional tests is not used. The method uses
Engine.execute() method which was removed in 2.0.
Change-Id: I6f9ea1dfd852a2d311e17964da0ea56c052b3fda
Made `centralized_db` cache driver as default driver so that we
can test it using tempest jobs in gate.
Implements blueprint centralized-cache-db
Depends-On: https://review.opendev.org/c/openstack/devstack/+/907110
Change-Id: Id94e93e3ba3fc207b39c7dbff92495805aa0f6f9
The following options set in functional tests do not exist.
[DEFAULT] send_identity_credentials
[DEFAULT] send_identity_headers
Change-Id: Ia739af13e0f88d0861e453a4d54ecc626e32b34b
Using the healthcheck middleware as a filter is deprecated and
the middleware should be used as an application[1].
[1] 6feaa13610c450c8486f969703768db5319b4846
This change updates definition and usage of the healthcheck middleware
accordingly to avoid the following deprecation warning.
DeprecationWarning: Using function/method 'Healthcheck.factory()' is
deprecated: The healthcheck middleware must now be configured as
an application, not as a filter.
Closes-Bug: #1937901
Change-Id: Id41e0313a481bea4e2bb14c69f2ad8a2070aa9be
PasteDeploy replaced pkg_resources with importlib_metadata[1] in 3.0.
This causes functional test failures because the line breaks in the
example configs from glance/tests/functional/__init__.py are read as a
"prefix" by PasteDeploy. For instance,
paste.filter_factory =
glance.api.middleware.version_negotiation:VersionNegotiationFilter.factory
is read as "\nglance.api.middleware.version_negotiation:VersionNegotiationFilter.factory".
This might be a bug in PasteDeploy or (quite unlikely) in
configparser.ConfigParser rather than in our code, so this issue should
be revisited at some point. For now, remove the cosmetic line breaks in
order to fix our gates.
[1] ef58c20961
Change-Id: I1e557e731cceb3cfbc4ebc3f0887adf372a91d1d
As per the revised SRBAC community goals, glance service is now
switching to new defaults by default hence removing the deprecated
``enforce_secure_rbac`` option which is no longer needed.
The ``enforce_secure_rbac`` option was introduced EXPERIMENTAL in
Wallaby release for operators to opt into enforcing authorization
based on common RBAC personas.
Related blueprint secure-rbac
Change-Id: I273527c85d30c1c09c086c73c892aaa6d127df6b
Till now glance was dependent on periodic job to get image
cached locally and also has dependency on ``cachemanage``
middleware for the same.
This patch removes the periodic job and dependency of
``cachemanage`` middleware and initiates immediate caching
of an image via ``PUT /cache/image_id`` API call.
Co-Author: Dan Smith <dms@danplanet.com>
Implements: blueprint instant-caching
Change-Id: I9ab3f1b7595e22dbb03af95168314352a44eb930
This is the same as the 'range' keyword in Python 3
Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Most of the functional tests contain a version of _url(). The
implementations don't really differ, outside of one that accepts a
protocol and another that is specific to IPV6 for a single test case.
This commit moves the common method to the functional base class and
removes the duplicated logic across the functional test implementations.
Change-Id: I0b161ad404bdb53e3c1cfe878f78a71c239cf40e
This reproduces the behavior described in bug 1932337, where a post-
upgrade GET by a non-owning user will return an error for a valid image
because the cinder store migration cannot complete.
This asserts the broken behavior to reproduce the problem, and the
rest of the test will be enabled when the bug is fixed.
Note that the test for this was duplicating code from the base class,
and since some changes were needed to those methods, this removes the
duplication and modifies only the base.
Change-Id: I741e6377b3aa40c14d637f0fbdf396dc468f6ebd
This patch enforces policy checks required for caching images
in API layer.
Partially-Implements: blueprint policy-refactor
Depends-On: https://review.opendev.org/c/openstack/nova/+/688802
Change-Id: Ie17b8f5bf308b8f07915ea18ace9b49955b8f0f0
This patch enforces policy checks required for importing/copying image
data to store in API layer.
Partially-Implements: blueprint policy-refactor
Change-Id: I18a5187d80bf76c0dc6f22dd8c96a8ffa0f46dc1
This change also includes change in index method for 'GET' all
namespaces to not include properties, objects, tags, associations
etc. in the response if policies of either one is disabled.
The MetadefAPIPolicy object will raise Forbidden for cases where an
operation is forbidden, but the namespace would be otherwise visible to
the user, and NotFound otherwise to obscure the existence of namespaces
that the user can not otherwise see.
Partially implements: blueprint policy-refactor
Change-Id: I59cd3bbd98065457bf0c9cfaf6888b7f56f7617c
Depends-On: https://review.opendev.org/c/openstack/grenade/+/803317
This makes it easier to run PATCH operations against the API from
synchronous functional tests. It also adds a status assertion to
one of the upload methods which makes it easier to determine when
an operation was denied.
Change-Id: Ie67cd0ada86c03d8eac06dfa3d0730d329591628
This adds a new tox target called functional-py38-rbac which enables
the new secure RBAC policy defaults for all functional tests. To do
this, the functional tests needed a little bit of extra work to
actually set those, and a new non-voting job is added to run these
in CI.
Related to blueprint policy-refactor
Change-Id: Id376193521671bdb0ebc08ea8e563578bbaa541f
Made changes to use default policies in our unit and functional tests
rather than referring it from glance/tests/etc/policy.yaml file. Existing
function 'set_policy_rules' can be used to test custom policy rules in
functional testing.
Improvements needed in followup patch:
1. Property protection related unit tests still reads the special policy
from the policy.yaml file, need to make provision to override it instead.
2. Need to remove 'set_policy' function which actually responsible for
above behavior.
Related to blueprint policy-refactor
Change-Id: I0de9b9f9a7de499574101e3366ced730b0cb5fd1
This simply adds some functionality and flexibility to this test base
so that subsequent patches can do some more things.
Related to blueprint glance-unified-quotas
Change-Id: Ic9359aac5dba2b4d7d0d2c7fa92a5b67440e22e0
This fixes the cache tests (and the 'caching' deployment flavor) to
send authorization headers so that they can be checked by later policy
patches.
Change-Id: I6099c7da24e06595e08fd292a5083327b9f0cc64
Currently it is not possible to configure the staging directory
URI of the functional workers. We need to be able to do that in order
to enable the stage cleaning behavior. Right now, they're all sharing
/tmp/staging, which will cause workers to conflict while running in
parallel. This causes them to use their private test directory,
which may also help some other spurious failures due to interaction.
Related-Bug: #1913625
Change-Id: Ic2ac2a528206c50c38e948a096daf9eb8e5eb715
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also convert the ./glance/tests/etc/policy.json to policy.yaml
file. Replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Depends-On: https://review.opendev.org/c/openstack/nova/+/773192
Change-Id: I17d0374dd4223688e5f95253802a4ae87377953a
This was a typo (by me) in the original implementation, but since
nothing was testing DELETE calls, it was never noticed.
Change-Id: I81f9d0a53493a1b9793bdaae082093f30df6e032
The ImageImportLocking test class has some useful helper functions
that are general enough to be used elsewhere. This moves those
to the SynchronousAPIBase test class.
Change-Id: Idbc6ab16ecac2eb9c83c58432f98414d68e16a52
This option was deprecated in the Ussuri release by change
I0f61f85a0aaa4f68e345fa08fbb6b039d3d32587 and it is now eligible
for removal following the standard OpenStack deprecation policy.
Change-Id: I78ec79f97bfdcc39772448296140f8d8f35adea1
In some cases, we spawn a process that we expect to run to completion
and exit. In those cases, disable the force-kill logic in wait_for_fork()
to avoid killing those processes too early.
Related-Bug: #1891190
Change-Id: Ia78d101ae3702d9f761ca8f04b4b10af4a6d84fb
This enhances our test for the import lock-busting case to include
freeing the stuck import task and letting the new and old ones proceed
to make sure that the end state looks like what we expect.
Note that this includes another task lock check after exiting the
ImportAction context before we call image save. While writing these
tests I determined that we can end up with the original task racing
to update the image locations. If set_data() took a very long time,
caused our lock to be stolen and another task is running, when our
set_data() finally finishes we may overwrite their newly-added
location when we go to save our (now stale) list. Thus, the extra
task check (imperfectly) tries to avoid us doing anything else after
our task lock is stolen.
Change-Id: I74baf53fac1c3e23f6dc743058165ecb39074626
The base class for the tests added in test_images_import_locking provides
a mechanism to make API cals directly against the WSGI stack, without
starting a separate server and using the local networking. This is useful
for cases where fault injection of global state needs to be altered, where
this is very difficult in the existing fork-and-exec functional test
model.
This moves that test base class out to the functional module, expands the
documentation a little, and also generalizes the request methods for
wider applicability.
Change-Id: I59e3b5d5d4b69f076092b9950c0d34467a6636ad
There is something about the process handling in the functional test base
that opens the scenario where we might signal our API master child,
waitpid() on it and never return (see bug 1891190). Specifically the
test_reload() case, where the API master is being sent SIGHUP various times,
which may cause it to temporarily mask out SIGTERM or something, which
means it ignores our request. Either way, a deadlock in the tests can
occur when the cleanup() routine goes to waitpid() on that master,
forever.
This change makes our wait_for_fork() helper call waitpid() with WNOHANG
and progressively ramp up the aggression on asking it to stop. First, it
waits for the original signal to be honored, then it starts sending SIGTERM,
followed by SIGKILLs until the overall timeout expires, after which it
fails the test. The latter should only ever happen if something gets
wedged in uninterruptible sleep.
Related-Bug: #1891190
Change-Id: Ie0e91e2ee86f33bcdb63eed74f2d9948108acfd4
These are super verbose and log the same thing many times, making the
test output very hard to read. They're not likely to be very useful in
debugging, so just quiet them down.
Change-Id: I7ce94addb5c9a28e9091fa47261638ee488d6f13
This teaches glance-api how to do async threading things when it is
running in pure-WSGI mode. In order to do that, a refactoring of things
that currently depend on eventlet is required.
It adds a [wsgi]/task_pool_threads configuration knob, which is used
in the case of pure-WSGI and native threads to constrain the number
of threads in that pool (and thus the task parallelism). This will
allow tuning by the operator, but also lets us default that to just
a single thread in the backport of these fixes so that we can avoid
introducing a new larger footprint in the backport unexpectedly.
Partial-Bug: #1888713
Depends-On: https://review.opendev.org/#/c/742047/
Change-Id: Ie15028b75fb8518ec2b0c0c0386d21782166f759
This patch removes majority of the registry and it's related
endpoints and config options that has been deprecated for
removal in various releases.
Change-Id: I75014bd50bf382efebe56bd89c20ffefbdde25f5
Added new import method 'copy-image' which will copy existing image into
specified list of stores. Introduced additional task which will serve
as internal plugin which will allow copying existing image into staging
area and then this data will be uploaded to specified stores via regula
import flow.
NOTE: This new import method 'copy-image' is only supported if multiple
stores are enabled in deployment.
APIImpact
Implements: blueprint copy-existing-image
Change-Id: I13eaab7ab013f44ce18465bdbdbe8052942570ff
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.
NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.
Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
As eventlet ssl termination is broken with python 3 and
we won't be supporting python 2.7 anymore we will just
remove ssl termination to glance-api and expect the
termination being handled by something else, like HAProxy.
This patch also removes the broken ssl test job as the
non-existing feature is not broken anymore.
Change-Id: Iaf16dfcfdb3a2c93312dcad1ea1229e6b3c8caaa
In Rocky multiple backend support is added as experimental feature. In
order to take advantage of this feature it is decided to deprecate
work_dir and node_staging_uri configuration options
and reserve two filesystem stores 'os_glance_tasks_store' and
'os_glance_staging_store', which can be used to get rid of initializing
store via internal functions.
These internal stores are considered "reserved stores" by Glance.
For the time being, these are hard-coded as filesystem stores. The
store prefix 'os_glance_' is reserved for internal Glance use and
the glance-api service will refuse to start if a store with this
prefix is included in the enabled_backends config option in
glance-api.conf.
NOTE: Because there are no sensible default values for the location
of the datadir for each of these stores, the operator must define
'os_glance_tasks_store' and 'os_glance_staging_store' in
glance-api.conf configuration file as shown below.
[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir/
[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging/
Each filesystem store must have a unique datadir.
Depends-On: https://review.openstack.org/#/c/639765/
Implements: blueprint rethinking-filesystem-access
Change-Id: I86ec513c5fc653dbb97b79d953d8430f014e684f
Added new periodic job which will run as per interval set using
'cache_prefetcher_interval' configuration option and fetch images
which are queued for caching in cache directory.
DocImpact
Change-Id: If2875f7a215aca10a6ed2febc89b02219f90a10d
In order to run the unit and functional Glance tests on Windows, we
have to:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
* we'll use subprocess.Popen when spinning up http servers.
* for the really simple ones defined in the test helpers, we'll just
use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
as well
* skip log rotation test, we can't move in-use files. Log rotation can
be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
markers rely on ordering, which can be flipped if the timestamps are
identical (can happen in case of resources created one after the
other, not sure yet if this happens really fast or the clock isn't
accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
in VMs).
blueprint windows-support
Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
In web-download import method functional tests we are trying to download
a file from 'https://www.openstack.org/assets/openstack-logo/2016R/OpenStack-Logo-Horizontal.eps.zip'.
Here we are assuming image is downloaded and will be active within 20
seconds else will be marked as failed. As of now these tests never
fails in local environment but, external networking will always be unreliable
from the CI environment which sometimes causes these tests to either
timeout or failure.
Instead of using external URL to download the image, deploying local http
server to dowload local image.
Closes-Bug: #1797571
Change-Id: I781666327f092ad439c23eca57128b8fd0334c89
Takashi Kajinami