Commit Graph

246 Commits

Author SHA1 Message Date
Abhishek Kekane 9b9111f819 Make `centralized_db` cache driver default
Made `centralized_db` cache driver as default driver so that we
can test it using tempest jobs in gate.

Implements blueprint centralized-cache-db
Depends-On: https://review.opendev.org/c/openstack/devstack/+/907110

Change-Id: Id94e93e3ba3fc207b39c7dbff92495805aa0f6f9
2024-03-05 19:35:20 +00:00
Takashi Kajinami 1e99767d0e Drop ineffective options from config file
The following options set in functional tests do not exist.
 [DEFAULT] send_identity_credentials
 [DEFAULT] send_identity_headers

Change-Id: Ia739af13e0f88d0861e453a4d54ecc626e32b34b
2024-01-31 01:16:12 +00:00
Takashi Kajinami 5378396b84 Fix test failures with oslo.limit 2.3.0
The oslo.limit 2.3.0 release contains the validation to ensure
the [oslo_limit] endpoint_id option is set[1]. This fixes the test
failures caused by that validation by overriding the option in all
test cases where unified quota implementation is used.

[1] https://review.opendev.org/c/openstack/oslo.limit/+/897509

Closes-Bug: #2049064
Change-Id: I704393892799ee0d5edf45fabe07afe01473e38d
2024-01-12 00:20:42 +09:00
Abhishek Kekane fd222f3128 Sort locations based on store weight
Related to blueprint store-weight

Change-Id: I2383a476cb7e79c7efecdf33203cff0b50ef3bbb
2023-08-01 14:16:08 +00:00
xuanyandong be997b53ab Remove unicode literal strings
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: Id9e1a5fb9c732c207ee08f0dbf387436a1783174
2022-06-30 19:37:03 +02:00
Durga Malleswari Varanasi cb65cc8e44 Internal server error if shared member tries to stage data
to image

Glance API throws an Internal Server Exception when a non
active image is shared with another project and any member
from that project tries to stage data for the same.

This patch will cover the Functional Test Case for the same

Closes-Bug: #1939922
Change-Id: I065036cff6e107b3438007db40981459400ccb6b
2022-04-28 16:26:12 +00:00
Zuul a42fda92dc Merge "[APIImpact] Quota usage API" 2022-02-08 03:33:58 +00:00
Dan Smith f865b8cac7 [APIImpact] Quota usage API
This adds a /v2/info/usage API endpoint which exposes to the user
their current limits and usage.

The discovery API does not (appear to) have existing tests, so this
adds a module for that, although only usage tests are added currently.

Implements: blueprint quota-api
Change-Id: I50c98bac50f815bdb9baae024e77afd388f74554
2022-02-03 09:55:50 -08:00
Stephen Finucane 87ba56161b Remove six.moves.urllib usage
Change-Id: I71be65ef7b8f710e9317419d7b38559b39f461b0
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-01-27 14:54:05 +00:00
Stephen Finucane 9679ffc463 Remove six.moves.http_client usage
This is a rather beefy change due to the number of usages of this
import. The changes are trivial though.

Change-Id: I7badeeaca438b0291f4ed86670e7f217e6372c61
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-01-27 14:54:05 +00:00
Stephen Finucane 39e667a145 Remove six.moves.range usage
This is the same as the 'range' keyword in Python 3

Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-01-27 14:54:05 +00:00
Stephen Finucane 6bd7c188ee Remove six.text_type, six.binary_type usage
Change-Id: I2ed464202f8b645aed11490e111c61d3c7423c11
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-01-27 14:54:00 +00:00
Stephen Finucane 502fa0ffc8 Remove six.iteritems usage
We also remove hacking tests for this, along with those for iterkeys and
itervalues (no usage of these).

Change-Id: If5b46580078eb756651ac6118f502eccdc693646
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2021-12-21 13:01:56 +00:00
Zuul a693e066b3 Merge "Fix typos" 2021-10-20 00:01:42 +00:00
Andre Aranha 947e541fc0 Update qcow test to use qcow2 instead
qemu-img package for Centos 8 doesn't support qcow disk format anymore,
due to this, tests using qcow are failing with
`Unknown file format 'qcow'`.
This patch updates the qcow tests for qcow2 instead.

Closes-Bug: #1946457

Change-Id: If488ae4299a6cab521eec9c5e9ce4f29f94057e1
2021-10-13 10:31:27 +00:00
Cyril Roelandt 0ca2f92161 Fix typos
Change-Id: I5e7776324c01e467799b1296c35b84dc0c10cce2
2021-10-13 03:02:52 +02:00
Lance Bragstad 4e68e5da6f Remove duplicate _url() methods from functional test classes
Most of the functional tests contain a version of _url(). The
implementations don't really differ, outside of one that accepts a
protocol and another that is specific to IPV6 for a single test case.

This commit moves the common method to the functional base class and
removes the duplicated logic across the functional test implementations.

Change-Id: I0b161ad404bdb53e3c1cfe878f78a71c239cf40e
2021-09-10 08:14:52 -07:00
Dan Smith ba37ea3227 Check get_image(s) in the API
This includes a change to catch Forbidden and convert to NotFound.
The previous Forbidden handler was not only correct (it shoud hide
the permissions error with "not found") but it was actually dead code,
since the DB was performing its own checks and would never raise
Forbidden.

This also includes a change of the default policy for get_images
to include the other states, like get_image does. I think this was
just an oversight in the original RBAC patches, which didn't matter
because they weren't really being honored strictly.

Partially implements: blueprint policy-refactor

Change-Id: I70100cd7f01da803e9740cea1f7ce7ae18ad6919
2021-08-04 07:56:05 -07:00
Abhishek Kekane 525b6addd0 Make our functional tests compatible with RBAC
Recently we have added new non voting job 'functional-py38-rbac' which
will run our functional tests by enabling Secure RBAC. To run functional
tests with Secure RBAC enabled we need to pass 'reader' role to the
requests as well.

Change-Id: I70e085bd6c07548a9d4802fee651e589276a00e5
2021-07-16 07:58:46 -07:00
Dan Smith 8d6ee6f822 Add image_count_uploading quota enforcement
This makes us enforce a quota on the total number of upload-related
image operations owned by a user.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I2a28750aaf968e6a6324eb194d4280a640bfa5aa
2021-07-02 08:29:07 -07:00
Zuul e788d68ef4 Merge "Use default policies in our tests" 2021-07-01 14:35:50 +00:00
Abhishek Kekane 21257615e7 Use default policies in our tests
Made changes to use default policies in our unit and functional tests
rather than referring it from glance/tests/etc/policy.yaml file. Existing
function 'set_policy_rules' can be used to test custom policy rules in
functional testing.

Improvements needed in followup patch:
1. Property protection related unit tests still reads the special policy
from the policy.yaml file, need to make provision to override it instead.

2. Need to remove 'set_policy' function which actually responsible for
above behavior.

Related to blueprint policy-refactor
Change-Id: I0de9b9f9a7de499574101e3366ced730b0cb5fd1
2021-06-30 16:12:25 +00:00
Dan Smith a36666e2fe Add image_count_total quota enforcement
This makes us enforce a quota on the total number of (non-deleted)
images owned by a user.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I8af124d9307263cd8289d0701fb9a745d13b1d56
2021-06-29 08:53:18 -07:00
Dan Smith 59990d513a Add image_stage_total quota enforcement
This makes us enforce a quota on the amount of data a user has in
staging.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I3cca4e589adc0aec138e5933c311aefd69ccee51
2021-06-29 08:53:18 -07:00
Dan Smith 76c3011a64 Enforce keystone limits for image upload
This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.

Note that this disables checking of the global size quota if keystone
quotas are enabled.

Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.

Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
2021-06-29 08:53:18 -07:00
Dan Smith c290c4af5a Make image stage set image.size
When we stage an image, we know the image size, so we should set it.
Since the user may be streaming an image and did not declare the
expected size, this lets them confirm before they import. It also
provides us a value to count for the staging quota coming in a later
patch.

The bulk of this change is adjusting existing tests to validate the
size for all of our stage/import tests.

This follows the change to set the image size during conversion,
ensuring that we can set it during stage for non-conversion jobs,
and that the conversion code that re-sets it after changing the
image data and size continues to work.

Related to blueprint glance-unified-quotas

Change-Id: I93a9145df27594a0cc59828619a7d0573e58d4fc
2021-06-21 07:19:01 -07:00
Zuul 878d7f49d8 Merge "Distributed image import" 2021-03-03 23:58:57 +00:00
Dan Smith 41e1cecbe6 Distributed image import
This implements distributed image import support, which addresses
the problem when one API worker has staged the image and another
receives the import request.

The general approach is that when a worker stages the image, it
records its self-reference URL in the image's extra_properties.  When
the import request comes in, any other host will proxy that HTTP
request direct to the original host instead of trying to do the import
itself.

Implements: blueprint distributed-image-import

Change-Id: I12daccb43c535b579c22f9d0742039b2ab42e929
2021-03-02 11:52:12 -08:00
Dan Smith e9852fb625 Make functional tests set node_staging_uri
Currently it is not possible to configure the staging directory
URI of the functional workers. We need to be able to do that in order
to enable the stage cleaning behavior. Right now, they're all sharing
/tmp/staging, which will cause workers to conflict while running in
parallel. This causes them to use their private test directory,
which may also help some other spurious failures due to interaction.

Related-Bug: #1913625
Change-Id: Ic2ac2a528206c50c38e948a096daf9eb8e5eb715
2021-03-02 11:49:47 -08:00
Abhishek Kekane 281fadc15c New API /v2/images/{id}/tasks
Added new API /v2/images/{id}/tasks to show tasks associated with
image. This API will return list of tasks associated for valid image
else returns 404 not found if image is not present. This API also
initiates task scrubbing before returning tasks to user.

Implements: blueprint messages-api
Change-Id: Ib3cacb4dd4d75de32e539f8a3b48bdaa762e6d8e
2021-02-24 05:19:43 +00:00
Dan Smith 2bfdc87a8c Stop raising 403 when image is not found
Glance has a vestigial policy override knob that allows an operator
to force a 403 response when attempting to get an image from the DB
that is not found. This runs contrary to the API documentation, and
creates a (potential) interoperability concern across clouds with
differing policies on this topic.

This removes that override and changes the test from validating this
override to validating that it can no longer happen.

APIImpact
Change-Id: Ie24e3eb2f31d10d2ab9af62a0b645e8bdd0c2ff2
Closes-Bug: #1915543
2021-02-22 17:35:50 +00:00
Ade Lee 8027d90710 Replace md5 with oslo version
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.

md5 is allowed when in a non-security context.  There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.

In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.

Some downstream python versions already support this parameter.  To
support these versions, a new encapsulation of md5() has been added to
oslo_utils.  See https://review.opendev.org/#/c/750031/

This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.

Reviewers need to pay particular attention as to whether the keyword
parameter (usedforsecurity) is set correctly.

It looks like the usage of md5() here is solely to determine a checksum
of an image.

With this patch and the dependent patch for glance_store, all the
unit and functional tests pass on a FIPS enabled system.

Depends-On: https://review.opendev.org/#/c/756157
Depends-On: https://review.opendev.org/#/c/760160
Change-Id: I3b6d78d9792d4655bf0f4989cf82aced3f27491b
2020-12-15 10:43:19 -05:00
Dan Smith 8c7342cbc1 Make test_copy_image_revert_lifecycle handle 409 on import retry
This test is very racy in general, and specifically on the post-revert
retry to do the import, it may start the next import before the previous
one has finished reverting. This patch makes it retry a few times with
delay if an HTTP 409 is received so it is tolerant of that situation once
it becomes possible in the subsequent patch to add import locking.

Change-Id: Ic933f170d43b290fd819e8527ecb60be7f7f3f89
2020-08-24 06:41:13 -07:00
Dan Smith 6c96319eeb Poll for final state on test_copy_image_revert_lifecycle()
This test currently simulates failure by pre-deleting the store
directory for 'file3' which is the second of a two-store import
operation. The goal is to assert that the later failure reverts
the import of the earlier 'file2' store. However, the way the
polling loop works is that we break out once 'file2' has completed,
and then assume that 'file3' has already failed and reverted.
This is a race, and one we're losing consistently in CI.

This patch waits for the failure of 'file3' to be reported, as
well as the revert of 'file2' to occur before exiting the polling
loop and checking the final state of things to ensure that the
revert has actually happened. This addresses the non-determinism
inherent in the original test.

Change-Id: I11c7edaefc96236d2757acfb70d9c338c0f51348
2020-08-18 16:57:25 +00:00
Dan Smith 737dfca83c Fix import failure status reporting when all_stores_must_succeed=True
As described in the referenced bug, if a store fails with
all_stores_must_succeed=True, we never add that store to the
failed list, nor remove it from the pending-import list, leaving a
polling client to wait forever. This fixes that by making the revert
handler of the import task do that if we are the one that failed.

Closes-Bug: #1891352
Change-Id: I3571960bbfb4f8f0a716937b541f5b1594cd0e16
2020-08-18 16:56:32 +00:00
Dan Smith ec372377b1 Functional reproducer for bug 1891352
This reproduces the behavior described in the referenced bug, showing
the behavior before the fix. This asserts the *broken* behavior and leaves
examples of what the asserts should be for the fix patch.

Partial-Bug: #1891352

Change-Id: I992a14abb7cb41e2472741bf08b93976d4320ace
2020-08-12 09:57:20 -07:00
Zuul ed3296b23a Merge "Inspect upload/import stream and set virtual_size" 2020-08-08 05:56:02 +00:00
Dan Smith 5a7fb9db6e Fix non-deterministic copy_image_revert_lifecycle test
This test is failing fairly often in zuul. The test attempts to delete
one of the stores while the import is happening to simulate a failure
and ensure that the first store is not reverted. This can race and fail
to delete before the task completes. Instead, if we just delete the
store directory before the import, we know that when the flow gets to
the second store's step, it will fail to write the image, giving the same
result, reliably.

Change-Id: Ibeea8a93f140f35daa47024d07ad806162ff690d
2020-08-07 10:07:35 -07:00
Dan Smith b019765668 Inspect upload/import stream and set virtual_size
If we are processing a disk_format that we know about, wrap the
data pipeline with the format inspector and set virtual_size
after upload is complete.

Related to blueprint calculate-virtual-size

Change-Id: I25cd3cde94fefaa5d8ac72f10a075fe34a5df7bf
2020-08-05 08:42:04 -07:00
Dan Smith ee8a69d506 Add a policy knob for allowing non-owned image copying
This adds a copy_image policy knob which can be used to grant users
the ability to copy images other than their own using the new
functionality just added to the api_image_import task. By default,
only admins are allowed to do this.

A functional test modification is included to show that users can
be granted permission to do this based on something like the
"public visibility" attribute of an image.

Change-Id: Idebf66e2944bcddb7a5c76b81e47c654b401c2a8
2020-07-15 12:59:13 -07:00
Zuul 1c6c1fa549 Merge "Removal of 'enable_v2_api'" 2020-07-13 20:56:10 +00:00
Dan Smith 019afb7894 Add a functional test for non-owned image copying
This adds a functional test that confirms the current expected
behavior of what happens when a user attempts to copy an image
that they don't own (i.e. they get a 401).

Change-Id: Id49f83a63e59c262ea20062c6f3aed50c56adb74
2020-07-10 08:59:52 -07:00
Dan Smith 1b172798cd Refactor common auth token code in images test
This pulls out the authentication token manipulation used by some of the
images functional tests, which is replicated in several places into a
single helper. This will also be used by a subsequent functional test for
cross-owner copy-image support.

Change-Id: I80cf190f122ce61a47340181e1ad02ee24b5cb2d
2020-07-10 08:58:08 -07:00
Erno Kuvaja bbb3ede895 Removal of 'enable_v2_api'
Removal of config option 'enable_v2_api' and its related
operations and tests.

Change-Id: Ic83e7e8077b3fc939990c1f8e9c99b13b7fcd7ad
2020-07-03 19:01:39 +00:00
Erno Kuvaja 673666cbf4 Deprecation cleanout Registry and related
This patch removes majority of the registry and it's related
endpoints and config options that has been deprecated for
removal in various releases.

Change-Id: I75014bd50bf382efebe56bd89c20ffefbdde25f5
2020-06-30 20:41:30 +01:00
Abhishek Kekane 7af0b68b79 Skip 'test_image_member_lifecycle_for_multiple_stores' on failure
While running above test using python 3.7 environment, sometimes it closes
the client connection before getting the response causing 'ConnectionError'
and results in failure of the test. The reason for abrupt connection close
is 'test_image_member_lifecycle_for_multiple_stores' test makes lots of
connection requests which might causes race condition or dropping of
connection.

As a temporary solution caught the 'requests.exceptions.ConnectionError' and
skipped the test to avoid failure.

Change-Id: Ica04d8878eeb748cc4bbfedae895a54896ce16b3
Related-Bug: #1873735
2020-04-20 09:47:00 +00:00
Abhishek Kekane 1754c9e2b0 Copy existing image in multiple stores
Added new import method 'copy-image' which will copy existing image into
specified list of stores. Introduced additional task which will serve
as internal plugin which will allow copying existing image into staging
area and then this data will be uploaded to specified stores via regula
import flow.

NOTE: This new import method 'copy-image' is only supported if multiple
stores are enabled in deployment.

APIImpact
Implements: blueprint copy-existing-image
Change-Id: I13eaab7ab013f44ce18465bdbdbe8052942570ff
2020-02-12 05:32:46 +00:00
Zuul 40ee49b217 Merge "Remove registry related functional and unit tests" 2020-02-11 10:27:38 +00:00
Grégoire Unbekandt 92492cf504 Add ability to import image into multi-stores
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.

Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
2020-02-10 09:39:01 +01:00
Abhishek Kekane b55c0a31c0 Remove registry related functional and unit tests
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.

NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.

Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
2019-12-18 09:23:37 +00:00