Commit Graph

27 Commits

Author SHA1 Message Date
Pierre-Samuel Le Stang 480ea3825f Implement glance-download internal plugin
Add a new import method called glance-download
that implements a glance to glance download in
a multi-region cloud with a federated Keystone.

This method will copy the image data and
selected metadata to the target glance, checking
that the downloaded size match the "size" image
attribute in the source glance.

Implements: blueprint glance-download-import
Co-Authored-By: Victor Coutellier <victor.coutellier@gmail.com>
Change-Id: Ic51c5fd87caf04d38aeaf758ad2d0e2f28098e4d
2022-08-23 08:26:52 -07:00
Dan Smith ded8ecf382 Fix failing copy_image flow init
The new networkx requirement apparently causes us to fail the
test_init_copy_flow_as_non_owner test because a None value can not be
added to the flow graph. This just mocks out the glance_store call
that is failing to get the staging store to avoid us choking there
so we can proceed. We are just testing get_flow so we never use it
anyway.

Change-Id: I6fcb14ad240105ed0b1b9086c6c6c95034b4dd42
2021-08-04 08:18:55 -07:00
Dan Smith 76c3011a64 Enforce keystone limits for image upload
This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.

Note that this disables checking of the global size quota if keystone
quotas are enabled.

Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.

Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
2021-06-29 08:53:18 -07:00
Dan Smith 154ef3fe94 Update image.size after conversion
When we convert an image to a specified format during import, we
update the disk_format to match. At that point, we also know the (new)
image.size, so we should set it.

This is somewhat related to setting image size on stage, in that once
it is set we will validate that it does not change in later steps.
Since this one comes between stage and the actual store upload, this
patch makes conversion set it and confirms that the later steps are
happy with that. A later patch sets it during stage, confirming that
we can change it here during conversion when we are changing the
actual image file itself.

Related to blueprint glance-unified-quotas

Change-Id: I795c52f606f85955e39efc29b75f2941be1264b4
2021-06-21 07:19:01 -07:00
Dan Smith 4f20e5007f Fix missing context args to get_flow()
TaskExecutor._get_flow() passes a context argument to the task
initialization routine, but a couple of tests that short-circuit that
do not. This makes those tests mimic the real behavior separately
from the later patch that needs it to make it clear that it is not
actually requiring a new argument at runtime.

Related to blueprint glance-unified-quotas

Change-Id: Ifc2adcb7f8eaa7da2e7b063a8b79175025582500
2021-06-04 07:11:14 -07:00
Zuul d154cb1058 Merge "Make copy_image plugin use action wrapper" 2021-03-07 21:58:56 +00:00
Zuul fc0ee38b8b Merge "Allow plugins to mutate image extra_properties" 2021-03-07 09:48:29 +00:00
Zuul cf67d36efe Merge "Make action wrapper support arbitrary properties" 2021-03-06 17:34:06 +00:00
Dan Smith 41e1cecbe6 Distributed image import
This implements distributed image import support, which addresses
the problem when one API worker has staged the image and another
receives the import request.

The general approach is that when a worker stages the image, it
records its self-reference URL in the image's extra_properties.  When
the import request comes in, any other host will proxy that HTTP
request direct to the original host instead of trying to do the import
itself.

Implements: blueprint distributed-image-import

Change-Id: I12daccb43c535b579c22f9d0742039b2ab42e929
2021-03-02 11:52:12 -08:00
Dan Smith a3af071997 Make copy_image plugin use action wrapper
This makes the copy_image plugin use the action wrapper for its
manipulation of the image, in line with all the others. This adds
image_size and image_locations properties to the wrapper, which
copy-image needs.

Change-Id: Id5251b77fe1671594912e0dcfef7296007dab9b5
2021-02-10 07:36:24 -08:00
Dan Smith e54a1a52ec Allow plugins to mutate image extra_properties
This adds a set_image_extra_properties() method to the action wrapper.
It specifically disallows setting os_glance_* properties, for two
reasons. First, several (such as the task lock and the store lists)
need special handling to be atomic. Second, setting os_glance_*
properties from image metadata injection is almost definitely
destined for failure, now or later, as it would muck with our
internals. As discussed during the work to formally reserve that
namespace from the API, we also need to make sure operators do not
set these keys during injection.

This makes us drop any such keys, with appropriate logging. The
next patch will make the metadata injection task use this and is
the point at which we will actually change that behavior.

Change-Id: I0574ee3daf08d59b4547e353c921451e756e09f6
2021-02-10 07:36:21 -08:00
Dan Smith 5331f1e0ba Make action wrapper support arbitrary properties
Previously, the only attribute of the image that an import plugin
could change was the status. In order to convert the rest of the
plugins to use the wrapper, we need to define additional properties
that they can mutate. This removes set_image_status() and replaces
it with set_image_attribute() which allows a subset of those
properties to be set.

This also ends up passing the action wrapper to all plugin init
methods by way of just putting it straight into the kwargs that we
give to everything. That is added here as setup for the patches to
follow which depend on it.

Change-Id: Id613cfd61760c383c7c3cc6aea3f37eecb5ed4d9
2021-02-09 12:33:05 -08:00
Dan Smith a86062c492 Functional test enhancement for lock busting
This enhances our test for the import lock-busting case to include
freeing the stuck import task and letting the new and old ones proceed
to make sure that the end state looks like what we expect.

Note that this includes another task lock check after exiting the
ImportAction context before we call image save. While writing these
tests I determined that we can end up with the original task racing
to update the image locations. If set_data() took a very long time,
caused our lock to be stolen and another task is running, when our
set_data() finally finishes we may overwrite their newly-added
location when we go to save our (now stale) list. Thus, the extra
task check (imperfectly) tries to avoid us doing anything else after
our task lock is stolen.

Change-Id: I74baf53fac1c3e23f6dc743058165ecb39074626
2020-08-24 06:41:13 -07:00
Dan Smith 3f6e349d08 Implement time-limited import locking
This attempts to provide a time-based import lock that is dependent
on the task actually making progress. While the task is copying
data, the task message is updated, which in turn touches the task
updated_at time. The API will break any lock after 30 minutes of
no activity on a stalled or dead task. The import taskflow will
check to see if it has lost the lock at any point, and/or if its
task status has changed and abort if so.

The logic in more detail:

1. API locks the image by task-id before we start the task thread, but
   before we return
2. Import thread will check the task-id lock on the image every time it
   tries to modify the image, and if it has changed, will abort
3. The data pipeline will heartbeat the task every minute by updating
   the task.message (bonus: we get some status)
4. If the data pipeline heartbeat ever finds the task state to be changed
   from the expected 'processing' it will abort
5. On task revert or completion, we drop the task-id lock from the image
6. If something ever gets stuck or dies, the heartbeating will stop
7. If the API gets a request for an import where the lock is held, it
   will grab the task by id (in the lock) and check the state and age.
   If the age is sufficiently old (no heartbeating) and the state is
   either 'processing' or terminal, it will mark the task as failed,
   steal the lock, and proceed.

Lots of logging throughout any time we encounter unexpected situations.

Closes-Bug: #1884596
Change-Id: Icb3c1d27e9a514d96fca7c1d824fd2183f69d8b3
2020-08-24 06:41:13 -07:00
Dan Smith 737dfca83c Fix import failure status reporting when all_stores_must_succeed=True
As described in the referenced bug, if a store fails with
all_stores_must_succeed=True, we never add that store to the
failed list, nor remove it from the pending-import list, leaving a
polling client to wait forever. This fixes that by making the revert
handler of the import task do that if we are the one that failed.

Closes-Bug: #1891352
Change-Id: I3571960bbfb4f8f0a716937b541f5b1594cd0e16
2020-08-18 16:56:32 +00:00
Dan Smith 77d9cfa66e Update task message during import
This updates the task.message field with details about the copy,
which will also update the updated_at field on the task. This will
facilitate some rudimentary liveness checking from outside the task
thread. Note that it also checks the state of our task, and will
abort if it has been pushed out of 'processing' state externally.
This will be used in the following patch to faciliate import lock
busting behavior.

Change-Id: I8667c17813f6e701db98595b0b30df9e7b275294
2020-08-07 12:22:31 -07:00
Dan Smith e49f23c04d Heartbeat the actual work of the task
This introduces a CallbackIterator object that can sit in the data
pipeline when setting image data and provide periodic callbacks to
allow recording status, checking for cancellation, etc. If the
callback raises an exception, that will bubble up to the read
operation and cause an abort.

The heartbeat is timer-based, but cooperative, which means if the
data pipeline stalls, we will stop heartbeating. The goal is to
provide a callback about once per minute, although blocking on
the pipeline could make that less regular of course.

Change-Id: I136ef792d601326a67266e9ea9fcadd79ddba69e
2020-08-07 12:22:15 -07:00
Dan Smith 94a672c7cd Add tests for _ImportToStore.execute()
There are some existing tests for the body of this task, but none for the
meat of the task when things go right.

Change-Id: I6fd0505a265c8ffb03bf8c3c0f2d4a1176485b51
2020-08-07 12:13:22 -07:00
Dan Smith 2e11ecb15e Add testing for _CompleteTask in api_image_import
This critical task has no testing. Add some before we add more
functionality to it.

Change-Id: I4af4202d6650b60aebbed813b7640cf018466e12
2020-08-07 12:13:22 -07:00
Erno Kuvaja 747ba1c1ca Use correct import order in test_api_import_image
glance_store is not part of glance package.

Change-Id: I028076d8e1706794a698398f8b2a59130284f1bd
2020-08-06 14:11:23 +01:00
Erno Kuvaja b985e44f79 Fix active image without data
Fixes image import resulting active image without any data
when the image happens to be deleted from staging before
_DeleteFromFS task is ran which is supposed to cleanup.

Change-Id: I6256a25d1907ec15fbdde560fc8eae37f3067213
Closes-Bug: #1889640
2020-08-05 06:02:21 +00:00
Erno Kuvaja e8b13ce1b4 Fix active image when all uploads fail
This fix is for the case where all stores fails and
the end result is active image with no image data.

Change-Id: Ie7558eeae488406a1696f0ee1bf40b644ae6d8cc
Partial-Bug: #1889640
2020-08-05 06:00:59 +00:00
Dan Smith 2fd0c25733 Make import task capable of running as admin on behalf of user
This makes the api_image_import task capable of running as an admin on
behalf of a user if so authorized by the API. It includes a new object
called ImportActionWrapper which provides a bundle of utility methods
which can be run either against the user-authorized or admin-authorized
ImageRepo passed in from the API. It encapsulates all the actions
we are able and willing to run as an admin for the user.

This is currently not drivable by the API because the policy check is
still statically defined as "admin or owner" but this change is
offered without any needed modification to the functional tests to
prove that it does not regress existing functionality. The following
patch will introduce a more robust knob for allowing users to do this,
and it brings the functional test changes with it.

Change-Id: Iac75956e314ec6f41db18430486bd8be9754e780
2020-07-10 08:59:54 -07:00
Dan Smith f450b9cf03 Add a test to replicate the owner-required behavior of copy-image
Right now, image_import(method='copy-image') requires you to be the owner
of the image. This adds a test that validates that behavior so we can measure
the difference of the next set of changes to modify that.

Change-Id: I284271a458117b753a88b847598ca10ce4fb94fa
2020-07-01 08:41:26 -07:00
Sean McGinnis 94b0876429 Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-20 15:07:00 +00:00
Grégoire Unbekandt 92492cf504 Add ability to import image into multi-stores
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.

Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
2020-02-10 09:39:01 +01:00
Corey Bryant c58e5e02af Rename async package to async_
In Python 3.7, "async" is a keyword. To prevent it from
conflicting, rename the async package to async_.

Change-Id: I1eaf87eedb86679d9ca9323aac05f0770c33efea
Closes-Bug: #1781617
2018-08-07 14:42:14 -04:00