Add a new import method called glance-download
that implements a glance to glance download in
a multi-region cloud with a federated Keystone.
This method will copy the image data and
selected metadata to the target glance, checking
that the downloaded size match the "size" image
attribute in the source glance.
Implements: blueprint glance-download-import
Co-Authored-By: Victor Coutellier <victor.coutellier@gmail.com>
Change-Id: Ic51c5fd87caf04d38aeaf758ad2d0e2f28098e4d
This makes us enforce a quota on the total number of upload-related
image operations owned by a user.
Partially-implements: blueprint glance-unified-quotas
Change-Id: I2a28750aaf968e6a6324eb194d4280a640bfa5aa
This makes us enforce a quota on the amount of data a user has in
staging.
Partially-implements: blueprint glance-unified-quotas
Change-Id: I3cca4e589adc0aec138e5933c311aefd69ccee51
This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.
Note that this disables checking of the global size quota if keystone
quotas are enabled.
Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.
Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
When we convert an image to a specified format during import, we
update the disk_format to match. At that point, we also know the (new)
image.size, so we should set it.
This is somewhat related to setting image size on stage, in that once
it is set we will validate that it does not change in later steps.
Since this one comes between stage and the actual store upload, this
patch makes conversion set it and confirms that the later steps are
happy with that. A later patch sets it during stage, confirming that
we can change it here during conversion when we are changing the
actual image file itself.
Related to blueprint glance-unified-quotas
Change-Id: I795c52f606f85955e39efc29b75f2941be1264b4
This implements distributed image import support, which addresses
the problem when one API worker has staged the image and another
receives the import request.
The general approach is that when a worker stages the image, it
records its self-reference URL in the image's extra_properties. When
the import request comes in, any other host will proxy that HTTP
request direct to the original host instead of trying to do the import
itself.
Implements: blueprint distributed-image-import
Change-Id: I12daccb43c535b579c22f9d0742039b2ab42e929
This makes the copy_image plugin use the action wrapper for its
manipulation of the image, in line with all the others. This adds
image_size and image_locations properties to the wrapper, which
copy-image needs.
Change-Id: Id5251b77fe1671594912e0dcfef7296007dab9b5
This adds a set_image_extra_properties() method to the action wrapper.
It specifically disallows setting os_glance_* properties, for two
reasons. First, several (such as the task lock and the store lists)
need special handling to be atomic. Second, setting os_glance_*
properties from image metadata injection is almost definitely
destined for failure, now or later, as it would muck with our
internals. As discussed during the work to formally reserve that
namespace from the API, we also need to make sure operators do not
set these keys during injection.
This makes us drop any such keys, with appropriate logging. The
next patch will make the metadata injection task use this and is
the point at which we will actually change that behavior.
Change-Id: I0574ee3daf08d59b4547e353c921451e756e09f6
Previously, the only attribute of the image that an import plugin
could change was the status. In order to convert the rest of the
plugins to use the wrapper, we need to define additional properties
that they can mutate. This removes set_image_status() and replaces
it with set_image_attribute() which allows a subset of those
properties to be set.
This also ends up passing the action wrapper to all plugin init
methods by way of just putting it straight into the kwargs that we
give to everything. That is added here as setup for the patches to
follow which depend on it.
Change-Id: Id613cfd61760c383c7c3cc6aea3f37eecb5ed4d9
The _internal_plugins/* tasks are not getting passed the
ImportActionWrapper, and thus are doing things against the image
that do not respect the task lock, nor apply multiple updates
atomically.
This fixes that and brings them in line with the base tasks, using the
wrapper for things like getting the image_id. The copy_image task
looks at the image for a few other things that are not exposed out
of the wrapper, so I left the image_repo.get() in that task until I
can extend it. They are read-only though, so not a big deal.
The web-download test also was not passing the right options to
that task (confusing task_repo and image_repo), which this cleans
up as well. It was also missing validation of the code that reverts
the state to 'queued' when web-download fails, so that is added here
as well.
Change-Id: I6db86b3e17a6a2f78745b40381b9419fb4404a4e
Related-Bug: #1914826
This enhances our test for the import lock-busting case to include
freeing the stuck import task and letting the new and old ones proceed
to make sure that the end state looks like what we expect.
Note that this includes another task lock check after exiting the
ImportAction context before we call image save. While writing these
tests I determined that we can end up with the original task racing
to update the image locations. If set_data() took a very long time,
caused our lock to be stolen and another task is running, when our
set_data() finally finishes we may overwrite their newly-added
location when we go to save our (now stale) list. Thus, the extra
task check (imperfectly) tries to avoid us doing anything else after
our task lock is stolen.
Change-Id: I74baf53fac1c3e23f6dc743058165ecb39074626
This attempts to provide a time-based import lock that is dependent
on the task actually making progress. While the task is copying
data, the task message is updated, which in turn touches the task
updated_at time. The API will break any lock after 30 minutes of
no activity on a stalled or dead task. The import taskflow will
check to see if it has lost the lock at any point, and/or if its
task status has changed and abort if so.
The logic in more detail:
1. API locks the image by task-id before we start the task thread, but
before we return
2. Import thread will check the task-id lock on the image every time it
tries to modify the image, and if it has changed, will abort
3. The data pipeline will heartbeat the task every minute by updating
the task.message (bonus: we get some status)
4. If the data pipeline heartbeat ever finds the task state to be changed
from the expected 'processing' it will abort
5. On task revert or completion, we drop the task-id lock from the image
6. If something ever gets stuck or dies, the heartbeating will stop
7. If the API gets a request for an import where the lock is held, it
will grab the task by id (in the lock) and check the state and age.
If the age is sufficiently old (no heartbeating) and the state is
either 'processing' or terminal, it will mark the task as failed,
steal the lock, and proceed.
Lots of logging throughout any time we encounter unexpected situations.
Closes-Bug: #1884596
Change-Id: Icb3c1d27e9a514d96fca7c1d824fd2183f69d8b3
As described in the referenced bug, if a store fails with
all_stores_must_succeed=True, we never add that store to the
failed list, nor remove it from the pending-import list, leaving a
polling client to wait forever. This fixes that by making the revert
handler of the import task do that if we are the one that failed.
Closes-Bug: #1891352
Change-Id: I3571960bbfb4f8f0a716937b541f5b1594cd0e16
This updates the task.message field with details about the copy,
which will also update the updated_at field on the task. This will
facilitate some rudimentary liveness checking from outside the task
thread. Note that it also checks the state of our task, and will
abort if it has been pushed out of 'processing' state externally.
This will be used in the following patch to faciliate import lock
busting behavior.
Change-Id: I8667c17813f6e701db98595b0b30df9e7b275294
This introduces a CallbackIterator object that can sit in the data
pipeline when setting image data and provide periodic callbacks to
allow recording status, checking for cancellation, etc. If the
callback raises an exception, that will bubble up to the read
operation and cause an abort.
The heartbeat is timer-based, but cooperative, which means if the
data pipeline stalls, we will stop heartbeating. The goal is to
provide a callback about once per minute, although blocking on
the pipeline could make that less regular of course.
Change-Id: I136ef792d601326a67266e9ea9fcadd79ddba69e
Fixes image import resulting active image without any data
when the image happens to be deleted from staging before
_DeleteFromFS task is ran which is supposed to cleanup.
Change-Id: I6256a25d1907ec15fbdde560fc8eae37f3067213
Closes-Bug: #1889640
This fix is for the case where all stores fails and
the end result is active image with no image data.
Change-Id: Ie7558eeae488406a1696f0ee1bf40b644ae6d8cc
Partial-Bug: #1889640
This makes the api_image_import task capable of running as an admin on
behalf of a user if so authorized by the API. It includes a new object
called ImportActionWrapper which provides a bundle of utility methods
which can be run either against the user-authorized or admin-authorized
ImageRepo passed in from the API. It encapsulates all the actions
we are able and willing to run as an admin for the user.
This is currently not drivable by the API because the policy check is
still statically defined as "admin or owner" but this change is
offered without any needed modification to the functional tests to
prove that it does not regress existing functionality. The following
patch will introduce a more robust knob for allowing users to do this,
and it brings the functional test changes with it.
Change-Id: Iac75956e314ec6f41db18430486bd8be9754e780
The import plugins were designed to interact with the image data
or metadata before the image becomes active. It does not make sense
to try to run them on 'copy-image' taskflow which is just copying
the data between stores on active image.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Closes-Bug: #1885725
Change-Id: I2df30aa26f96a0f210989758868b42bcb9d2d72f
Added new import method 'copy-image' which will copy existing image into
specified list of stores. Introduced additional task which will serve
as internal plugin which will allow copying existing image into staging
area and then this data will be uploaded to specified stores via regula
import flow.
NOTE: This new import method 'copy-image' is only supported if multiple
stores are enabled in deployment.
APIImpact
Implements: blueprint copy-existing-image
Change-Id: I13eaab7ab013f44ce18465bdbdbe8052942570ff
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.
Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
If multiple stores configured in glance and Image is deleted while import
operation is in progress then image data stays in staging area
(filesystem backend) and there is no other way than clearing it
manually.
Modified delete method to delete the data from staging area if image is
deleted while import operation is in progress.
Change-Id: Ib58accd6514e589dccde57fe063815b1ab1ce496
Closes-Bug: #1855417
In Rocky multiple backend support is added as experimental feature. In
order to take advantage of this feature it is decided to deprecate
work_dir and node_staging_uri configuration options
and reserve two filesystem stores 'os_glance_tasks_store' and
'os_glance_staging_store', which can be used to get rid of initializing
store via internal functions.
These internal stores are considered "reserved stores" by Glance.
For the time being, these are hard-coded as filesystem stores. The
store prefix 'os_glance_' is reserved for internal Glance use and
the glance-api service will refuse to start if a store with this
prefix is included in the enabled_backends config option in
glance-api.conf.
NOTE: Because there are no sensible default values for the location
of the datadir for each of these stores, the operator must define
'os_glance_tasks_store' and 'os_glance_staging_store' in
glance-api.conf configuration file as shown below.
[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir/
[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging/
Each filesystem store must have a unique datadir.
Depends-On: https://review.openstack.org/#/c/639765/
Implements: blueprint rethinking-filesystem-access
Change-Id: I86ec513c5fc653dbb97b79d953d8430f014e684f
When operator has not enabled 'file' store and using other stores like ceph,
swift etc. the uploading to staging area works as we explicitly
build 'file' store during this operation, while cleaning up we directly
use 'glance_store.delete_from_backend' which only works if 'file'
store is enabled.
Modified '_DeleteFromFS' task and _unstage call which will use os
module to unlink the file present in staging area explicitly to
delete the data from staging area.
Closes-Bug: #1803498
Change-Id: If0b3b0af9300301291758c67267890e0959ebb3c
In Python 3.7, "async" is a keyword. To prevent it from
conflicting, rename the async package to async_.
Change-Id: I1eaf87eedb86679d9ca9323aac05f0770c33efea
Closes-Bug: #1781617
Pierre-Samuel Le Stang