Glance was not supporting uwsgi deployment when we added periodic job
to prefetch images into cache. Later in ussuri uwsgi support was added
but we missed to implement periodic job to pre-cache the image.
This patch add this support for glance + uwsgi. For WSGI, we run the
prefetcher with an external lock, which makes sure that multiple API
workers will not attempt to cache an image at the same time. In this
case, if multiple workers attempt to run at the same time, only one
will grab the lock and do the work. When completed, the other worker
will grab the lock and either find all the work completed, or complete
new work that has been queued since the first one started.
Closes-Bug: #1939307
Co-Authored-By: Dan Smith <dms@danplanet.com>
Change-Id: I2abd1e60f414fbd68ce84e0b280f8b3e4e791a82
TaskExecutor._get_flow() passes a context argument to the task
initialization routine, but a couple of tests that short-circuit that
do not. This makes those tests mimic the real behavior separately
from the later patch that needs it to make it clear that it is not
actually requiring a new argument at runtime.
Related to blueprint glance-unified-quotas
Change-Id: Ifc2adcb7f8eaa7da2e7b063a8b79175025582500
This adds the ImageLock flow atom to the test_async flow assertions to
make sure it gets included. Also changed are the magic numbers for
expected flow lengths, per feedback on the
Icb3c1d27e9a514d96fca7c1d824fd2183f69d8b3 review.
Change-Id: I0795e3835a500f20102e0c393a9a15f00672c87a
This attempts to provide a time-based import lock that is dependent
on the task actually making progress. While the task is copying
data, the task message is updated, which in turn touches the task
updated_at time. The API will break any lock after 30 minutes of
no activity on a stalled or dead task. The import taskflow will
check to see if it has lost the lock at any point, and/or if its
task status has changed and abort if so.
The logic in more detail:
1. API locks the image by task-id before we start the task thread, but
before we return
2. Import thread will check the task-id lock on the image every time it
tries to modify the image, and if it has changed, will abort
3. The data pipeline will heartbeat the task every minute by updating
the task.message (bonus: we get some status)
4. If the data pipeline heartbeat ever finds the task state to be changed
from the expected 'processing' it will abort
5. On task revert or completion, we drop the task-id lock from the image
6. If something ever gets stuck or dies, the heartbeating will stop
7. If the API gets a request for an import where the lock is held, it
will grab the task by id (in the lock) and check the state and age.
If the age is sufficiently old (no heartbeating) and the state is
either 'processing' or terminal, it will mark the task as failed,
steal the lock, and proceed.
Lots of logging throughout any time we encounter unexpected situations.
Closes-Bug: #1884596
Change-Id: Icb3c1d27e9a514d96fca7c1d824fd2183f69d8b3
This fix is for the case where all stores fails and
the end result is active image with no image data.
Change-Id: Ie7558eeae488406a1696f0ee1bf40b644ae6d8cc
Partial-Bug: #1889640
This teaches glance-api how to do async threading things when it is
running in pure-WSGI mode. In order to do that, a refactoring of things
that currently depend on eventlet is required.
It adds a [wsgi]/task_pool_threads configuration knob, which is used
in the case of pure-WSGI and native threads to constrain the number
of threads in that pool (and thus the task parallelism). This will
allow tuning by the operator, but also lets us default that to just
a single thread in the backport of these fixes so that we can avoid
introducing a new larger footprint in the backport unexpectedly.
Partial-Bug: #1888713
Depends-On: https://review.opendev.org/#/c/742047/
Change-Id: Ie15028b75fb8518ec2b0c0c0386d21782166f759
This refactors the task creation path to allow optional passing
of an admin-enabled ImageRepo to tasks. This will be used for
allowing import_image(method='copy-image') to update non-owned
image metadata during a copy.
Change-Id: Ief3440759fda4bd90979caa79641770cb022b7da
The import plugins were designed to interact with the image data
or metadata before the image becomes active. It does not make sense
to try to run them on 'copy-image' taskflow which is just copying
the data between stores on active image.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Closes-Bug: #1885725
Change-Id: I2df30aa26f96a0f210989758868b42bcb9d2d72f
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
In Python 3.7, "async" is a keyword. To prevent it from
conflicting, rename the async package to async_.
Change-Id: I1eaf87eedb86679d9ca9323aac05f0770c33efea
Closes-Bug: #1781617
Abhishek Kekane