7.0 KiB
Allow abort live migrations in queued status
https://blueprints.launchpad.net/nova/+spec/abort-live-migration-in-queued-status
This blueprint adds support to allow abort live migrations in
queued
status.
Problem description
The functionality of abort live migration was added in microversion
2.241, and currently only migrations in
running
status are allowed to be aborted.
There is a config option max_concurrent_live_migrations
that can be used to control the max number of concurrent live
migrations, the default value is 1. When the number of live migration
requests could be greater than the max concurrent live migration
configuration, there will be migrations wait in queue. The migrations
could remain in queued
status for a very long time depend
on the queue length and the processing speed.
Admins may want to abort migrations in queue due to time consumption
considerations etc. It will be unreasonable to make admins wait until
the status turn to running
before they can be aborted.
Use Cases
Migrations could be stuck in queued
status for a very
long time because of the migration queue length and processing speed.
Admins may want to abort migrations in queue due to time consumption
considerations etc.
Proposed change
The whole change will be divided into two steps:
Step1 - Fix the problem of lack of queue
In the current implementation, the code that serializes the live
migrations on compute node uses a python semaphore, the value of the
semaphore is set to be CONF.max_concurrent_live_migrations
,
each incoming migration will try to acquire this semaphore, if the
acquire succeed, the value of the semaphore will decrease by one, and
the status of the migration will turn to status other than
queued
. When the value decreased to 0, new incoming
migrations will be blocked(migration status will be queued
)
until some of the previous migration was finished(succeed, failed or
aborted) and releases the semaphore.
According to the above mentioned implementation, it is unable to
abort a migration in queued
status as there is actually no
QUEUE, so we are not able to control the migrations blocked by the
semaphore.
This spec will propose a design that can achieve the above mentioned goal:
- Using
ThreadPoolExecutor
fromconcurrent.futures
lib instead of the currenteventlet.spawn_n()
+ pythonSemaphore
implementation. The size of the Thread Pool will be limited byCONF.max_concurrent_live_migrations
. When a live migration request came in, we submit the_do_live_migration
calls to the pool, and it will return aFuture
object, we will use that later. If the pool is full, the new comming request will be blocked and kept inqueued
status. - Add a new
_waiting_live_migration
variable to theComputeManager
class of the compute node, this will be a dict, and will be initialized as an empty dict. We will:- Record the connection between
migration_uuid
and theFuture
object when the thread is created in previous step, we will usemigration_uuid
as key andFuture
object as value in our dict. - Remove the corresponding key/value the first thing if the thread
successfully acquired the executor and enter
_do_live_migration()
2. In this way, we will have a queue-like thing to store Futures and make it possible to get them bymigration_uuid
.
- Record the connection between
Step2 - Allow abort live migrations in queued status
After the modification proposed in step 1, we will be able to get
threads blocked by migration_uuid
and then we can abort
them:
- First check whether the provided
migration_uuid
is in the_waiting_live_migration
dict or not, if it is not in, then it will be in status other thanqueued
, we can switch to the workflow as is today. - If the provided
migration_uuid
is in_waiting_live_migration
dict then get the correspondingFuture
object and callcancel()
method of theThreadPoolExecutor
. - If the cancel call succeed, we perform roll back and clean ups for
the migration in
queued
status. The cancel call will returnFalse
if the providedFuture
object is currently executing, which means the provided thread is no longer blocked, so we can switch to the workflow of abort migration inrunning
status as is today. - Add an API microversion to
DELETE /servers/{id}/migrations/{migration_id}
API to allow abort live migration inqueued
status. If the microversion of the request is equal or beyond the newly added microversion, API will check theinstance.host's
nova-compute service version and make sure it is new enough for the new support, if not, API will still return 400 as today. - We will also add a cleanup to the pool when the compute manager is
shutting down. This can simply be done by calling
ThreadPoolExecutor.shutdown(wait=False)
.
Alternatives
None
Data model impact
None
REST API impact
The proposal would add API microversion to
DELETE /servers/{id}/migrations/{migration_id}
API to allow
abort live migration in queued
status. When request with
API microversion larger than the newly added microversion, the response
will change from HTTP 400 BadRequest
to
HTTP 202 Accepted
if the status of requested live migration
is in queued
status.
Security impact
None
Notifications impact
None
Other end user impact
Python-novaclient will be modified to handle the new microversion to
allow abort live migrations in queued
status.
Performance Impact
None
Other deployer impact
None
Developer impact
None
Upgrade impact
Compute API will still return 400 for trying abort a migration in queued state if the compute service that the instance is running on is too old.
Implementation
Assignee(s)
- Primary assignee:
-
Zhenyu Zheng
Work Items
- Create a new API microversion to allow abort live migrations in
queued
status. - Modify the Nova client to handle the new microversion.
Dependencies
None
Testing
Would need new in-tree functional and unit tests.
Documentation Impact
Docs needed for new API microversion and usage.
References
History
Release Name | Description |
---|---|
Rocky | Proposed |