10 KiB
Use extend volume completion action
https://blueprints.launchpad.net/nova/+spec/assisted-volume-extend
This blueprint proposes to use the
os-extend_volume_completion
volume action that has been
proposed for Cinder in1, to provide feedback on success or
failure when handling volume-extended
external server
events.
Problem description
Many remotefs-based volume drivers in Cinder use the
qemu-img resize
command to extend volume files. However,
when the volume is attached to a guest, QEMU will lock the file and
qemu-img
will be unable to resize it.
In this case, only the QEMU process holding the lock can resize the
volume, which can be triggered through the QEMU monitor command
block-resize
.
There is currently no adequate way for Cinder to use this feature, so the NFS, NetApp NFS, Powerstore NFS, and Quobyte volume drivers all disable extending attached volumes.
Use Cases
As a user, I want to extend a NFS/NetApp NFS/Powerstore NFS/Quobyte volume while it is attached to an instance and I want the volume size and status to reflect the success or failure of the operation.
Proposed change
Nova's libvirt driver uses the block-resize
command when
handling the volume-extended
external server event, to
inform QEMU that the size of an attached volume has changed. It is in
principle also capable of extending a volume file, but is currently
unable to provide feedback to Cinder on the success of the
operation.
Currently, Cinder will send the volume-extended
external
server event to Nova only after it has finalized the extend operation
and reset the volume status from extending
back to
in-use
.
With2, Cinder will allow volume drivers to
hold off finalizing the extend operation and leave the volume status as
extending
, until after it has send the
volume-extended
event and received feedback from Nova in
form of the os-extend_volume_completion
volume action, with
an error
argument indicating whether to finalize or to roll
back the operation.
This will currently affect only the volume drivers mentioned above,
all of which did not previously support online extend. All other drivers
will continue to send the volume-extended
event after
finalizing the operation and resetting to in-use
status,
and will not expect a os-extend_volume_completion
volume
action.
Compute Agent
Nova's compute agent will use the volume status to differentiate
between the two behaviors when handling volume-extended
events:
If the volume status is
extending
, then it will attempt to readextend_new_size
from the volume's metadata and use this value as the new size of the volume, instead of the volume size field.After successfully extending the volume, it will call the extend volume completion action of the volume, with
"error": false
.If anything goes wrong, including
extend_new_size
being missing from the metadata, or being smaller than the current size of the volume, it will log the error and call theos-extend_volume_completion
action with"error": true
, so Cinder can roll back the operation.For any other volume status, including
in-use
, the event will be handled as before.
API
Nova's API will introduce a new microversion, so that Cinder can make sure the new behavior is available, before leaving an extend operation unfinished.
To handle older compute agents during a rolling upgrade, the API will
also check the compute service version of the target agent when
receiving a volume-extended
event with the new
microversion. If a target compute agent is too old to support the
feature, the API will discard the event and call the
os-extend_volume_completion
action with
"error": true
.
Alternatives
A previous change tried to use the
volume-extended
external server event to support online extend for the NFS driver3, but did not rely on feedback from Nova to Cinder at all. Instead, it would just set the new size of the volume, change the status back toin-use
, notify Nova, and hope for the best.If anything went wrong on Nova's side, this would still result in a volume state indicating that the operation was successful, which is not acceptable.
A previous version of this spec proposed a new synchronous API in Nova4, that would directly call
CompVirtAPI.extend_image
of the nova-compute instance managing the guest that a volume was attached to. This API would provide a single mechanism to trigger the resize operation, communicate the new size to Nova, and get feedback on the success of the operation.The problem with a synchronous API is, that RPC and API timeouts limit the maximum time an extend operation can take. For QEMU, this seemed to be acceptable, because storage preallocation is hard disabled for the
block-resize
command, and because all currently plausible file systems support sparse file operations.However, this may not be true for other volume or virt drivers that might require this API in the future. It would also break with the established pattern of asynchronous coordination between Nova and Cinder, which includes the assisted snapshot and volume migration features.
Following this pattern, we could make the proposed API asynchronous and use a new callback in Cinder, similar to Nova's
os-assisted-volume-snapshots
API, which uses theos-update_snapshot_status
snapshot action to provide feedback to Cinder.The function of the new Nova API would then just be to trigger the operation and to communicate the new size. The question is then, whether that warrants adding a new API to Nova, since there are existing mechanisms that could be used for either.
The existing mechanism for triggering the extend operation in Nova is of course the
volume-extended
external server event. Using it for this purpose, as this spec proposes, requires the target size to be transferred separately, because external server events only have a single text field that is freely usable, which forvolume-extended
is already used for the volume ID.Besides storing it in the admin metadata, as5 and this spec propose, there is also the option of updating the size field of the volume, as6 was essentially doing.
This would require the volume size field to be reset on a failure. If an error response from Nova was lost, the volume would just keep the new size. We would need to extend
os-reset_status
to allow a size reset, or something similar to clean up volumes like this. This would be possible, but updating the size field only after the volume was successfully extended seems like a cleaner solution.We could also extend the external server event API to accept additional data for events, and use this to communicate the new size to Nova.
This option was judged favorably by reviewers on the previous version of this spec,7, but it would be a more complex change to the Nova API.
However, if additional data fields become available in a future version of the external server event API, it would be a relatively minor change to use this instead of volume metadata.
Data model impact
None
REST API impact
The behavior of the external server event API will change.
If Nova receives a
volume-extended
event, and the referenced volume has status ofextending
, Nova will look for theextend_new_size
key in the volume metadata, and use this instead of the volume size field as the target size to update the block device mapping and to pass to the virt driver'sextend_volume
method.Nova will also attempt to call Cinder's new
os-extend_volume_completion
volume action proposed in8 to let Cinder know if the operation was successful or not.Otherwise, the API will behave as before.
Security impact
None
Notifications impact
None
Other end user impact
None
Performance Impact
None
Other deployer impact
None
Developer impact
None
Upgrade impact
Checking the target compute service version allows the API to handle rolling upgrades gracefully.
Implementation
Assignee(s)
- Primary assignee:
-
kgube
- Other contributors:
-
None
Feature Liaison
- Feature liaison:
-
None yet
Work Items
- Update the external server event API to check the target compute
service version for
volume-extended
events. - Update the
ComputeVirtAPI.extend_volume
method to follow the behavior outlined in Compute Agent. - Add unit tests.
- Adapt NFS job in the Nova gate to validate online extend.
Dependencies
- The extend volume completion action9
Testing
We should test that the os-extend_volume_completion
gets
called correctly in all possible error or success condition if a volume
has extending
status.
We should test the case that the call to
os-extend_volume_completion
fails.
We also need to test that volume-extended
continues to
be handled correctly for volumes not in extending
status.
Documentation Impact
The new behavior of the volume-extended
event should be
added to the documentation of the external server event API.
References
History
Release Name | Description |
---|---|
2023.1 Antelope | Accepted |
2023.2 Bobcat | Accepted |
2024.1 Caracal | Reproposed |
https://review.opendev.org/c/openstack/cinder-specs/+/877230↩︎
https://review.opendev.org/c/openstack/cinder-specs/+/877230↩︎
https://review.opendev.org/c/openstack/nova-specs/+/855490/6↩︎
https://review.opendev.org/c/openstack/cinder-specs/+/877230↩︎
https://review.opendev.org/c/openstack/nova-specs/+/855490/6↩︎
https://review.opendev.org/c/openstack/cinder-specs/+/877230↩︎
https://review.opendev.org/c/openstack/cinder-specs/+/877230↩︎