openstack
/
nova-specs
Code Issues Proposed changes
nova-specs/specs/rocky/implemented/enhanced-kvm-storage-qos.rst

5.6 KiB
Raw Blame History

Enhanced KVM Storage QoS

https://blueprints.launchpad.net/nova/+spec/enhanced-kvm-storage-qos

QEMU 1.71 and Libvirt 1.2.1123 provides options to specify maximum burst IOPS and maximum burst bandwidth per disk. Additionally disk IO size can be specified as part of this version of QEMU.

This blueprint proposes to add support for these additonal settings to QoS specs for connected volumes.

Problem description

At the moment, the Nova libvirt driver does not support setting storage IOPS limits. For this reason, some instances might exhaust storage resources, impacting other tenants.

Use Cases

  • Associate burst IOPs and bandwidth front-end QoS specs for volumes exported through Cinder, which will be handled on the hypervisor side. This is in addition to the existing IOPs and bandwidth caps.
  • Set block IO size for IOPs to volumes exported through Cinder.
  • Set IOPs and bandwidth burst limits and block IO sizes for instance attached disks by using Cinder extra specs

Proposed change

Cinder attached volumes can have additional QoS specs assigned. Front-end QoS specs should be applied by Nova when the volume is attached. These are applied per volume.

This blueprint proposes additional per volume QoS specs that will be specified using Cinder volume extra specs. The libvirt driver will apply those IOPS and bandwidth caps to the instance disks on a per volume basis.

Additionally, this blueprint proposes adding the block IO size control using cinder volume extra specs to cinder attached volumes on a per volume basis.

Front-end volume specs will be supported only in case of volumes exported through Cinder. No QoS specs are provided for local drives provided directly by Nova.

Use case examples:

  • Admin sets front-end QoS specs on a specific volume type

    cinder qos-create my-qos consumer=front-end total_bytes_sec_max=300000000 cinder qos-associate my-qos <volume_type_id>

    cinder create <size> --volume-type <volume_type_id>

    # Those QoS specs are applied when the volume is # attached to a KVM instance nova volume-attach <instance_id> <volume_id>

Available additional QoS specs, where each will add an extra line into the libvirt XML definition, specifically in the <iotune> section for each device, are:

  • read_bytes_sec_max

    add <read_bytes_sec_max>value</read_bytes_sec_max>

  • write_bytes_sec_max

    add <write_bytes_sec_max>value</write_bytes_sec_max>

  • total_bytes_sec_max - includes read/writes

    add <total_bytes_sec_max>value</total_bytes_sec_max>

  • read_iops_sec_max

    add <read_iops_sec_max>value</read_iops_sec_max>

  • write_iops_sec_max

    add <write_iops_sec_max>value</write_iops_sec_max>

  • total_iops_sec_max - includes read/writes

    add <total_iops_sec_max>value</total_iops_sec_max>

  • size_iops_sec

    add <size_iops_sec>value</size_iops_sec>

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

As Nova supports N-1 version computes, if these new qoS specs are applied to a compute node running Queens, these new specs will be ignored. No error message will be provided from the Queens node.

Existing Cinder QoS specs are documented in the Cinder Administration documentation set.4

Performance Impact

Allowing burst IOPs and bandwidth for certain volumes will allow some applications to better perform when required.

Other deployer impact

None

Developer impact

None

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:

simon-dodsley

Work Items

  • Add burst and IO size QoS specs support in the libvirt volume driver by extending the nova/virt/libvirt/volumes/ block volume drivers to pass the new properties of a volume to LibvirtConfigGuestDisk via the qos_specs' in the `connection_info dict.
  • Extend the LibvirtConfigGuestDisk class to add the disk burst limits of a disk device5

Dependencies

  • QEMU 1.76
  • Libvirt 1.2.1178

QEMU included in Ubuntu 16.04910 and libvirt at a higher version in Ubuntu 16.041112. Also already included in Fedora 241314

Testing

  • Unit tests
  • Existing tests will ensure that the quest XML is formatted correctly assuming the required versions of libvirt and QEMU are present.

Documentation Impact

The additional QoS features are described in the libvirt driver documentation15.

Will update the Cinder Administrators Guide to add these new front-end QoS storage parameters.

References

History

Revisions
Release Name Description
Rocky Introduced

  1. https://libvirt.org/formatdomain.html↩︎

  2. https://libvirt.org/news-2014.html↩︎

  3. https://www.redhat.com/archives/libvir-list/2014-August/msg01354.html↩︎

  4. https://docs.openstack.org/cinder/latest/admin/blockstorage-capacity-based-qos.html↩︎

  5. https://libvirt.org/news-2014.html↩︎

  6. https://libvirt.org/formatdomain.html↩︎

  7. https://libvirt.org/news-2014.html↩︎

  8. https://www.redhat.com/archives/libvir-list/2014-August/msg01354.html↩︎

  9. https://launchpad.net/ubuntu/+source/libvirt↩︎

  10. https://launchpad.net/ubuntu/+source/qemu↩︎

  11. https://launchpad.net/ubuntu/+source/libvirt↩︎

  12. https://launchpad.net/ubuntu/+source/qemu↩︎

  13. https://apps.fedoraproject.org/packages/qemu↩︎

  14. https://apps.fedoraproject.org/packages/libvirt↩︎

  15. https://libvirt.org/formatdomain.html↩︎